* [Qemu-devel] [POC]colo-proxy in qemu
@ 2015-11-10 5:26 Tkid
2015-11-10 7:35 ` Jason Wang
2015-11-10 10:54 ` Dr. David Alan Gilbert
0 siblings, 2 replies; 63+ messages in thread
From: Tkid @ 2015-11-10 5:26 UTC (permalink / raw)
To: qemu-devel, stefanha, jasowang
Cc: zhang.zhanghailiang, lizhijian, jan.kiszka, eddie.dong, dgilbert,
peter.huangpeng, arei.gonglei, guijianfeng, zhangchen.fnst
[-- Attachment #1: Type: text/plain, Size: 4580 bytes --]
Hi,all
We are planning to reimplement colo proxy in userspace (Here is in qemu) to
cache and compare net packets.This module is one of the important components
of COLO project and now it is still in early stage, so any comments and
feedback are warmly welcomed,thanks in advance.
## Background
COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
Service)
project is a high availability solution. Both Primary VM (PVM) and
Secondary VM
(SVM) run in parallel. They receive the same request from client, and
generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on
demand)
is conducted.
Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO
By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking
called
colo-proxy.
This document describes the design of the colo-proxy module
## Glossary
PVM - Primary VM, which provides services to clients.
SVM - Secondary VM, a hot standby and replication of PVM.
PN - Primary Node, the host which PVM runs on
SN - Secondary Node, the host which SVM runs on
## Our Idea ##
COLO-Proxy
COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for
qemu net filter.the function keep SVM connect normal to PVM and compare
PVM's packets to SVM's packets.if difference,notify COLO do checkpoint.
== Workflow ==
+--+ +--+
|PN| |SN|
+-----------------------+ +-----------------------+
| +-------------------+ | | +-------------------+ |
| | | | | | | |
| | PVM | | | | SVM | |
| | | | | | | |
| +--+-^--------------+ | | +-------------^----++ |
| | | | | | | |
| | | +------------+ | | +-----------+ | | |
| | | | COLO | | (socket) | | COLO | | | |
| | | | CheckPoint +---------------------> CheckPoint| | | |
| | | | | | (6) | | | | | |
| | | +-----^------+ | | +-----------+ | | |
| | | (5) | | | | | |
| | | | | | | | |
| +--v-+--------------+ | Forward(socket) | +-------------+----v+ |
| |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | |
| | +-----+------+ | | +-----------------+ | |
| | | Compare(4) <-------+(3)+---------+ COLO Proxy | |
| +-------------------+ | Forward(socket) | +-------------------+ |
++Qemu+-----------------+ ++Qemu+-----------------+
| ^
| |
| |
+--------v-+--------+
| |
| Client |
| |
+-------------------+
(1)When PN receive client packets,PN COLO-Proxy copy and forward packets to
SN COLO-Proxy.
(2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send
adjusted packets to SVM
(3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu
COLO-Proxy.
(4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then
compare PVM's packets data with SVM's packets data. If packets is
different, compare
module notify COLO CheckPoint module to do a checkpoint then send PVM's
packets to
client and drop SVM's packets, otherwise, just send PVM's packets to
client and
drop SVM's packets.
(5)notify COLO-Checkpoint module checkpoint is needed
(6)Do COLO-Checkpoint
### QEMU space TCP/IP stack(Based on SLIRP) ###
We need a QEMU space TCP/IP stack to help us to analysis packet. After
looking
into QEMU, we found that SLIRP
http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU,
it can
help use to handle the packet written to/read from backend(tap) device
which is
just like a link layer(L2) packet.
### Packet enqueue and compare ###
Together with QEMU space TCP/IP stack, we enqueue all packets sent by
PVM and
SVM on Primary QEMU, and then compare the packet payload for each
connection.
[-- Attachment #2: Type: text/html, Size: 6443 bytes --]
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 5:26 [Qemu-devel] [POC]colo-proxy in qemu Tkid @ 2015-11-10 7:35 ` Jason Wang 2015-11-10 8:30 ` zhanghailiang ` (3 more replies) 2015-11-10 10:54 ` Dr. David Alan Gilbert 1 sibling, 4 replies; 63+ messages in thread From: Jason Wang @ 2015-11-10 7:35 UTC (permalink / raw) To: Tkid, qemu-devel, stefanha Cc: zhang.zhanghailiang, lizhijian, jan.kiszka, eddie.dong, dgilbert, peter.huangpeng, arei.gonglei, guijianfeng On 11/10/2015 01:26 PM, Tkid wrote: > Hi,all > > We are planning to reimplement colo proxy in userspace (Here is in > qemu) to > cache and compare net packets.This module is one of the important > components > of COLO project and now it is still in early stage, so any comments and > feedback are warmly welcomed,thanks in advance. > > ## Background > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > Service) > project is a high availability solution. Both Primary VM (PVM) and > Secondary VM > (SVM) run in parallel. They receive the same request from client, and > generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint > (on demand) > is conducted. > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and > finding out > whether they are identical, we introduce a new module to qemu > networking called > colo-proxy. > > This document describes the design of the colo-proxy module > > ## Glossary > PVM - Primary VM, which provides services to clients. > SVM - Secondary VM, a hot standby and replication of PVM. > PN - Primary Node, the host which PVM runs on > SN - Secondary Node, the host which SVM runs on > > ## Our Idea ## > > COLO-Proxy > COLO-Proxy is a part of COLO,based on qemu net filter and it's a > plugin for > qemu net filter.the function keep SVM connect normal to PVM and compare > PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. > > == Workflow == > > > +--+ +--+ > |PN| |SN| > +-----------------------+ +-----------------------+ > | +-------------------+ | | +-------------------+ | > | | | | | | | | > | | PVM | | | | SVM | | > | | | | | | | | > | +--+-^--------------+ | | +-------------^----++ | > | | | | | | | | > | | | +------------+ | | +-----------+ | | | > | | | | COLO | | (socket) | | COLO | | | | > | | | | CheckPoint +---------------------> CheckPoint| | | | > | | | | | | (6) | | | | | | > | | | +-----^------+ | | +-----------+ | | | > | | | (5) | | | | | | > | | | | | | | | | > | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | > | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | > | | +-----+------+ | | +-----------------+ | | > | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | > | +-------------------+ | Forward(socket) | +-------------------+ | > ++Qemu+-----------------+ ++Qemu+-----------------+ > | ^ > | | > | | > +--------v-+--------+ > | | > | Client | > | | > +-------------------+ > > > > > (1)When PN receive client packets,PN COLO-Proxy copy and forward > packets to > SN COLO-Proxy. > (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send > adjusted packets to SVM > (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu > COLO-Proxy. > (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then > compare PVM's packets data with SVM's packets data. If packets is > different, compare > module notify COLO CheckPoint module to do a checkpoint then send > PVM's packets to > client and drop SVM's packets, otherwise, just send PVM's packets to > client and > drop SVM's packets. > (5)notify COLO-Checkpoint module checkpoint is needed > (6)Do COLO-Checkpoint > > ### QEMU space TCP/IP stack(Based on SLIRP) ### > We need a QEMU space TCP/IP stack to help us to analysis packet. After > looking > into QEMU, we found that SLIRP > > http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 > > is a good choice for us. SLIRP proivdes a full TCP/IP stack within > QEMU, it can > help use to handle the packet written to/read from backend(tap) device > which is > just like a link layer(L2) packet. > > ### Packet enqueue and compare ### > Together with QEMU space TCP/IP stack, we enqueue all packets sent by > PVM and > SVM on Primary QEMU, and then compare the packet payload for each > connection. > Hi: Just have the following questions in my mind (some has been raised in the previous rounds of discussion without a conclusion): - What's the plan for management layer? The setup seems complicated so we could not simply depend on user to do each step. (And for security reason, qemu was usually run as unprivileged user) - What's the plan for vhost? Userspace network in qemu is rather slow, most user will choose vhost. - What if application generate packet based on hwrng device? This will produce always different packets. - Not sure SLIRP is perfect matched for this task. As has been raised, another method is to decouple the packet comparing from qemu. In this way, lots of open source userspace stack could be used. - Haven't read the code of packet comparing, but if it needs to keep track the state of each connection, it could be easily DOS from guest. Thanks ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 7:35 ` Jason Wang @ 2015-11-10 8:30 ` zhanghailiang 2015-11-11 2:28 ` Jason Wang 2015-11-10 9:35 ` Tkid ` (2 subsequent siblings) 3 siblings, 1 reply; 63+ messages in thread From: zhanghailiang @ 2015-11-10 8:30 UTC (permalink / raw) To: Jason Wang, Tkid, qemu-devel, stefanha Cc: lizhijian, jan.kiszka, eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, guijianfeng On 2015/11/10 15:35, Jason Wang wrote: > > > On 11/10/2015 01:26 PM, Tkid wrote: >> Hi,all >> >> We are planning to reimplement colo proxy in userspace (Here is in >> qemu) to >> cache and compare net packets.This module is one of the important >> components >> of COLO project and now it is still in early stage, so any comments and >> feedback are warmly welcomed,thanks in advance. >> >> ## Background >> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >> Service) >> project is a high availability solution. Both Primary VM (PVM) and >> Secondary VM >> (SVM) run in parallel. They receive the same request from client, and >> generate >> responses in parallel too. If the response packets from PVM and SVM are >> identical, they are released immediately. Otherwise, a VM checkpoint >> (on demand) >> is conducted. >> Paper: >> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >> COLO on Xen: >> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >> COLO on Qemu/KVM: >> http://wiki.qemu.org/Features/COLO >> >> By the needs of capturing response packets from PVM and SVM and >> finding out >> whether they are identical, we introduce a new module to qemu >> networking called >> colo-proxy. >> >> This document describes the design of the colo-proxy module >> >> ## Glossary >> PVM - Primary VM, which provides services to clients. >> SVM - Secondary VM, a hot standby and replication of PVM. >> PN - Primary Node, the host which PVM runs on >> SN - Secondary Node, the host which SVM runs on >> >> ## Our Idea ## >> >> COLO-Proxy >> COLO-Proxy is a part of COLO,based on qemu net filter and it's a >> plugin for >> qemu net filter.the function keep SVM connect normal to PVM and compare >> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >> >> == Workflow == >> >> >> +--+ +--+ >> |PN| |SN| >> +-----------------------+ +-----------------------+ >> | +-------------------+ | | +-------------------+ | >> | | | | | | | | >> | | PVM | | | | SVM | | >> | | | | | | | | >> | +--+-^--------------+ | | +-------------^----++ | >> | | | | | | | | >> | | | +------------+ | | +-----------+ | | | >> | | | | COLO | | (socket) | | COLO | | | | >> | | | | CheckPoint +---------------------> CheckPoint| | | | >> | | | | | | (6) | | | | | | >> | | | +-----^------+ | | +-----------+ | | | >> | | | (5) | | | | | | >> | | | | | | | | | >> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >> | | +-----+------+ | | +-----------------+ | | >> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >> | +-------------------+ | Forward(socket) | +-------------------+ | >> ++Qemu+-----------------+ ++Qemu+-----------------+ >> | ^ >> | | >> | | >> +--------v-+--------+ >> | | >> | Client | >> | | >> +-------------------+ >> >> >> >> >> (1)When PN receive client packets,PN COLO-Proxy copy and forward >> packets to >> SN COLO-Proxy. >> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send >> adjusted packets to SVM >> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >> COLO-Proxy. >> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then >> compare PVM's packets data with SVM's packets data. If packets is >> different, compare >> module notify COLO CheckPoint module to do a checkpoint then send >> PVM's packets to >> client and drop SVM's packets, otherwise, just send PVM's packets to >> client and >> drop SVM's packets. >> (5)notify COLO-Checkpoint module checkpoint is needed >> (6)Do COLO-Checkpoint >> >> ### QEMU space TCP/IP stack(Based on SLIRP) ### >> We need a QEMU space TCP/IP stack to help us to analysis packet. After >> looking >> into QEMU, we found that SLIRP >> >> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 >> >> is a good choice for us. SLIRP proivdes a full TCP/IP stack within >> QEMU, it can >> help use to handle the packet written to/read from backend(tap) device >> which is >> just like a link layer(L2) packet. >> >> ### Packet enqueue and compare ### >> Together with QEMU space TCP/IP stack, we enqueue all packets sent by >> PVM and >> SVM on Primary QEMU, and then compare the packet payload for each >> connection. >> > > Hi: > > Just have the following questions in my mind (some has been raised in > the previous rounds of discussion without a conclusion): > > - What's the plan for management layer? The setup seems complicated so > we could not simply depend on user to do each step. (And for security > reason, qemu was usually run as unprivileged user) We will do most of the setup works automatically in qemu as possible as we can. Compared with kernel proxy scheme, it is not a big deal. :) > - What's the plan for vhost? Userspace network in qemu is rather slow, > most user will choose vhost. > - What if application generate packet based on hwrng device? This will > produce always different packets. Yes, that is really a big problem, actually, we have discussed it for many times, it seems that there is no perfect way to solve it. :( We have a compromise approach, when we find there are too many continuous checkpoint requests, we switch COLO from normal mode to periodic mode which SVM will stop running. (Dave have realized this before, which called Hybrid mode. The patches is "[RFC/COLO: 0/3] Hybrid mode and parameterisation") > - Not sure SLIRP is perfect matched for this task. As has been raised, > another method is to decouple the packet comparing from qemu. In this > way, lots of open source userspace stack could be used. Hmm, it seems to be a good idea, maybe we can add a checkpoint request command in COLO to support more packets comparing scheme ... Thanks， zhanghailiang > - Haven't read the code of packet comparing, but if it needs to keep > track the state of each connection, it could be easily DOS from guest. > > Thanks > > . > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 8:30 ` zhanghailiang @ 2015-11-11 2:28 ` Jason Wang 0 siblings, 0 replies; 63+ messages in thread From: Jason Wang @ 2015-11-11 2:28 UTC (permalink / raw) To: zhanghailiang, Tkid, qemu-devel, stefanha Cc: lizhijian, jan.kiszka, eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, guijianfeng On 11/10/2015 04:30 PM, zhanghailiang wrote: > On 2015/11/10 15:35, Jason Wang wrote: >> >> >> On 11/10/2015 01:26 PM, Tkid wrote: >>> Hi,all >>> >>> We are planning to reimplement colo proxy in userspace (Here is in >>> qemu) to >>> cache and compare net packets.This module is one of the important >>> components >>> of COLO project and now it is still in early stage, so any comments and >>> feedback are warmly welcomed,thanks in advance. >>> >>> ## Background >>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >>> Service) >>> project is a high availability solution. Both Primary VM (PVM) and >>> Secondary VM >>> (SVM) run in parallel. They receive the same request from client, and >>> generate >>> responses in parallel too. If the response packets from PVM and SVM are >>> identical, they are released immediately. Otherwise, a VM checkpoint >>> (on demand) >>> is conducted. >>> Paper: >>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >>> COLO on Xen: >>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >>> COLO on Qemu/KVM: >>> http://wiki.qemu.org/Features/COLO >>> >>> By the needs of capturing response packets from PVM and SVM and >>> finding out >>> whether they are identical, we introduce a new module to qemu >>> networking called >>> colo-proxy. >>> >>> This document describes the design of the colo-proxy module >>> >>> ## Glossary >>> PVM - Primary VM, which provides services to clients. >>> SVM - Secondary VM, a hot standby and replication of PVM. >>> PN - Primary Node, the host which PVM runs on >>> SN - Secondary Node, the host which SVM runs on >>> >>> ## Our Idea ## >>> >>> COLO-Proxy >>> COLO-Proxy is a part of COLO,based on qemu net filter and it's a >>> plugin for >>> qemu net filter.the function keep SVM connect normal to PVM and compare >>> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >>> >>> == Workflow == >>> >>> >>> +--+ +--+ >>> |PN| |SN| >>> +-----------------------+ +-----------------------+ >>> | +-------------------+ | | +-------------------+ | >>> | | | | | | | | >>> | | PVM | | | | SVM | | >>> | | | | | | | | >>> | +--+-^--------------+ | | +-------------^----++ | >>> | | | | | | | | >>> | | | +------------+ | | +-----------+ | | | >>> | | | | COLO | | (socket) | | COLO | | | | >>> | | | | CheckPoint +---------------------> CheckPoint| | | | >>> | | | | | | (6) | | | | | | >>> | | | +-----^------+ | | +-----------+ | | | >>> | | | (5) | | | | | | >>> | | | | | | | | | >>> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >>> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >>> | | +-----+------+ | | +-----------------+ | | >>> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >>> | +-------------------+ | Forward(socket) | +-------------------+ | >>> ++Qemu+-----------------+ ++Qemu+-----------------+ >>> | ^ >>> | | >>> | | >>> +--------v-+--------+ >>> | | >>> | Client | >>> | | >>> +-------------------+ >>> >>> >>> >>> >>> (1)When PN receive client packets,PN COLO-Proxy copy and forward >>> packets to >>> SN COLO-Proxy. >>> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's >>> ack,send >>> adjusted packets to SVM >>> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >>> COLO-Proxy. >>> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's >>> packets,then >>> compare PVM's packets data with SVM's packets data. If packets is >>> different, compare >>> module notify COLO CheckPoint module to do a checkpoint then send >>> PVM's packets to >>> client and drop SVM's packets, otherwise, just send PVM's packets to >>> client and >>> drop SVM's packets. >>> (5)notify COLO-Checkpoint module checkpoint is needed >>> (6)Do COLO-Checkpoint >>> >>> ### QEMU space TCP/IP stack(Based on SLIRP) ### >>> We need a QEMU space TCP/IP stack to help us to analysis packet. After >>> looking >>> into QEMU, we found that SLIRP >>> >>> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 >>> >>> >>> is a good choice for us. SLIRP proivdes a full TCP/IP stack within >>> QEMU, it can >>> help use to handle the packet written to/read from backend(tap) device >>> which is >>> just like a link layer(L2) packet. >>> >>> ### Packet enqueue and compare ### >>> Together with QEMU space TCP/IP stack, we enqueue all packets sent by >>> PVM and >>> SVM on Primary QEMU, and then compare the packet payload for each >>> connection. >>> >> >> Hi: >> >> Just have the following questions in my mind (some has been raised in >> the previous rounds of discussion without a conclusion): >> >> - What's the plan for management layer? The setup seems complicated so >> we could not simply depend on user to do each step. (And for security >> reason, qemu was usually run as unprivileged user) > > We will do most of the setup works automatically in qemu as possible > as we can. > Compared with kernel proxy scheme, it is not a big deal. :) Yes, but what I mean is for host. E.g setting up network like bridge or others. > >> - What's the plan for vhost? Userspace network in qemu is rather slow, >> most user will choose vhost. >> - What if application generate packet based on hwrng device? This will >> produce always different packets. > > Yes, that is really a big problem, actually, we have discussed it for > many > times, it seems that there is no perfect way to solve it. :( > We have a compromise approach, when we find there are too many continuous > checkpoint requests, we switch COLO from normal mode to periodic mode > which SVM will stop running. > (Dave have realized this before, which called Hybrid mode. The patches is > "[RFC/COLO: 0/3] Hybrid mode and parameterisation") Aha, I see. Thanks for the pointer. > >> - Not sure SLIRP is perfect matched for this task. As has been raised, >> another method is to decouple the packet comparing from qemu. In this >> way, lots of open source userspace stack could be used. > > Hmm, it seems to be a good idea, maybe we can add a checkpoint request > command > in COLO to support more packets comparing scheme ... > Yes, then when to synchronize could be determined by external program. (Just an idea FYI). > Thanks， > zhanghailiang > >> - Haven't read the code of packet comparing, but if it needs to keep >> track the state of each connection, it could be easily DOS from guest. >> >> Thanks >> >> . >> > > > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 7:35 ` Jason Wang 2015-11-10 8:30 ` zhanghailiang @ 2015-11-10 9:35 ` Tkid 2015-11-11 3:04 ` Jason Wang 2015-11-10 9:41 ` Dr. David Alan Gilbert 2015-11-11 1:23 ` Dong, Eddie 3 siblings, 1 reply; 63+ messages in thread From: Tkid @ 2015-11-10 9:35 UTC (permalink / raw) To: Jason Wang, qemu-devel, stefanha Cc: zhang.zhanghailiang, lizhijian, jan.kiszka, eddie.dong, dgilbert, peter.huangpeng, arei.gonglei, guijianfeng On 11/10/2015 03:35 PM, Jason Wang wrote: > On 11/10/2015 01:26 PM, Tkid wrote: >> Hi,all >> >> We are planning to reimplement colo proxy in userspace (Here is in >> qemu) to >> cache and compare net packets.This module is one of the important >> components >> of COLO project and now it is still in early stage, so any comments and >> feedback are warmly welcomed,thanks in advance. >> >> ## Background >> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >> Service) >> project is a high availability solution. Both Primary VM (PVM) and >> Secondary VM >> (SVM) run in parallel. They receive the same request from client, and >> generate >> responses in parallel too. If the response packets from PVM and SVM are >> identical, they are released immediately. Otherwise, a VM checkpoint >> (on demand) >> is conducted. >> Paper: >> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >> COLO on Xen: >> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >> COLO on Qemu/KVM: >> http://wiki.qemu.org/Features/COLO >> >> By the needs of capturing response packets from PVM and SVM and >> finding out >> whether they are identical, we introduce a new module to qemu >> networking called >> colo-proxy. >> >> This document describes the design of the colo-proxy module >> >> ## Glossary >> PVM - Primary VM, which provides services to clients. >> SVM - Secondary VM, a hot standby and replication of PVM. >> PN - Primary Node, the host which PVM runs on >> SN - Secondary Node, the host which SVM runs on >> >> ## Our Idea ## >> >> COLO-Proxy >> COLO-Proxy is a part of COLO,based on qemu net filter and it's a >> plugin for >> qemu net filter.the function keep SVM connect normal to PVM and compare >> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >> >> == Workflow == >> >> +--+ +--+ >> |PN| |SN| >> +-----------------------+ +-----------------------+ >> | +-------------------+ | | +-------------------+ | >> | | | | | | | | >> | | PVM | | | | SVM | | >> | | | | | | | | >> | +--+-^--------------+ | | +-------------^----++ | >> | | | | | | | | >> | | | +------------+ | | +-----------+ | | | >> | | | | COLO | | (socket) | | COLO | | | | >> | | | | CheckPoint +---------------------> CheckPoint| | | | >> | | | | | | (6) | | | | | | >> | | | +-----^------+ | | +-----------+ | | | >> | | | (5) | | | | | | >> | | | | | | | | | >> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >> | | +-----+------+ | | +-----------------+ | | >> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >> | +-------------------+ | Forward(socket) | +-------------------+ | >> ++Qemu+-----------------+ ++Qemu+-----------------+ >> | ^ >> | | >> | | >> +--------v-+--------+ >> | | >> | Client | >> | | >> +-------------------+ >> >> >> (1)When PN receive client packets,PN COLO-Proxy copy and forward >> packets to >> SN COLO-Proxy. >> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send >> adjusted packets to SVM >> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >> COLO-Proxy. >> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then >> compare PVM's packets data with SVM's packets data. If packets is >> different, compare >> module notify COLO CheckPoint module to do a checkpoint then send >> PVM's packets to >> client and drop SVM's packets, otherwise, just send PVM's packets to >> client and >> drop SVM's packets. >> (5)notify COLO-Checkpoint module checkpoint is needed >> (6)Do COLO-Checkpoint >> >> ### QEMU space TCP/IP stack(Based on SLIRP) ### >> We need a QEMU space TCP/IP stack to help us to analysis packet. After >> looking >> into QEMU, we found that SLIRP >> >> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 >> >> is a good choice for us. SLIRP proivdes a full TCP/IP stack within >> QEMU, it can >> help use to handle the packet written to/read from backend(tap) device >> which is >> just like a link layer(L2) packet. >> >> ### Packet enqueue and compare ### >> Together with QEMU space TCP/IP stack, we enqueue all packets sent by >> PVM and >> SVM on Primary QEMU, and then compare the packet payload for each >> connection. >> Thanks for review ~ > Hi: > > Just have the following questions in my mind (some has been raised in > the previous rounds of discussion without a conclusion): > > - What's the plan for management layer? The setup seems complicated so > we could not simply depend on user to do each step. (And for security > reason, qemu was usually run as unprivileged user) -We don't need to run as privileged user, colo-proxy just run like filter-buffer. usage: primary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,side=primary,host=3.3.3.8,port=xxx secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object colo-proxy,id=f0,netdev=bn0,queue=all,side=secondary,server=tcp:xxxx:port > - What's the plan for vhost? Userspace network in qemu is rather slow, > most user will choose vhost. colo-proxy in qemu space don't support vhost. people who want to use colo must disable vhost, but virtio-net is another choice which is enough in most case. > - What if application generate packet based on hwrng device? This will > produce always different packets. just like hailiang said. > - Not sure SLIRP is perfect matched for this task. As has been raised, > another method is to decouple the packet comparing from qemu. In this > way, lots of open source userspace stack could be used. -we just need the some capabilities(such as IP frag/defrag) of SLIRP。 We have investigated some open source userspace stack,but not find one better to SLIRP. if you know,please tell me. > - Haven't read the code of packet comparing, but if it needs to keep > track the state of each connection, it could be easily DOS from guest. -We think preventDOS from guest is out of our focus,it should be firewall to concerned. > Thanks > . > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 9:35 ` Tkid @ 2015-11-11 3:04 ` Jason Wang 0 siblings, 0 replies; 63+ messages in thread From: Jason Wang @ 2015-11-11 3:04 UTC (permalink / raw) To: Tkid, qemu-devel, stefanha Cc: zhang.zhanghailiang, lizhijian, jan.kiszka, eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, guijianfeng On 11/10/2015 05:35 PM, Tkid wrote: > > > On 11/10/2015 03:35 PM, Jason Wang wrote: >> On 11/10/2015 01:26 PM, Tkid wrote: >>> Hi,all >>> >>> We are planning to reimplement colo proxy in userspace (Here is in >>> qemu) to >>> cache and compare net packets.This module is one of the important >>> components >>> of COLO project and now it is still in early stage, so any comments and >>> feedback are warmly welcomed,thanks in advance. >>> >>> ## Background >>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >>> Service) >>> project is a high availability solution. Both Primary VM (PVM) and >>> Secondary VM >>> (SVM) run in parallel. They receive the same request from client, and >>> generate >>> responses in parallel too. If the response packets from PVM and SVM are >>> identical, they are released immediately. Otherwise, a VM checkpoint >>> (on demand) >>> is conducted. >>> Paper: >>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >>> COLO on Xen: >>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >>> COLO on Qemu/KVM: >>> http://wiki.qemu.org/Features/COLO >>> >>> By the needs of capturing response packets from PVM and SVM and >>> finding out >>> whether they are identical, we introduce a new module to qemu >>> networking called >>> colo-proxy. >>> >>> This document describes the design of the colo-proxy module >>> >>> ## Glossary >>> PVM - Primary VM, which provides services to clients. >>> SVM - Secondary VM, a hot standby and replication of PVM. >>> PN - Primary Node, the host which PVM runs on >>> SN - Secondary Node, the host which SVM runs on >>> >>> ## Our Idea ## >>> >>> COLO-Proxy >>> COLO-Proxy is a part of COLO,based on qemu net filter and it's a >>> plugin for >>> qemu net filter.the function keep SVM connect normal to PVM and compare >>> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >>> >>> == Workflow == >>> >>> +--+ +--+ >>> |PN| |SN| >>> +-----------------------+ +-----------------------+ >>> | +-------------------+ | | +-------------------+ | >>> | | | | | | | | >>> | | PVM | | | | SVM | | >>> | | | | | | | | >>> | +--+-^--------------+ | | +-------------^----++ | >>> | | | | | | | | >>> | | | +------------+ | | +-----------+ | | | >>> | | | | COLO | | (socket) | | COLO | | | | >>> | | | | CheckPoint +---------------------> CheckPoint| | | | >>> | | | | | | (6) | | | | | | >>> | | | +-----^------+ | | +-----------+ | | | >>> | | | (5) | | | | | | >>> | | | | | | | | | >>> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >>> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >>> | | +-----+------+ | | +-----------------+ | | >>> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >>> | +-------------------+ | Forward(socket) | +-------------------+ | >>> ++Qemu+-----------------+ ++Qemu+-----------------+ >>> | ^ >>> | | >>> | | >>> +--------v-+--------+ >>> | | >>> | Client | >>> | | >>> +-------------------+ >>> >>> >>> (1)When PN receive client packets,PN COLO-Proxy copy and forward >>> packets to >>> SN COLO-Proxy. >>> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's >>> ack,send >>> adjusted packets to SVM >>> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >>> COLO-Proxy. >>> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's >>> packets,then >>> compare PVM's packets data with SVM's packets data. If packets is >>> different, compare >>> module notify COLO CheckPoint module to do a checkpoint then send >>> PVM's packets to >>> client and drop SVM's packets, otherwise, just send PVM's packets to >>> client and >>> drop SVM's packets. >>> (5)notify COLO-Checkpoint module checkpoint is needed >>> (6)Do COLO-Checkpoint >>> >>> ### QEMU space TCP/IP stack(Based on SLIRP) ### >>> We need a QEMU space TCP/IP stack to help us to analysis packet. After >>> looking >>> into QEMU, we found that SLIRP >>> >>> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 >>> >>> >>> is a good choice for us. SLIRP proivdes a full TCP/IP stack within >>> QEMU, it can >>> help use to handle the packet written to/read from backend(tap) device >>> which is >>> just like a link layer(L2) packet. >>> >>> ### Packet enqueue and compare ### >>> Together with QEMU space TCP/IP stack, we enqueue all packets sent by >>> PVM and >>> SVM on Primary QEMU, and then compare the packet payload for each >>> connection. >>> > Thanks for review ~ >> Hi: >> >> Just have the following questions in my mind (some has been raised in >> the previous rounds of discussion without a conclusion): >> >> - What's the plan for management layer? The setup seems complicated so >> we could not simply depend on user to do each step. (And for security >> reason, qemu was usually run as unprivileged user) > -We don't need to run as privileged user, colo-proxy just run like > filter-buffer. usage: primary: -netdev tap,id=bn0 -device > e1000,netdev=bn0 -object > colo-proxy,id=f0,netdev=bn0,queue=all,side=primary,host=3.3.3.8,port=xxx > secondary: -netdev tap,id=bn0 -device e1000,netdev=bn0 -object > colo-proxy,id=f0,netdev=bn0,queue=all,side=secondary,server=tcp:xxxx:port Ok. >> - What's the plan for vhost? Userspace network in qemu is rather slow, >> most user will choose vhost. > colo-proxy in qemu space don't support vhost. people who want to use > colo must disable vhost, but virtio-net is another choice which is > enough in most case. Ok for function but not for performance :) There're lots of users that cares about performance. >> - What if application generate packet based on hwrng device? This will >> produce always different packets. > just like hailiang said. >> - Not sure SLIRP is perfect matched for this task. As has been raised, >> another method is to decouple the packet comparing from qemu. In this >> way, lots of open source userspace stack could be used. > -we just need the some capabilities(such as IP frag/defrag) of SLIRP。 > We have investigated some open source userspace stack,but not find one > better to SLIRP. if you know,please tell me. I think it's ok to use SLIRP. But it has some drawbacks: - Lacking ipv6 support. Which means you need implement this (there're rfc posted in the list) even if you may only want the (de)fragmentation. - Not used in any production environment AFAIK, so maybe buggy and you need to fix the bugs. So if possible choosing an certified ip stack which may save lots of attentions. For userspace IP implementation, not very familiar, google gives me this something like uip, lwip and libuinet. >> - Haven't read the code of packet comparing, but if it needs to keep >> track the state of each connection, it could be easily DOS from guest. > > -We think preventDOS from guest is out of our focus,it should be > firewall to concerned. Maybe net clear for the question. I mean e.g if you need to track the state of each connection, is there a limitation of the maximum connections that is accepted? If yes, what if guest have more connections than this, switch to periodic mode? If no, guest could exhaust the host memory by faking connections in guest. > >> Thanks >> . >> > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 7:35 ` Jason Wang 2015-11-10 8:30 ` zhanghailiang 2015-11-10 9:35 ` Tkid @ 2015-11-10 9:41 ` Dr. David Alan Gilbert 2015-11-11 3:09 ` Jason Wang 2015-11-11 1:23 ` Dong, Eddie 3 siblings, 1 reply; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-10 9:41 UTC (permalink / raw) To: Jason Wang Cc: Tkid, lizhijian, jan.kiszka, eddie.dong, qemu-devel, peter.huangpeng, arei.gonglei, luis, stefanha, guijianfeng, zhang.zhanghailiang * Jason Wang (jasowang@redhat.com) wrote: > > > On 11/10/2015 01:26 PM, Tkid wrote: > > Hi,all > > > > We are planning to reimplement colo proxy in userspace (Here is in > > qemu) to > > cache and compare net packets.This module is one of the important > > components > > of COLO project and now it is still in early stage, so any comments and > > feedback are warmly welcomed,thanks in advance. > > > > ## Background > > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > > Service) > > project is a high availability solution. Both Primary VM (PVM) and > > Secondary VM > > (SVM) run in parallel. They receive the same request from client, and > > generate > > responses in parallel too. If the response packets from PVM and SVM are > > identical, they are released immediately. Otherwise, a VM checkpoint > > (on demand) > > is conducted. > > Paper: > > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > > COLO on Xen: > > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > > COLO on Qemu/KVM: > > http://wiki.qemu.org/Features/COLO > > > > By the needs of capturing response packets from PVM and SVM and > > finding out > > whether they are identical, we introduce a new module to qemu > > networking called > > colo-proxy. > > > > This document describes the design of the colo-proxy module > > > > ## Glossary > > PVM - Primary VM, which provides services to clients. > > SVM - Secondary VM, a hot standby and replication of PVM. > > PN - Primary Node, the host which PVM runs on > > SN - Secondary Node, the host which SVM runs on > > > > ## Our Idea ## > > > > COLO-Proxy > > COLO-Proxy is a part of COLO,based on qemu net filter and it's a > > plugin for > > qemu net filter.the function keep SVM connect normal to PVM and compare > > PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. > > > > == Workflow == > > > > > > +--+ +--+ > > |PN| |SN| > > +-----------------------+ +-----------------------+ > > | +-------------------+ | | +-------------------+ | > > | | | | | | | | > > | | PVM | | | | SVM | | > > | | | | | | | | > > | +--+-^--------------+ | | +-------------^----++ | > > | | | | | | | | > > | | | +------------+ | | +-----------+ | | | > > | | | | COLO | | (socket) | | COLO | | | | > > | | | | CheckPoint +---------------------> CheckPoint| | | | > > | | | | | | (6) | | | | | | > > | | | +-----^------+ | | +-----------+ | | | > > | | | (5) | | | | | | > > | | | | | | | | | > > | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | > > | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | > > | | +-----+------+ | | +-----------------+ | | > > | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | > > | +-------------------+ | Forward(socket) | +-------------------+ | > > ++Qemu+-----------------+ ++Qemu+-----------------+ > > | ^ > > | | > > | | > > +--------v-+--------+ > > | | > > | Client | > > | | > > +-------------------+ > > > > > > > > > > (1)When PN receive client packets,PN COLO-Proxy copy and forward > > packets to > > SN COLO-Proxy. > > (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send > > adjusted packets to SVM > > (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu > > COLO-Proxy. > > (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then > > compare PVM's packets data with SVM's packets data. If packets is > > different, compare > > module notify COLO CheckPoint module to do a checkpoint then send > > PVM's packets to > > client and drop SVM's packets, otherwise, just send PVM's packets to > > client and > > drop SVM's packets. > > (5)notify COLO-Checkpoint module checkpoint is needed > > (6)Do COLO-Checkpoint > > > > ### QEMU space TCP/IP stack(Based on SLIRP) ### > > We need a QEMU space TCP/IP stack to help us to analysis packet. After > > looking > > into QEMU, we found that SLIRP > > > > http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 > > > > is a good choice for us. SLIRP proivdes a full TCP/IP stack within > > QEMU, it can > > help use to handle the packet written to/read from backend(tap) device > > which is > > just like a link layer(L2) packet. > > > > ### Packet enqueue and compare ### > > Together with QEMU space TCP/IP stack, we enqueue all packets sent by > > PVM and > > SVM on Primary QEMU, and then compare the packet payload for each > > connection. > > > > Hi: > > Just have the following questions in my mind (some has been raised in > the previous rounds of discussion without a conclusion): > > - What's the plan for management layer? The setup seems complicated so > we could not simply depend on user to do each step. (And for security > reason, qemu was usually run as unprivileged user) It's certainly easier than the current COLO code that relies on a very complex set of bridges, extra network interfaces and kernel modules. UMU (cc'd) have been working on a libvirt set that starts COLO up, although one bit that's very messy is the curretn kernel based network comparison code. > - What's the plan for vhost? Userspace network in qemu is rather slow, > most user will choose vhost. > - What if application generate packet based on hwrng device? This will > produce always different packets. Yes, there are cases this happens - COLO's worst case is similar to simple checkpointing (because it has a limit to the smallest checkpoint), but it's best case is much better, on a compute heavy load, it ends up taking a checkpoint very rarely. Actually the big problem is where randomness occurs in unexpected places, e.g. where things like Perl's hash randomisation means that the two hosts produce the same data in different orders. > - Not sure SLIRP is perfect matched for this task. As has been raised, > another method is to decouple the packet comparing from qemu. In this > way, lots of open source userspace stack could be used. > - Haven't read the code of packet comparing, but if it needs to keep > track the state of each connection, it could be easily DOS from guest. The guest can only break it's own networking; so shooting itself in the foot is no big deal. Dave > > Thanks -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 9:41 ` Dr. David Alan Gilbert @ 2015-11-11 3:09 ` Jason Wang 2015-11-11 9:03 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-11-11 3:09 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: Tkid, lizhijian, jan.kiszka, eddie.dong, peter.huangpeng, qemu-devel, arei.gonglei, luis, stefanha, guijianfeng, zhang.zhanghailiang On 11/10/2015 05:41 PM, Dr. David Alan Gilbert wrote: > * Jason Wang (jasowang@redhat.com) wrote: >> >> On 11/10/2015 01:26 PM, Tkid wrote: >>> Hi,all >>> >>> We are planning to reimplement colo proxy in userspace (Here is in >>> qemu) to >>> cache and compare net packets.This module is one of the important >>> components >>> of COLO project and now it is still in early stage, so any comments and >>> feedback are warmly welcomed,thanks in advance. >>> >>> ## Background >>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >>> Service) >>> project is a high availability solution. Both Primary VM (PVM) and >>> Secondary VM >>> (SVM) run in parallel. They receive the same request from client, and >>> generate >>> responses in parallel too. If the response packets from PVM and SVM are >>> identical, they are released immediately. Otherwise, a VM checkpoint >>> (on demand) >>> is conducted. >>> Paper: >>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >>> COLO on Xen: >>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >>> COLO on Qemu/KVM: >>> http://wiki.qemu.org/Features/COLO >>> >>> By the needs of capturing response packets from PVM and SVM and >>> finding out >>> whether they are identical, we introduce a new module to qemu >>> networking called >>> colo-proxy. >>> >>> This document describes the design of the colo-proxy module >>> >>> ## Glossary >>> PVM - Primary VM, which provides services to clients. >>> SVM - Secondary VM, a hot standby and replication of PVM. >>> PN - Primary Node, the host which PVM runs on >>> SN - Secondary Node, the host which SVM runs on >>> >>> ## Our Idea ## >>> >>> COLO-Proxy >>> COLO-Proxy is a part of COLO,based on qemu net filter and it's a >>> plugin for >>> qemu net filter.the function keep SVM connect normal to PVM and compare >>> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >>> >>> == Workflow == >>> >>> >>> +--+ +--+ >>> |PN| |SN| >>> +-----------------------+ +-----------------------+ >>> | +-------------------+ | | +-------------------+ | >>> | | | | | | | | >>> | | PVM | | | | SVM | | >>> | | | | | | | | >>> | +--+-^--------------+ | | +-------------^----++ | >>> | | | | | | | | >>> | | | +------------+ | | +-----------+ | | | >>> | | | | COLO | | (socket) | | COLO | | | | >>> | | | | CheckPoint +---------------------> CheckPoint| | | | >>> | | | | | | (6) | | | | | | >>> | | | +-----^------+ | | +-----------+ | | | >>> | | | (5) | | | | | | >>> | | | | | | | | | >>> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >>> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >>> | | +-----+------+ | | +-----------------+ | | >>> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >>> | +-------------------+ | Forward(socket) | +-------------------+ | >>> ++Qemu+-----------------+ ++Qemu+-----------------+ >>> | ^ >>> | | >>> | | >>> +--------v-+--------+ >>> | | >>> | Client | >>> | | >>> +-------------------+ >>> >>> >>> >>> >>> (1)When PN receive client packets,PN COLO-Proxy copy and forward >>> packets to >>> SN COLO-Proxy. >>> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send >>> adjusted packets to SVM >>> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >>> COLO-Proxy. >>> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then >>> compare PVM's packets data with SVM's packets data. If packets is >>> different, compare >>> module notify COLO CheckPoint module to do a checkpoint then send >>> PVM's packets to >>> client and drop SVM's packets, otherwise, just send PVM's packets to >>> client and >>> drop SVM's packets. >>> (5)notify COLO-Checkpoint module checkpoint is needed >>> (6)Do COLO-Checkpoint >>> >>> ### QEMU space TCP/IP stack(Based on SLIRP) ### >>> We need a QEMU space TCP/IP stack to help us to analysis packet. After >>> looking >>> into QEMU, we found that SLIRP >>> >>> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 >>> >>> is a good choice for us. SLIRP proivdes a full TCP/IP stack within >>> QEMU, it can >>> help use to handle the packet written to/read from backend(tap) device >>> which is >>> just like a link layer(L2) packet. >>> >>> ### Packet enqueue and compare ### >>> Together with QEMU space TCP/IP stack, we enqueue all packets sent by >>> PVM and >>> SVM on Primary QEMU, and then compare the packet payload for each >>> connection. >>> >> Hi: >> >> Just have the following questions in my mind (some has been raised in >> the previous rounds of discussion without a conclusion): >> >> - What's the plan for management layer? The setup seems complicated so >> we could not simply depend on user to do each step. (And for security >> reason, qemu was usually run as unprivileged user) > It's certainly easier than the current COLO code that relies on a very > complex set of bridges, extra network interfaces and kernel modules. > UMU (cc'd) have been working on a libvirt set that starts COLO up, although > one bit that's very messy is the curretn kernel based network comparison > code. Ok. >> - What's the plan for vhost? Userspace network in qemu is rather slow, >> most user will choose vhost. >> - What if application generate packet based on hwrng device? This will >> produce always different packets. > Yes, there are cases this happens - COLO's worst case is similar to simple > checkpointing (because it has a limit to the smallest checkpoint), but it's > best case is much better, on a compute heavy load, it ends up taking > a checkpoint very rarely. > Actually the big problem is where randomness occurs in unexpected places, > e.g. where things like Perl's hash randomisation means that the two > hosts produce the same data in different orders. Not familiar with this, but unlike the hwrng, if the random data was computed by software, after a synchronization, it still has the possibility to produce the same result for a while. > >> - Not sure SLIRP is perfect matched for this task. As has been raised, >> another method is to decouple the packet comparing from qemu. In this >> way, lots of open source userspace stack could be used. >> - Haven't read the code of packet comparing, but if it needs to keep >> track the state of each connection, it could be easily DOS from guest. > The guest can only break it's own networking; so shooting itself in the foot > is no big deal. > > Dave The question is for the packet comparing, if the number of connections in guest exceed the maximum connections it could track, what will it do? > >> Thanks > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-11 3:09 ` Jason Wang @ 2015-11-11 9:03 ` Dr. David Alan Gilbert 0 siblings, 0 replies; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-11 9:03 UTC (permalink / raw) To: Jason Wang Cc: Tkid, lizhijian, jan.kiszka, eddie.dong, peter.huangpeng, qemu-devel, arei.gonglei, luis, stefanha, guijianfeng, zhang.zhanghailiang * Jason Wang (jasowang@redhat.com) wrote: > > > On 11/10/2015 05:41 PM, Dr. David Alan Gilbert wrote: > > * Jason Wang (jasowang@redhat.com) wrote: > >> > >> On 11/10/2015 01:26 PM, Tkid wrote: > >>> Hi,all > >>> > >>> We are planning to reimplement colo proxy in userspace (Here is in > >>> qemu) to > >>> cache and compare net packets.This module is one of the important > >>> components > >>> of COLO project and now it is still in early stage, so any comments and > >>> feedback are warmly welcomed,thanks in advance. > >>> > >>> ## Background > >>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > >>> Service) > >>> project is a high availability solution. Both Primary VM (PVM) and > >>> Secondary VM > >>> (SVM) run in parallel. They receive the same request from client, and > >>> generate > >>> responses in parallel too. If the response packets from PVM and SVM are > >>> identical, they are released immediately. Otherwise, a VM checkpoint > >>> (on demand) > >>> is conducted. > >>> Paper: > >>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > >>> COLO on Xen: > >>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > >>> COLO on Qemu/KVM: > >>> http://wiki.qemu.org/Features/COLO > >>> > >>> By the needs of capturing response packets from PVM and SVM and > >>> finding out > >>> whether they are identical, we introduce a new module to qemu > >>> networking called > >>> colo-proxy. > >>> > >>> This document describes the design of the colo-proxy module > >>> > >>> ## Glossary > >>> PVM - Primary VM, which provides services to clients. > >>> SVM - Secondary VM, a hot standby and replication of PVM. > >>> PN - Primary Node, the host which PVM runs on > >>> SN - Secondary Node, the host which SVM runs on > >>> > >>> ## Our Idea ## > >>> > >>> COLO-Proxy > >>> COLO-Proxy is a part of COLO,based on qemu net filter and it's a > >>> plugin for > >>> qemu net filter.the function keep SVM connect normal to PVM and compare > >>> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. > >>> > >>> == Workflow == > >>> > >>> > >>> +--+ +--+ > >>> |PN| |SN| > >>> +-----------------------+ +-----------------------+ > >>> | +-------------------+ | | +-------------------+ | > >>> | | | | | | | | > >>> | | PVM | | | | SVM | | > >>> | | | | | | | | > >>> | +--+-^--------------+ | | +-------------^----++ | > >>> | | | | | | | | > >>> | | | +------------+ | | +-----------+ | | | > >>> | | | | COLO | | (socket) | | COLO | | | | > >>> | | | | CheckPoint +---------------------> CheckPoint| | | | > >>> | | | | | | (6) | | | | | | > >>> | | | +-----^------+ | | +-----------+ | | | > >>> | | | (5) | | | | | | > >>> | | | | | | | | | > >>> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | > >>> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | > >>> | | +-----+------+ | | +-----------------+ | | > >>> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | > >>> | +-------------------+ | Forward(socket) | +-------------------+ | > >>> ++Qemu+-----------------+ ++Qemu+-----------------+ > >>> | ^ > >>> | | > >>> | | > >>> +--------v-+--------+ > >>> | | > >>> | Client | > >>> | | > >>> +-------------------+ > >>> > >>> > >>> > >>> > >>> (1)When PN receive client packets,PN COLO-Proxy copy and forward > >>> packets to > >>> SN COLO-Proxy. > >>> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send > >>> adjusted packets to SVM > >>> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu > >>> COLO-Proxy. > >>> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then > >>> compare PVM's packets data with SVM's packets data. If packets is > >>> different, compare > >>> module notify COLO CheckPoint module to do a checkpoint then send > >>> PVM's packets to > >>> client and drop SVM's packets, otherwise, just send PVM's packets to > >>> client and > >>> drop SVM's packets. > >>> (5)notify COLO-Checkpoint module checkpoint is needed > >>> (6)Do COLO-Checkpoint > >>> > >>> ### QEMU space TCP/IP stack(Based on SLIRP) ### > >>> We need a QEMU space TCP/IP stack to help us to analysis packet. After > >>> looking > >>> into QEMU, we found that SLIRP > >>> > >>> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 > >>> > >>> is a good choice for us. SLIRP proivdes a full TCP/IP stack within > >>> QEMU, it can > >>> help use to handle the packet written to/read from backend(tap) device > >>> which is > >>> just like a link layer(L2) packet. > >>> > >>> ### Packet enqueue and compare ### > >>> Together with QEMU space TCP/IP stack, we enqueue all packets sent by > >>> PVM and > >>> SVM on Primary QEMU, and then compare the packet payload for each > >>> connection. > >>> > >> Hi: > >> > >> Just have the following questions in my mind (some has been raised in > >> the previous rounds of discussion without a conclusion): > >> > >> - What's the plan for management layer? The setup seems complicated so > >> we could not simply depend on user to do each step. (And for security > >> reason, qemu was usually run as unprivileged user) > > It's certainly easier than the current COLO code that relies on a very > > complex set of bridges, extra network interfaces and kernel modules. > > UMU (cc'd) have been working on a libvirt set that starts COLO up, although > > one bit that's very messy is the curretn kernel based network comparison > > code. > > Ok. > > >> - What's the plan for vhost? Userspace network in qemu is rather slow, > >> most user will choose vhost. > >> - What if application generate packet based on hwrng device? This will > >> produce always different packets. > > Yes, there are cases this happens - COLO's worst case is similar to simple > > checkpointing (because it has a limit to the smallest checkpoint), but it's > > best case is much better, on a compute heavy load, it ends up taking > > a checkpoint very rarely. > > Actually the big problem is where randomness occurs in unexpected places, > > e.g. where things like Perl's hash randomisation means that the two > > hosts produce the same data in different orders. > > Not familiar with this, but unlike the hwrng, if the random data was > computed by software, after a synchronization, it still has the > possibility to produce the same result for a while. The hwrng, variation in tsc or anything else that feeds the entropy pool can do cause the divergence. > >> - Not sure SLIRP is perfect matched for this task. As has been raised, > >> another method is to decouple the packet comparing from qemu. In this > >> way, lots of open source userspace stack could be used. > >> - Haven't read the code of packet comparing, but if it needs to keep > >> track the state of each connection, it could be easily DOS from guest. > > The guest can only break it's own networking; so shooting itself in the foot > > is no big deal. > > > > Dave > > The question is for the packet comparing, if the number of connections > in guest exceed the maximum connections it could track, what will it do? The slow case choice is to just force simple checkpointing mode. However, I don't think this case is any different from the kernel's stateful connection tracking used in iptables for firewalling. Dave > > > > >> Thanks > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 7:35 ` Jason Wang ` (2 preceding siblings ...) 2015-11-10 9:41 ` Dr. David Alan Gilbert @ 2015-11-11 1:23 ` Dong, Eddie 2015-11-11 3:26 ` Jason Wang 3 siblings, 1 reply; 63+ messages in thread From: Dong, Eddie @ 2015-11-11 1:23 UTC (permalink / raw) To: Jason Wang, Tkid, qemu-devel@nongnu.org, stefanha@redhat.com Cc: zhang.zhanghailiang@huawei.com, lizhijian@cn.fujitsu.com, jan.kiszka@siemens.com, dgilbert@redhat.com, peter.huangpeng@huawei.com, arei.gonglei@huawei.com, guijianfeng@cn.fujitsu.com > - What's the plan for vhost? Userspace network in qemu is rather slow, most > user will choose vhost. [Dong, Eddie] Hi Jason: How about we take staging approach? In general, COLO opens a door of high performance HA solution, but it will take very long time to make everything perfect. As for the network virtualization, I think we may start from usage with moderate network bandwidth, like 1Gbps. Otherwise, the performance of COLO may be not that good (of course, like David mentioned, the worst case is same with periodic checkpoint). At the moment, how about we start from in Qemu virtio network, and enhance for vhost case in future? The good thing is that we get more people working on the patch series, and glad to see UMU also joined the effort. Thanks and welcome... Thx Eddie ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-11 1:23 ` Dong, Eddie @ 2015-11-11 3:26 ` Jason Wang 0 siblings, 0 replies; 63+ messages in thread From: Jason Wang @ 2015-11-11 3:26 UTC (permalink / raw) To: Dong, Eddie, Tkid, qemu-devel@nongnu.org, stefanha@redhat.com Cc: zhang.zhanghailiang@huawei.com, lizhijian@cn.fujitsu.com, jan.kiszka@siemens.com, peter.huangpeng@huawei.com, dgilbert@redhat.com, arei.gonglei@huawei.com, guijianfeng@cn.fujitsu.com On 11/11/2015 09:23 AM, Dong, Eddie wrote: >> - What's the plan for vhost? Userspace network in qemu is rather slow, most >> user will choose vhost. > [Dong, Eddie] Hi Jason: > How about we take staging approach? In general, COLO opens a door of high performance HA solution, but it will take very long time to make everything perfect. As for the network virtualization, I think we may start from usage with moderate network bandwidth, like 1Gbps. Otherwise, the performance of COLO may be not that good (of course, like David mentioned, the worst case is same with periodic checkpoint). At the moment, how about we start from in Qemu virtio network, and enhance for vhost case in future? Yes, of course and it makes sense. Mentioning vhost here is to avoid re-inventing or abandoning some exist infrastructures. For example, thinking how netfiler can work with vhost from the beginning does not harm ... > > The good thing is that we get more people working on the patch series, and glad to see UMU also joined the effort. Thanks and welcome... > > Thx Eddie Thanks ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 5:26 [Qemu-devel] [POC]colo-proxy in qemu Tkid 2015-11-10 7:35 ` Jason Wang @ 2015-11-10 10:54 ` Dr. David Alan Gilbert 2015-11-11 2:46 ` Zhang Chen 1 sibling, 1 reply; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-10 10:54 UTC (permalink / raw) To: Tkid Cc: zhang.zhanghailiang, lizhijian, jan.kiszka, jasowang, eddie.dong, qemu-devel, peter.huangpeng, arei.gonglei, stefanha, guijianfeng * Tkid (zhangchen.fnst@cn.fujitsu.com) wrote: > Hi,all > > We are planning to reimplement colo proxy in userspace (Here is in qemu) to > cache and compare net packets.This module is one of the important components > of COLO project and now it is still in early stage, so any comments and > feedback are warmly welcomed,thanks in advance. > > ## Background > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > Service) > project is a high availability solution. Both Primary VM (PVM) and Secondary > VM > (SVM) run in parallel. They receive the same request from client, and > generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint (on > demand) > is conducted. > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and finding out > whether they are identical, we introduce a new module to qemu networking > called > colo-proxy. > > This document describes the design of the colo-proxy module > > ## Glossary > PVM - Primary VM, which provides services to clients. > SVM - Secondary VM, a hot standby and replication of PVM. > PN - Primary Node, the host which PVM runs on > SN - Secondary Node, the host which SVM runs on > > ## Our Idea ## > > COLO-Proxy > COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for > qemu net filter.the function keep SVM connect normal to PVM and compare > PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. > > == Workflow == > > > +--+ +--+ > |PN| |SN| > +-----------------------+ +-----------------------+ > | +-------------------+ | | +-------------------+ | > | | | | | | | | > | | PVM | | | | SVM | | > | | | | | | | | > | +--+-^--------------+ | | +-------------^----++ | > | | | | | | | | > | | | +------------+ | | +-----------+ | | | > | | | | COLO | | (socket) | | COLO | | | | > | | | | CheckPoint +---------------------> CheckPoint| | | | > | | | | | | (6) | | | | | | > | | | +-----^------+ | | +-----------+ | | | > | | | (5) | | | | | | > | | | | | | | | | > | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | > | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | > | | +-----+------+ | | +-----------------+ | | > | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | > | +-------------------+ | Forward(socket) | +-------------------+ | > ++Qemu+-----------------+ ++Qemu+-----------------+ > | ^ > | | > | | > +--------v-+--------+ > | | > | Client | > | | > +-------------------+ > > > (1)When PN receive client packets,PN COLO-Proxy copy and forward packets to > SN COLO-Proxy. > (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send > adjusted packets to SVM > (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu > COLO-Proxy. What protocol are you using for the data carried over the Forward(socket)? I'm just wondering if there's an existing layer2 tunneling protocol that it would be best to use. > (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then > compare PVM's packets data with SVM's packets data. If packets is different, > compare > module notify COLO CheckPoint module to do a checkpoint then send PVM's > packets to > client and drop SVM's packets, otherwise, just send PVM's packets to client > and > drop SVM's packets. > (5)notify COLO-Checkpoint module checkpoint is needed > (6)Do COLO-Checkpoint > > ### QEMU space TCP/IP stack(Based on SLIRP) ### > We need a QEMU space TCP/IP stack to help us to analysis packet. After > looking > into QEMU, we found that SLIRP > > http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 > > is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it > can > help use to handle the packet written to/read from backend(tap) device which > is > just like a link layer(L2) packet. I still think SLIRP might be painful; but it might be an easy one to start with. > ### Packet enqueue and compare ### > Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM > and > SVM on Primary QEMU, and then compare the packet payload for each > connection. Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-10 10:54 ` Dr. David Alan Gilbert @ 2015-11-11 2:46 ` Zhang Chen 2015-11-13 12:33 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 63+ messages in thread From: Zhang Chen @ 2015-11-11 2:46 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: zhang.zhanghailiang, lizhijian, jan.kiszka, jasowang, eddie.dong, qemu-devel, peter.huangpeng, arei.gonglei, stefanha, guijianfeng On 11/10/2015 06:54 PM, Dr. David Alan Gilbert wrote: > * Tkid (zhangchen.fnst@cn.fujitsu.com) wrote: >> Hi,all >> >> We are planning to reimplement colo proxy in userspace (Here is in qemu) to >> cache and compare net packets.This module is one of the important components >> of COLO project and now it is still in early stage, so any comments and >> feedback are warmly welcomed,thanks in advance. >> >> ## Background >> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >> Service) >> project is a high availability solution. Both Primary VM (PVM) and Secondary >> VM >> (SVM) run in parallel. They receive the same request from client, and >> generate >> responses in parallel too. If the response packets from PVM and SVM are >> identical, they are released immediately. Otherwise, a VM checkpoint (on >> demand) >> is conducted. >> Paper: >> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >> COLO on Xen: >> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >> COLO on Qemu/KVM: >> http://wiki.qemu.org/Features/COLO >> >> By the needs of capturing response packets from PVM and SVM and finding out >> whether they are identical, we introduce a new module to qemu networking >> called >> colo-proxy. >> >> This document describes the design of the colo-proxy module >> >> ## Glossary >> PVM - Primary VM, which provides services to clients. >> SVM - Secondary VM, a hot standby and replication of PVM. >> PN - Primary Node, the host which PVM runs on >> SN - Secondary Node, the host which SVM runs on >> >> ## Our Idea ## >> >> COLO-Proxy >> COLO-Proxy is a part of COLO,based on qemu net filter and it's a plugin for >> qemu net filter.the function keep SVM connect normal to PVM and compare >> PVM's packets to SVM's packets.if difference,notify COLO do checkpoint. >> >> == Workflow == >> >> >> +--+ +--+ >> |PN| |SN| >> +-----------------------+ +-----------------------+ >> | +-------------------+ | | +-------------------+ | >> | | | | | | | | >> | | PVM | | | | SVM | | >> | | | | | | | | >> | +--+-^--------------+ | | +-------------^----++ | >> | | | | | | | | >> | | | +------------+ | | +-----------+ | | | >> | | | | COLO | | (socket) | | COLO | | | | >> | | | | CheckPoint +---------------------> CheckPoint| | | | >> | | | | | | (6) | | | | | | >> | | | +-----^------+ | | +-----------+ | | | >> | | | (5) | | | | | | >> | | | | | | | | | >> | +--v-+--------------+ | Forward(socket) | +-------------+----v+ | >> | |COLO Proxy | +-------+(1)+--------->seq&ack adjust(2)| | | >> | | +-----+------+ | | +-----------------+ | | >> | | | Compare(4) <-------+(3)+---------+ COLO Proxy | | >> | +-------------------+ | Forward(socket) | +-------------------+ | >> ++Qemu+-----------------+ ++Qemu+-----------------+ >> | ^ >> | | >> | | >> +--------v-+--------+ >> | | >> | Client | >> | | >> +-------------------+ >> >> >> (1)When PN receive client packets,PN COLO-Proxy copy and forward packets to >> SN COLO-Proxy. >> (2)SN COLO-Proxy record PVM's packet inital seq & adjust client's ack,send >> adjusted packets to SVM >> (3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu >> COLO-Proxy. > What protocol are you using for the data carried over the Forward(socket)? > I'm just wondering if there's an existing layer2 tunneling protocol that > it would be best to use. Currently,we will use raw tcp socket send a packet's length then send the packet. >> (4)PN Qemu COLO-Proxy enqueue SVM's packets and enqueue PVM's packets,then >> compare PVM's packets data with SVM's packets data. If packets is different, >> compare >> module notify COLO CheckPoint module to do a checkpoint then send PVM's >> packets to >> client and drop SVM's packets, otherwise, just send PVM's packets to client >> and >> drop SVM's packets. >> (5)notify COLO-Checkpoint module checkpoint is needed >> (6)Do COLO-Checkpoint >> >> ### QEMU space TCP/IP stack(Based on SLIRP) ### >> We need a QEMU space TCP/IP stack to help us to analysis packet. After >> looking >> into QEMU, we found that SLIRP >> >> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29 >> >> is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it >> can >> help use to handle the packet written to/read from backend(tap) device which >> is >> just like a link layer(L2) packet. > I still think SLIRP might be painful; but it might be an easy one to start > with. > >> ### Packet enqueue and compare ### >> Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM >> and >> SVM on Primary QEMU, and then compare the packet payload for each >> connection. > Dave > > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > . > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC]colo-proxy in qemu 2015-11-11 2:46 ` Zhang Chen @ 2015-11-13 12:33 ` Dr. David Alan Gilbert 0 siblings, 0 replies; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-11-13 12:33 UTC (permalink / raw) To: Zhang Chen Cc: zhang.zhanghailiang, lizhijian, jan.kiszka, jasowang, eddie.dong, qemu-devel, peter.huangpeng, arei.gonglei, stefanha, guijianfeng * Zhang Chen (zhangchen.fnst@cn.fujitsu.com) wrote: > > > On 11/10/2015 06:54 PM, Dr. David Alan Gilbert wrote: > >* Tkid (zhangchen.fnst@cn.fujitsu.com) wrote: > >>(3)SN Qemu COLO-Proxy recieve SVM's packets and forward to PN Qemu > >>COLO-Proxy. > >What protocol are you using for the data carried over the Forward(socket)? > >I'm just wondering if there's an existing layer2 tunneling protocol that > >it would be best to use. > Currently,we will use raw tcp socket send a packet's length then send the > packet. Yes OK (I think there's a qemu -net socket option that does something similar). Dave > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* [Qemu-devel] [POC] colo-proxy in qemu
@ 2015-07-20 6:42 Li Zhijian
2015-07-20 10:32 ` Stefan Hajnoczi
` (2 more replies)
0 siblings, 3 replies; 63+ messages in thread
From: Li Zhijian @ 2015-07-20 6:42 UTC (permalink / raw)
To: qemu-devel, stefanha, jasowang
Cc: zhanghailiang, jan.kiszka, peter.huangpeng, dgilbert,
Gonglei (Arei), Yang Hongyang
Hi, all
We are planning to implement colo-proxy in qemu to cache and compare packets.
This module is one of the important component of COLO project and now it is
still in early stage, so any comments and feedback are warmly welcomed,
thanks in advance.
## Background
COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
is conducted.
Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO
By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking called
colo-proxy.
This document describes the design of the colo-proxy module
## Glossary
PVM - Primary VM, which provides services to clients.
SVM - Secondary VM, a hot standby and replication of PVM.
PN - Primary Node, the host which PVM runs on
SN - Secondary Node, the host which SVM runs on
## Workflow ##
The following image show the qemu networking packet datapath between
guest's NIC and qemu's backend in colo-proxy.
+---+ +---+
|PN | |SN |
+---+--------------------------+ +------------------------------+
| +-------+ | | +-------+ |
+--------+ |chkpoint<--------[socket]------->chkpoint +--------+
|PVM | +---^---+ | | +---+---+ |SVM |
| | +proxy--v--------+ | | | | |
| | | | | | | | |
| +---+ | | +TCP/IP stack+ | | | +-----v-------proxy | +---+ |
+-|NIC|--+ | | | | | | | | +-|NIC|--+
| +^-++ | | +--------+ | | | | | +TCP/IP stack-+ | +^--+ |
| | +------> | | compare| | <-[socket]-forward- | +--------+ | | | |
| | | | +---+----+ | | | | | | |seq&ack | | <----+ |
| | | +-----|------+ | | | | | |adjust | | | |
| | | | | | | | | +--------+ | | |
| +-----------<+>-----copy&forward-[socket]---> +-------------+ | |
| +---|---|--------+ | | +------------^----+ |
| | | | | | |
| | | | | x |
| +--+---v----+ | | +-v---------+ |
| QEMU | backend | | | QEMU | backend | |
+------------+ (tap) +-----+ +------------+ (tap) +-----+
+-----------+ +-----------+
## Our Idea ##
### Net filter
In current QEMU, a packet is transported between networking backend(tap) and
qemu network adapter(NIC) directly. Backend and adapter is linked by
NetClientState->peer in qemu as following
+----------------------------------------+
v |
+NetClientState+ +------->+NetClientState+ |
|info->type=TAP| | |info->type=NIC| |
+--------------+ | +--------------+ |
| *peer +---+ | *peer +----+
+--------------+ +--------------+
|name="tap0" | |name="e1000" |
+--------------+ +--------------+
| ... | | ... |
+--------------+ +--------------+
In COLO QEMU, we insert a net filter named colo-proxy between backend and
adapter like below:
typedef struct COLOState {
NetClientState nc;
NetClientState *peer;
} COLOState;
+------->+NetClientState+ +NetClientState+<--------+
| |info->type=TAP| |info->type=NIC| |
| +--------------+ +--------------+ |
+-----------+ *peer | | *peer +------------+
| | +--------------+ +--------------+ | |
| | |name="tap0" | |name="e1000" | | |
| | +--------------+ +--------------+ | |
| | | ... | | ... | | |
| | +--------------+ +--------------+ | |
| | | |
| | +-COLOState------------+ +-COLOState------------+ | |
+--------->+NetClientState+<- - + +---->+NetClientState+<---------+
| | |info->type=COLO | | | | |info->type=COLO | |
| | +--------------+ | | | | +--------------+ | |
+-------+ *peer | | | | | | *peer +-------+
| +--------------+ | | | | +--------------+ |
| |name="colo1" | | | | | |name="colo2" | |
| +--------------+ | | | | +--------------+ |
| | | | | |
| +--------------+---------+ | +--------------+ |
| | *peer | | | | | *peer | |
| +--------------+ | +---------+--------------+ |
+----------------------+ +----------------------+
After we insert colo-proxy filter, all packets will pass by this filter
and more important thing is that we can analysis packet by ourselves.
### QEMU space TCP/IP stack(re-use SLIRP) ###
We need a QEMU space TCP/IP stack to help us to analysis packet. After looking
into QEMU, we found that SLIRP
http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it can
help use to handle the packet written to/read from backend(tap) device which is
just like a link layer(L2) packet.
### packet enqueue and compare ###
Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM and
SVM on Primary QEMU, and then compare the packet payload for each connection.
### Net filter Usage ###
On both Primary/Secondary host, invoke QEMU with the following parameters to insert
a net filter(colo-proxy):
"-netdev tap,id=hn0 -device e1000,netdev=hn0 \
-netdev colo,id=colo,backend=hn0"
^ permalink raw reply [flat|nested] 63+ messages in thread* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 6:42 [Qemu-devel] [POC] colo-proxy " Li Zhijian @ 2015-07-20 10:32 ` Stefan Hajnoczi 2015-07-20 11:55 ` zhanghailiang ` (2 more replies) 2015-07-24 2:05 ` Dong, Eddie 2015-07-30 4:23 ` Jason Wang 2 siblings, 3 replies; 63+ messages in thread From: Stefan Hajnoczi @ 2015-07-20 10:32 UTC (permalink / raw) To: Li Zhijian Cc: zhanghailiang, jan.kiszka, jasowang, peter.huangpeng, qemu-devel, Gonglei (Arei), stefanha, Yang Hongyang, dgilbert [-- Attachment #1: Type: text/plain, Size: 718 bytes --] On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: > We are planning to implement colo-proxy in qemu to cache and compare packets. I thought there is a kernel module to do that? Why does the proxy need to be part of the QEMU process? -netdev socket or host network stack features allow you to process packets in a separate process. Without details on what the proxy does it's hard to discuss this. What happens in the non-TCP case? What happens in the TCP case? Does the proxy need to perform privileged operations, create sockets, open files, etc? The slirp code is not actively developed or used much in production. It might be a good idea to audit the code for bugs if you want to use it. Stefan [-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 10:32 ` Stefan Hajnoczi @ 2015-07-20 11:55 ` zhanghailiang 2015-07-20 13:12 ` Vasiliy Tolstov 2015-07-20 12:02 ` Li Zhijian 2015-07-24 2:04 ` Dong, Eddie 2 siblings, 1 reply; 63+ messages in thread From: zhanghailiang @ 2015-07-20 11:55 UTC (permalink / raw) To: Stefan Hajnoczi, Li Zhijian Cc: qemu-devel, jan.kiszka, jasowang, dgilbert, peter.huangpeng, Gonglei (Arei), stefanha, Yang Hongyang On 2015/7/20 18:32, Stefan Hajnoczi wrote: > On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >> We are planning to implement colo-proxy in qemu to cache and compare packets. > > I thought there is a kernel module to do that? > Yes, but we decided to re-implement it in userspace (Here is in qemu), there are mainly two reasons that we made this change. One is the colo-proxy in kernel is narrowly used, which can only be used for COLO FT, besides, we have to modify iptables and nftables to support this capability. IMHO, it is hardly been accepted by the kernel community. The other reason is that the kernel proxy scenario can't been used in all situations, for example, evs + vhost-user + dpdk, it can't work if VM's network packets don't go through host's network stack. (For the new userspace colo proxy scheme, we also can't use it with vhost-net, we have to use virtio-net instead). > Why does the proxy need to be part of the QEMU process? -netdev socket > or host network stack features allow you to process packets in a > separate process. > > Without details on what the proxy does it's hard to discuss this. What > happens in the non-TCP case? What happens in the TCP case? > > Does the proxy need to perform privileged operations, create sockets, > open files, etc? > > The slirp code is not actively developed or used much in production. It > might be a good idea to audit the code for bugs if you want to use it. > Agreed, besides, it is seemed that slirp is not supporting ipv6, we also have to supplement it. Thanks, zhanghailiang ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 11:55 ` zhanghailiang @ 2015-07-20 13:12 ` Vasiliy Tolstov 2015-07-20 15:01 ` Stefan Hajnoczi 0 siblings, 1 reply; 63+ messages in thread From: Vasiliy Tolstov @ 2015-07-20 13:12 UTC (permalink / raw) To: zhanghailiang Cc: Li Zhijian, Stefan Hajnoczi, jasowang, dgilbert, qemu-devel, Gonglei (Arei), stefanha, jan.kiszka, peter.huangpeng, Yang Hongyang 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>: > Agreed, besides, it is seemed that slirp is not supporting ipv6, we also > have to supplement it. patch for ipv6 slirp support some times ago sended to qemu list, but i don't know why in not accepted. -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 13:12 ` Vasiliy Tolstov @ 2015-07-20 15:01 ` Stefan Hajnoczi 2015-07-21 1:59 ` zhanghailiang 2015-07-21 6:13 ` Jan Kiszka 0 siblings, 2 replies; 63+ messages in thread From: Stefan Hajnoczi @ 2015-07-20 15:01 UTC (permalink / raw) To: J. Kiszka Cc: zhanghailiang, Li Zhijian, Jason Wang, Dave Gilbert, Vasiliy Tolstov, qemu-devel, Gonglei (Arei), Stefan Hajnoczi, Huangpeng (Peter), Yang Hongyang On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote: > 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>: >> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also >> have to supplement it. > > > patch for ipv6 slirp support some times ago sended to qemu list, but i > don't know why in not accepted. I think no one reviewed it but there was no objection against IPv6 support in principle. Jan: Can we merge slirp IPv6 support for QEMU 2.5? Stefan ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 15:01 ` Stefan Hajnoczi @ 2015-07-21 1:59 ` zhanghailiang 2015-07-28 22:13 ` Samuel Thibault 2015-07-21 6:13 ` Jan Kiszka 1 sibling, 1 reply; 63+ messages in thread From: zhanghailiang @ 2015-07-21 1:59 UTC (permalink / raw) To: Stefan Hajnoczi, J. Kiszka Cc: Li Zhijian, qemu-devel, Jason Wang, Dave Gilbert, Vasiliy Tolstov, peter.huangpeng, Gonglei (Arei), Stefan Hajnoczi, samuel.thibault, Yang Hongyang On 2015/7/20 23:01, Stefan Hajnoczi wrote: > On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote: >> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>: >>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also >>> have to supplement it. >> >> >> patch for ipv6 slirp support some times ago sended to qemu list, but i >> don't know why in not accepted. > > I think no one reviewed it but there was no objection against IPv6 > support in principle. > > Jan: Can we merge slirp IPv6 support for QEMU 2.5? > I have found the corresponding patch series: https://lists.gnu.org/archive/html/qemu-devel/2014-02/msg01832.html Cc: Samuel Thibault <samuel.thibault@ens-lyon.org> Hi Samuel, What's the status of the 'slirp: Adding IPv6 support to Qemu -net use' series ? I didn't find any news since that version, are you still trying to push them to qemu upstream ? Thanks, zhanghailiang ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-21 1:59 ` zhanghailiang @ 2015-07-28 22:13 ` Samuel Thibault 0 siblings, 0 replies; 63+ messages in thread From: Samuel Thibault @ 2015-07-28 22:13 UTC (permalink / raw) To: zhanghailiang Cc: Li Zhijian, Stefan Hajnoczi, Jason Wang, qemu-devel, Vasiliy Tolstov, peter.huangpeng, Gonglei (Arei), Stefan Hajnoczi, J. Kiszka, Yang Hongyang, Dave Gilbert zhanghailiang, le Tue 21 Jul 2015 09:59:22 +0800, a écrit : > I didn't find any news since that version, are you still trying to > push them to qemu upstream ? I'd still be trying if I had any actual answer other than "we need to find time to deal about it" :) I can rebase the patch series over the current master and submit again the patches. Samuel ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 15:01 ` Stefan Hajnoczi 2015-07-21 1:59 ` zhanghailiang @ 2015-07-21 6:13 ` Jan Kiszka 2015-07-21 9:49 ` Stefan Hajnoczi 1 sibling, 1 reply; 63+ messages in thread From: Jan Kiszka @ 2015-07-21 6:13 UTC (permalink / raw) To: Stefan Hajnoczi Cc: zhanghailiang, Li Zhijian, Jason Wang, Dave Gilbert, Vasiliy Tolstov, qemu-devel, Gonglei (Arei), Stefan Hajnoczi, Huangpeng (Peter), Yang Hongyang On 2015-07-20 17:01, Stefan Hajnoczi wrote: > On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote: >> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>: >>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also >>> have to supplement it. >> >> >> patch for ipv6 slirp support some times ago sended to qemu list, but i >> don't know why in not accepted. > > I think no one reviewed it but there was no objection against IPv6 > support in principle. > > Jan: Can we merge slirp IPv6 support for QEMU 2.5? Sorry, as I pointed out some time back, I don't have the bandwidth to look into slirp. Someone need to do a review, then send a pull request. Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-21 6:13 ` Jan Kiszka @ 2015-07-21 9:49 ` Stefan Hajnoczi 2015-07-27 10:13 ` Stefan Hajnoczi 0 siblings, 1 reply; 63+ messages in thread From: Stefan Hajnoczi @ 2015-07-21 9:49 UTC (permalink / raw) To: Jan Kiszka Cc: zhanghailiang, Li Zhijian, Stefan Hajnoczi, Jason Wang, qemu-devel, Vasiliy Tolstov, Dave Gilbert, Gonglei (Arei), Huangpeng (Peter), Yang Hongyang [-- Attachment #1: Type: text/plain, Size: 1055 bytes --] On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote: > On 2015-07-20 17:01, Stefan Hajnoczi wrote: > > On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote: > >> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>: > >>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also > >>> have to supplement it. > >> > >> > >> patch for ipv6 slirp support some times ago sended to qemu list, but i > >> don't know why in not accepted. > > > > I think no one reviewed it but there was no objection against IPv6 > > support in principle. > > > > Jan: Can we merge slirp IPv6 support for QEMU 2.5? > > Sorry, as I pointed out some time back, I don't have the bandwidth to > look into slirp. Someone need to do a review, then send a pull request. Do you want to remove yourself from the slirp section of the MAINTAINERS file? Going forward we'll need to find someone familiar with the QEMU development process and with enough time to review slirp patches. Stefan [-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-21 9:49 ` Stefan Hajnoczi @ 2015-07-27 10:13 ` Stefan Hajnoczi 2015-07-27 11:24 ` zhanghailiang 2015-07-27 13:33 ` Jan Kiszka 0 siblings, 2 replies; 63+ messages in thread From: Stefan Hajnoczi @ 2015-07-27 10:13 UTC (permalink / raw) To: Stefan Hajnoczi Cc: zhanghailiang, Li Zhijian, Jan Kiszka, Jason Wang, qemu-devel, Vasiliy Tolstov, Dave Gilbert, Gonglei (Arei), Huangpeng (Peter), Yang Hongyang [-- Attachment #1: Type: text/plain, Size: 1324 bytes --] On Tue, Jul 21, 2015 at 10:49:29AM +0100, Stefan Hajnoczi wrote: > On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote: > > On 2015-07-20 17:01, Stefan Hajnoczi wrote: > > > On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote: > > >> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>: > > >>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also > > >>> have to supplement it. > > >> > > >> > > >> patch for ipv6 slirp support some times ago sended to qemu list, but i > > >> don't know why in not accepted. > > > > > > I think no one reviewed it but there was no objection against IPv6 > > > support in principle. > > > > > > Jan: Can we merge slirp IPv6 support for QEMU 2.5? > > > > Sorry, as I pointed out some time back, I don't have the bandwidth to > > look into slirp. Someone need to do a review, then send a pull request. > > Do you want to remove yourself from the slirp section of the MAINTAINERS > file? > > Going forward we'll need to find someone familiar with the QEMU > development process and with enough time to review slirp patches. Ping? I hoped this would raise some discussion and that maybe we could find a new maintainer or co-maintainer to get slirp moving. Any thoughts? Stefan [-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --] ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 10:13 ` Stefan Hajnoczi @ 2015-07-27 11:24 ` zhanghailiang 2015-07-27 11:31 ` Samuel Thibault 2015-07-27 13:33 ` Jan Kiszka 1 sibling, 1 reply; 63+ messages in thread From: zhanghailiang @ 2015-07-27 11:24 UTC (permalink / raw) To: Stefan Hajnoczi, Stefan Hajnoczi Cc: Li Zhijian, Jan Kiszka, Jason Wang, peter.huangpeng, Vasiliy Tolstov, qemu-devel, Gonglei (Arei), samuel.thibault, samuel.thibault, Yang Hongyang, Dave Gilbert On 2015/7/27 18:13, Stefan Hajnoczi wrote: > On Tue, Jul 21, 2015 at 10:49:29AM +0100, Stefan Hajnoczi wrote: >> On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote: >>> On 2015-07-20 17:01, Stefan Hajnoczi wrote: >>>> On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote: >>>>> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>: >>>>>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also >>>>>> have to supplement it. >>>>> >>>>> >>>>> patch for ipv6 slirp support some times ago sended to qemu list, but i >>>>> don't know why in not accepted. >>>> >>>> I think no one reviewed it but there was no objection against IPv6 >>>> support in principle. >>>> >>>> Jan: Can we merge slirp IPv6 support for QEMU 2.5? >>> >>> Sorry, as I pointed out some time back, I don't have the bandwidth to >>> look into slirp. Someone need to do a review, then send a pull request. >> >> Do you want to remove yourself from the slirp section of the MAINTAINERS >> file? >> >> Going forward we'll need to find someone familiar with the QEMU >> development process and with enough time to review slirp patches. > > Ping? > > I hoped this would raise some discussion and that maybe we could find a > new maintainer or co-maintainer to get slirp moving. > Yes, please, this is important, we need slirp's maintainer to help reviewing the COLO proxy patches that will be implemented based on slirp. (If we finally come to an agreement on realizing it in qemu) We also need to support ipv6 for slirp, i have emailed Samuel who have sent ipv6 patches for slirp before, but got no response. (I would like to respin and test Samuel's ipv6 slirp patch if he don't have time to do this, but firstly, i need to get his permission ：） ） Cc: Samuel Thibault <samuel.thibault@ens-lyon.org> <samuel.thibault@gnu.org>. Thanks, zhanghailiang > Any thoughts? > > Stefan > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 11:24 ` zhanghailiang @ 2015-07-27 11:31 ` Samuel Thibault 0 siblings, 0 replies; 63+ messages in thread From: Samuel Thibault @ 2015-07-27 11:31 UTC (permalink / raw) To: zhanghailiang Cc: Li Zhijian, Stefan Hajnoczi, Jason Wang, peter.huangpeng, Vasiliy Tolstov, qemu-devel, Gonglei (Arei), Stefan Hajnoczi, Jan Kiszka, Yang Hongyang, Dave Gilbert Hello, I'm just back from vacancy with no Internet access, so will answer shortly :) Samuel ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 10:13 ` Stefan Hajnoczi 2015-07-27 11:24 ` zhanghailiang @ 2015-07-27 13:33 ` Jan Kiszka 2015-07-28 22:12 ` Samuel Thibault 1 sibling, 1 reply; 63+ messages in thread From: Jan Kiszka @ 2015-07-27 13:33 UTC (permalink / raw) To: Stefan Hajnoczi, Stefan Hajnoczi, Samuel Thibault Cc: zhanghailiang, Li Zhijian, Jason Wang, Dave Gilbert, Vasiliy Tolstov, qemu-devel, Gonglei (Arei), Huangpeng (Peter), Yang Hongyang -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-07-27 12:13, Stefan Hajnoczi wrote: > On Tue, Jul 21, 2015 at 10:49:29AM +0100, Stefan Hajnoczi wrote: >> On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote: >>> On 2015-07-20 17:01, Stefan Hajnoczi wrote: >>>> On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov >>>> <v.tolstov@selfip.ru> wrote: >>>>> 2015-07-20 14:55 GMT+03:00 zhanghailiang >>>>> <zhang.zhanghailiang@huawei.com>: >>>>>> Agreed, besides, it is seemed that slirp is not >>>>>> supporting ipv6, we also have to supplement it. >>>>> >>>>> >>>>> patch for ipv6 slirp support some times ago sended to qemu >>>>> list, but i don't know why in not accepted. >>>> >>>> I think no one reviewed it but there was no objection against >>>> IPv6 support in principle. >>>> >>>> Jan: Can we merge slirp IPv6 support for QEMU 2.5? >>> >>> Sorry, as I pointed out some time back, I don't have the >>> bandwidth to look into slirp. Someone need to do a review, then >>> send a pull request. >> >> Do you want to remove yourself from the slirp section of the >> MAINTAINERS file? >> >> Going forward we'll need to find someone familiar with the QEMU >> development process and with enough time to review slirp >> patches. > > Ping? > > I hoped this would raise some discussion and that maybe we could > find a new maintainer or co-maintainer to get slirp moving. > > Any thoughts? Of course, I'm fine with handing this over to someone who'd like to pick up. Do we have volunteers? Samuel, would you like to do this? As a subsystem maintainer, you are already familiar with QEMU processes. Well, this still wouldn't resolve the independent review need for slirp-ipv6. Jan - -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlW2MycACgkQitSsb3rl5xSCaACePNubKPBkrdxQkcThUGD7w56B Q6oAoIgCzT9qVRzDf5IhY2eKFXgTZ+Ul =yk6R -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 13:33 ` Jan Kiszka @ 2015-07-28 22:12 ` Samuel Thibault 2015-07-29 7:36 ` Jan Kiszka 0 siblings, 1 reply; 63+ messages in thread From: Samuel Thibault @ 2015-07-28 22:12 UTC (permalink / raw) To: Jan Kiszka Cc: zhanghailiang, Li Zhijian, Stefan Hajnoczi, Jason Wang, qemu-devel, Vasiliy Tolstov, Dave Gilbert, Gonglei (Arei), Stefan Hajnoczi, Huangpeng (Peter), Yang Hongyang Hello, Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit : > Of course, I'm fine with handing this over to someone who'd like to > pick up. Do we have volunteers? > > Samuel, would you like to do this? As a subsystem maintainer, you are > already familiar with QEMU processes. I can help with maintenance, yes. > Well, this still wouldn't resolve the independent review need for > slirp-ipv6. Well, actually I didn't write slirp-ipv6, Guillaume Subiron did, and I reviewed it (and we iterated quite a bit) before we submit the patch series to qemu-devel. Samuel ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-28 22:12 ` Samuel Thibault @ 2015-07-29 7:36 ` Jan Kiszka 0 siblings, 0 replies; 63+ messages in thread From: Jan Kiszka @ 2015-07-29 7:36 UTC (permalink / raw) To: Samuel Thibault Cc: zhanghailiang, Li Zhijian, Stefan Hajnoczi, Jason Wang, qemu-devel, Vasiliy Tolstov, Dave Gilbert, Gonglei (Arei), Stefan Hajnoczi, Huangpeng (Peter), Yang Hongyang On 2015-07-29 00:12, Samuel Thibault wrote: > Hello, > > Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit : >> Of course, I'm fine with handing this over to someone who'd like to >> pick up. Do we have volunteers? >> >> Samuel, would you like to do this? As a subsystem maintainer, you are >> already familiar with QEMU processes. > > I can help with maintenance, yes. "Help with" will easily mean "be the one and only". ;) If you prefer, send a patch which only adds you as a maintainer, but I would also ack one that drops me from the list as well. > >> Well, this still wouldn't resolve the independent review need for >> slirp-ipv6. > > Well, actually I didn't write slirp-ipv6, Guillaume Subiron did, and I > reviewed it (and we iterated quite a bit) before we submit the patch > series to qemu-devel. Perfekt! Thanks, Jan -- Siemens AG, Corporate Technology, CT RTC ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 10:32 ` Stefan Hajnoczi 2015-07-20 11:55 ` zhanghailiang @ 2015-07-20 12:02 ` Li Zhijian 2015-07-24 2:04 ` Dong, Eddie 2 siblings, 0 replies; 63+ messages in thread From: Li Zhijian @ 2015-07-20 12:02 UTC (permalink / raw) To: Stefan Hajnoczi Cc: zhanghailiang, jan.kiszka, jasowang, peter.huangpeng, qemu-devel, Gonglei (Arei), stefanha, Yang Hongyang, dgilbert CC Wen Congyang On 07/20/2015 06:32 PM, Stefan Hajnoczi wrote: > On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >> We are planning to implement colo-proxy in qemu to cache and compare packets. > I thought there is a kernel module to do that? > > Why does the proxy need to be part of the QEMU process? -netdev socket > or host network stack features allow you to process packets in a > separate process. yes, it used to be a kernel module. we plan to re-implement a QEMU space colo-proxy by the following reasons: 1. colo-proxy in kernel was based on netfilter, it was impletmented by add a new nf_ct_ext_id, but this will touch the existed kernel code and we must re-build the kernel before we install the colo-proxy modules. For this reason, less people is like to test colo-proxy and it become harder to post to kenel 2. COLO is the only scene of colo-proxy in kernel 3. colo-proxy in kernel only works on the case where packet will deliver to kernel tcp/ip stack. COLO project is mainly including 3 components, COLO-Frame COLO-Block and COLO-Proxy. The first tow components is being post to QEMU, if we integrate proxy into QEMU, it will become convenienter to manage the whole COLO project. further more, COLO will become easier to configure without depending on kernel > Without details on what the proxy does it's hard to discuss this. What > happens in the non-TCP case? What happens in the TCP case? more details will be post soon > Does the proxy need to perform privileged operations, create sockets, > open files, etc? IMO, we just need to create a new socket like the migration socket to forward packet between PVM and SVM. Best regards. Li Zhijian > The slirp code is not actively developed or used much in production. It > might be a good idea to audit the code for bugs if you want to use it. > > Stefan ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 10:32 ` Stefan Hajnoczi 2015-07-20 11:55 ` zhanghailiang 2015-07-20 12:02 ` Li Zhijian @ 2015-07-24 2:04 ` Dong, Eddie 2015-07-24 2:12 ` Jason Wang 2 siblings, 1 reply; 63+ messages in thread From: Dong, Eddie @ 2015-07-24 2:04 UTC (permalink / raw) To: Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, jasowang@redhat.com, Dong, Eddie, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, Yang Hongyang, dgilbert@redhat.com Hi Stefan: Thanks for your comments! > > On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: > > We are planning to implement colo-proxy in qemu to cache and compare > packets. > > I thought there is a kernel module to do that? Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :) More comments are welcome. > > Why does the proxy need to be part of the QEMU process? -netdev socket or > host network stack features allow you to process packets in a separate process. Hailiang did a very good summary, and we don't need privilege ops so far. The main thing that motivated us to revisit is because the former kernel land driver would be a pure virtualization (and for high availability) component. It may have limited user only at very beginning and thus slower to be accepted. And, people typically like to put component to be in user land as if it can. In addition, as a pure virtualization feature, we guess people in Qemu mailing list may be much more interested to support all the VMMs, such as pure Qemu VMM, KVM, etc. Thx Eddie ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-24 2:04 ` Dong, Eddie @ 2015-07-24 2:12 ` Jason Wang 2015-07-24 8:04 ` Yang Hongyang 0 siblings, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-07-24 2:12 UTC (permalink / raw) To: Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, Yang Hongyang, dgilbert@redhat.com On 07/24/2015 10:04 AM, Dong, Eddie wrote: > Hi Stefan: > Thanks for your comments! > >> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>> We are planning to implement colo-proxy in qemu to cache and compare >> packets. >> >> I thought there is a kernel module to do that? > Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :) More comments are welcome. > Hi: Could you pls describe more details on this decision? What's the reason that you realize it was not the best choice? Thanks ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-24 2:12 ` Jason Wang @ 2015-07-24 8:04 ` Yang Hongyang 2015-07-27 3:24 ` Jason Wang 2015-07-27 10:40 ` Dr. David Alan Gilbert 0 siblings, 2 replies; 63+ messages in thread From: Yang Hongyang @ 2015-07-24 8:04 UTC (permalink / raw) To: Jason Wang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com Hi Jason, On 07/24/2015 10:12 AM, Jason Wang wrote: > > > On 07/24/2015 10:04 AM, Dong, Eddie wrote: >> Hi Stefan: >> Thanks for your comments! >> >>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>> We are planning to implement colo-proxy in qemu to cache and compare >>> packets. >>> >>> I thought there is a kernel module to do that? >> Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :) More comments are welcome. >> > > Hi: > > Could you pls describe more details on this decision? What's the reason > that you realize it was not the best choice? Below is my opinion: We realized that there're disadvantages do it in kernel spaces: 1. We need to recompile kernel: the colo-proxy kernel module is implemented as a nf conntrack extension. Adding a extension need to modify the extension struct in-kernel, so recompile kernel is needed. 2. We need to recompile iptables/nftables to use together with the colo-proxy kernel module. 3. Need to configure primary host to forward input packets to secondary as well as configure secondary to forward output packets to primary host, the network topology and configuration is too complex for a regular user. You can refer to http://wiki.qemu.org/Features/COLO to see the network topology and the steps to setup an env. Setup a test env is too complex. The usability is so important to a feature like COLO which provide VM FT solution, if fewer people can/willing to setup the env, the feature is useless. So we decide to develop user space colo-proxy. The advantage is obvious, 1. we do not need to recompile kernel. 2. No need to recompile iptables/nftables. 3. we do not need to deal with the network configuration, we just using a socket connection between 2 QEMUs to forward packets. 4. A complete VM FT solution in one go, we have already developed the block replication in QEMU, so with the network replication in QEMU, all components we needed are within QEMU, this is very important, it greatly improves the usability of COLO feature! We hope it will gain more testers, users and developers. 5. QEMU will gain a complete VM FT solution and the most advantage FT solution so far! Overall, usability is the most important factor that impact our choice. > > Thanks > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-24 8:04 ` Yang Hongyang @ 2015-07-27 3:24 ` Jason Wang 2015-07-27 3:54 ` Yang Hongyang 2015-07-27 10:40 ` Dr. David Alan Gilbert 1 sibling, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-07-27 3:24 UTC (permalink / raw) To: Yang Hongyang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, peter.huangpeng, qemu-devel@nongnu.org, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/24/2015 04:04 PM, Yang Hongyang wrote: > Hi Jason, > > On 07/24/2015 10:12 AM, Jason Wang wrote: >> >> >> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>> Hi Stefan: >>> Thanks for your comments! >>> >>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>> We are planning to implement colo-proxy in qemu to cache and compare >>>> packets. >>>> >>>> I thought there is a kernel module to do that? >>> Yes, that is the previous solution the COLO sub-community choose >>> to go, but we realized it might be not the best choices, and thus we >>> want to bring discussion back here :) More comments are welcome. >>> >> >> Hi: >> >> Could you pls describe more details on this decision? What's the reason >> that you realize it was not the best choice? > > Below is my opinion: > > We realized that there're disadvantages do it in kernel spaces: > 1. We need to recompile kernel: the colo-proxy kernel module is > implemented as a nf conntrack extension. Adding a extension need to > modify the extension struct in-kernel, so recompile kernel is needed. There's no need to do all in kernel, you can use a separate process to do the comparing and trigger the state sync through monitor. > 2. We need to recompile iptables/nftables to use together with the > colo-proxy > kernel module. > 3. Need to configure primary host to forward input packets to > secondary as > well as configure secondary to forward output packets to primary > host, the > network topology and configuration is too complex for a regular user. > You can use current kernel primitives to mirror the traffic of both PVM and SVM to another process without any modification of kernel. And qemu can offload all network configuration to management in this case. And what's more import, this works for vhost. Filtering in qemu won't work for vhost. > You can refer to http://wiki.qemu.org/Features/COLO > to see the network topology and the steps to setup an env. The figure "COLO Framework" shows there's a proxy kernel module in primary node but in secondary node this is done through a process? This will complicate the environment a bit more. > > Setup a test env is too complex. The usability is so important to a > feature > like COLO which provide VM FT solution, if fewer people can/willing to > setup the env, the feature is useless. So we decide to develop user space > colo-proxy. If the setup is too complex, need to consider to simplify or reuse codes and designs. Otherwise you probably introduce something new that needs fault tolerance. > > The advantage is obvious, > 1. we do not need to recompile kernel. > 2. No need to recompile iptables/nftables. As I descried above, looks like there's no need to modify kernel. > 3. we do not need to deal with the network configuration, we just using a > socket connection between 2 QEMUs to forward packets. All network configurations should be offloaded to management. And you still need a dedicated topology according to the wiki. > 4. A complete VM FT solution in one go, we have already developed the > block > replication in QEMU, so with the network replication in QEMU, all > components we needed are within QEMU, this is very important, it > greatly > improves the usability of COLO feature! We hope it will gain more > testers, > users and developers. Is your block solution works for vhost? > 5. QEMU will gain a complete VM FT solution and the most advantage FT > solution > so far! > > Overall, usability is the most important factor that impact our choice. > > Usability will be improved if you can use exist primitives and decouple unnecessary codes from qemu. Thanks >> >> Thanks >> . >> > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 3:24 ` Jason Wang @ 2015-07-27 3:54 ` Yang Hongyang 2015-07-27 4:49 ` Jason Wang 0 siblings, 1 reply; 63+ messages in thread From: Yang Hongyang @ 2015-07-27 3:54 UTC (permalink / raw) To: Jason Wang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 11:24 AM, Jason Wang wrote: > > > On 07/24/2015 04:04 PM, Yang Hongyang wrote: >> Hi Jason, >> >> On 07/24/2015 10:12 AM, Jason Wang wrote: >>> >>> >>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>> Hi Stefan: >>>> Thanks for your comments! >>>> >>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>> We are planning to implement colo-proxy in qemu to cache and compare >>>>> packets. >>>>> >>>>> I thought there is a kernel module to do that? >>>> Yes, that is the previous solution the COLO sub-community choose >>>> to go, but we realized it might be not the best choices, and thus we >>>> want to bring discussion back here :) More comments are welcome. >>>> >>> >>> Hi: >>> >>> Could you pls describe more details on this decision? What's the reason >>> that you realize it was not the best choice? >> >> Below is my opinion: >> >> We realized that there're disadvantages do it in kernel spaces: >> 1. We need to recompile kernel: the colo-proxy kernel module is >> implemented as a nf conntrack extension. Adding a extension need to >> modify the extension struct in-kernel, so recompile kernel is needed. > > There's no need to do all in kernel, you can use a separate process to > do the comparing and trigger the state sync through monitor. I don't get it, colo-proxy kernel module using a kthread do the comparing and trigger the state sync. We implemented it as a nf conntrack extension module, so we need to extend the extension struct in-kernel, although it just needs few lines changes to kernel, but a recompile of kernel is needed. Are you talking about not implement it as a nf conntrack extension? > >> 2. We need to recompile iptables/nftables to use together with the >> colo-proxy >> kernel module. >> 3. Need to configure primary host to forward input packets to >> secondary as >> well as configure secondary to forward output packets to primary >> host, the >> network topology and configuration is too complex for a regular user. >> > > You can use current kernel primitives to mirror the traffic of both PVM > and SVM to another process without any modification of kernel. And qemu > can offload all network configuration to management in this case. And > what's more import, this works for vhost. Filtering in qemu won't work > for vhost. We are using tc to mirror/forward packets now. Implement in QEMU do have some limits, but there're also limits in kernel, if the packet do not pass the host kernel TCP/IP stack, such as vhost-user. > > >> You can refer to http://wiki.qemu.org/Features/COLO >> to see the network topology and the steps to setup an env. > > The figure "COLO Framework" shows there's a proxy kernel module in > primary node but in secondary node this is done through a process? This > will complicate the environment a bit more. proxy kernel module also works for secondary node. > >> >> Setup a test env is too complex. The usability is so important to a >> feature >> like COLO which provide VM FT solution, if fewer people can/willing to >> setup the env, the feature is useless. So we decide to develop user space >> colo-proxy. > > If the setup is too complex, need to consider to simplify or reuse codes > and designs. Otherwise you probably introduce something new that needs > fault tolerance. > >> >> The advantage is obvious, >> 1. we do not need to recompile kernel. >> 2. No need to recompile iptables/nftables. > > As I descried above, looks like there's no need to modify kernel. > >> 3. we do not need to deal with the network configuration, we just using a >> socket connection between 2 QEMUs to forward packets. > > All network configurations should be offloaded to management. And you > still need a dedicated topology according to the wiki. > >> 4. A complete VM FT solution in one go, we have already developed the >> block >> replication in QEMU, so with the network replication in QEMU, all >> components we needed are within QEMU, this is very important, it >> greatly >> improves the usability of COLO feature! We hope it will gain more >> testers, >> users and developers. > > Is your block solution works for vhost? No, it can't works for vhost and dataplane, migration also won't work for dataplane IIRC. > >> 5. QEMU will gain a complete VM FT solution and the most advantage FT >> solution >> so far! >> >> Overall, usability is the most important factor that impact our choice. >> >> > > Usability will be improved if you can use exist primitives and decouple > unnecessary codes from qemu. > > Thanks > >>> >>> Thanks >>> . >>> >> > > > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 3:54 ` Yang Hongyang @ 2015-07-27 4:49 ` Jason Wang 2015-07-27 5:51 ` Yang Hongyang 0 siblings, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-07-27 4:49 UTC (permalink / raw) To: Yang Hongyang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 11:54 AM, Yang Hongyang wrote: > > > On 07/27/2015 11:24 AM, Jason Wang wrote: >> >> >> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>> Hi Jason, >>> >>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>> >>>> >>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>> Hi Stefan: >>>>> Thanks for your comments! >>>>> >>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>> compare >>>>>> packets. >>>>>> >>>>>> I thought there is a kernel module to do that? >>>>> Yes, that is the previous solution the COLO sub-community choose >>>>> to go, but we realized it might be not the best choices, and thus we >>>>> want to bring discussion back here :) More comments are welcome. >>>>> >>>> >>>> Hi: >>>> >>>> Could you pls describe more details on this decision? What's the >>>> reason >>>> that you realize it was not the best choice? >>> >>> Below is my opinion: >>> >>> We realized that there're disadvantages do it in kernel spaces: >>> 1. We need to recompile kernel: the colo-proxy kernel module is >>> implemented as a nf conntrack extension. Adding a extension need to >>> modify the extension struct in-kernel, so recompile kernel is >>> needed. >> >> There's no need to do all in kernel, you can use a separate process to >> do the comparing and trigger the state sync through monitor. > > I don't get it, colo-proxy kernel module using a kthread do the > comparing and > trigger the state sync. We implemented it as a nf conntrack extension > module, > so we need to extend the extension struct in-kernel, although it just > needs > few lines changes to kernel, but a recompile of kernel is needed. Are you > talking about not implement it as a nf conntrack extension? Yes, I mean implement the comparing in userspace but not in qemu. > >> >>> 2. We need to recompile iptables/nftables to use together with the >>> colo-proxy >>> kernel module. >>> 3. Need to configure primary host to forward input packets to >>> secondary as >>> well as configure secondary to forward output packets to primary >>> host, the >>> network topology and configuration is too complex for a regular >>> user. >>> >> >> You can use current kernel primitives to mirror the traffic of both PVM >> and SVM to another process without any modification of kernel. And qemu >> can offload all network configuration to management in this case. And >> what's more import, this works for vhost. Filtering in qemu won't work >> for vhost. > > We are using tc to mirror/forward packets now. Implement in QEMU do > have some > limits, but there're also limits in kernel, if the packet do not pass > the host kernel TCP/IP stack, such as vhost-user. But the limits are much less than userspace, no? For vhost-user, maybe we could extend the backed to mirror the traffic also. > >> >> >>> You can refer to http://wiki.qemu.org/Features/COLO >>> to see the network topology and the steps to setup an env. >> >> The figure "COLO Framework" shows there's a proxy kernel module in >> primary node but in secondary node this is done through a process? This >> will complicate the environment a bit more. > > proxy kernel module also works for secondary node. > >> >>> >>> Setup a test env is too complex. The usability is so important to a >>> feature >>> like COLO which provide VM FT solution, if fewer people can/willing to >>> setup the env, the feature is useless. So we decide to develop user >>> space >>> colo-proxy. >> >> If the setup is too complex, need to consider to simplify or reuse codes >> and designs. Otherwise you probably introduce something new that needs >> fault tolerance. >> >>> >>> The advantage is obvious, >>> 1. we do not need to recompile kernel. >>> 2. No need to recompile iptables/nftables. >> >> As I descried above, looks like there's no need to modify kernel. >> >>> 3. we do not need to deal with the network configuration, we just >>> using a >>> socket connection between 2 QEMUs to forward packets. >> >> All network configurations should be offloaded to management. And you >> still need a dedicated topology according to the wiki. >> >>> 4. A complete VM FT solution in one go, we have already developed the >>> block >>> replication in QEMU, so with the network replication in QEMU, all >>> components we needed are within QEMU, this is very important, it >>> greatly >>> improves the usability of COLO feature! We hope it will gain more >>> testers, >>> users and developers. >> >> Is your block solution works for vhost? > > No, it can't works for vhost and dataplane, migration also won't work > for dataplane IIRC. > >> >>> 5. QEMU will gain a complete VM FT solution and the most advantage FT >>> solution >>> so far! >>> >>> Overall, usability is the most important factor that impact our choice. >>> >>> >> >> Usability will be improved if you can use exist primitives and decouple >> unnecessary codes from qemu. >> >> Thanks >> >>>> >>>> Thanks >>>> . >>>> >>> >> >> >> . >> > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 4:49 ` Jason Wang @ 2015-07-27 5:51 ` Yang Hongyang 2015-07-27 7:37 ` Jason Wang 2015-07-27 7:53 ` Jason Wang 0 siblings, 2 replies; 63+ messages in thread From: Yang Hongyang @ 2015-07-27 5:51 UTC (permalink / raw) To: Jason Wang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 12:49 PM, Jason Wang wrote: > > > On 07/27/2015 11:54 AM, Yang Hongyang wrote: >> >> >> On 07/27/2015 11:24 AM, Jason Wang wrote: >>> >>> >>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>>> Hi Jason, >>>> >>>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>>> >>>>> >>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>>> Hi Stefan: >>>>>> Thanks for your comments! >>>>>> >>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>>> compare >>>>>>> packets. >>>>>>> >>>>>>> I thought there is a kernel module to do that? >>>>>> Yes, that is the previous solution the COLO sub-community choose >>>>>> to go, but we realized it might be not the best choices, and thus we >>>>>> want to bring discussion back here :) More comments are welcome. >>>>>> >>>>> >>>>> Hi: >>>>> >>>>> Could you pls describe more details on this decision? What's the >>>>> reason >>>>> that you realize it was not the best choice? >>>> >>>> Below is my opinion: >>>> >>>> We realized that there're disadvantages do it in kernel spaces: >>>> 1. We need to recompile kernel: the colo-proxy kernel module is >>>> implemented as a nf conntrack extension. Adding a extension need to >>>> modify the extension struct in-kernel, so recompile kernel is >>>> needed. >>> >>> There's no need to do all in kernel, you can use a separate process to >>> do the comparing and trigger the state sync through monitor. >> >> I don't get it, colo-proxy kernel module using a kthread do the >> comparing and >> trigger the state sync. We implemented it as a nf conntrack extension >> module, >> so we need to extend the extension struct in-kernel, although it just >> needs >> few lines changes to kernel, but a recompile of kernel is needed. Are you >> talking about not implement it as a nf conntrack extension? > > Yes, I mean implement the comparing in userspace but not in qemu. Yes, it is an alternative, that requires other components such as netfilter userspace tools, it will add the complexity I think, we wanted to implement a simple solution in QEMU. Another reason is that using other userspace tools will affect the performance, the context switch between kernel and userspace may be an overhead. > >> >>> >>>> 2. We need to recompile iptables/nftables to use together with the >>>> colo-proxy >>>> kernel module. >>>> 3. Need to configure primary host to forward input packets to >>>> secondary as >>>> well as configure secondary to forward output packets to primary >>>> host, the >>>> network topology and configuration is too complex for a regular >>>> user. >>>> >>> >>> You can use current kernel primitives to mirror the traffic of both PVM >>> and SVM to another process without any modification of kernel. And qemu >>> can offload all network configuration to management in this case. And >>> what's more import, this works for vhost. Filtering in qemu won't work >>> for vhost. >> >> We are using tc to mirror/forward packets now. Implement in QEMU do >> have some >> limits, but there're also limits in kernel, if the packet do not pass >> the host kernel TCP/IP stack, such as vhost-user. > > But the limits are much less than userspace, no? For vhost-user, maybe > we could extend the backed to mirror the traffic also. IMO the limits are more or less. Besides, for mirror/forward packets, using tc requires a separate physical nic or a vlan, the nic should not be used for other purpose. if we implement it in QEMU, using an socket connection to forward packets, we no longer need an separate nic, it will reduce the network topology complexity. > >> >>> >>> >>>> You can refer to http://wiki.qemu.org/Features/COLO >>>> to see the network topology and the steps to setup an env. >>> >>> The figure "COLO Framework" shows there's a proxy kernel module in >>> primary node but in secondary node this is done through a process? This >>> will complicate the environment a bit more. >> >> proxy kernel module also works for secondary node. >> >>> >>>> >>>> Setup a test env is too complex. The usability is so important to a >>>> feature >>>> like COLO which provide VM FT solution, if fewer people can/willing to >>>> setup the env, the feature is useless. So we decide to develop user >>>> space >>>> colo-proxy. >>> >>> If the setup is too complex, need to consider to simplify or reuse codes >>> and designs. Otherwise you probably introduce something new that needs >>> fault tolerance. >>> >>>> >>>> The advantage is obvious, >>>> 1. we do not need to recompile kernel. >>>> 2. No need to recompile iptables/nftables. >>> >>> As I descried above, looks like there's no need to modify kernel. >>> >>>> 3. we do not need to deal with the network configuration, we just >>>> using a >>>> socket connection between 2 QEMUs to forward packets. >>> >>> All network configurations should be offloaded to management. And you >>> still need a dedicated topology according to the wiki. >>> >>>> 4. A complete VM FT solution in one go, we have already developed the >>>> block >>>> replication in QEMU, so with the network replication in QEMU, all >>>> components we needed are within QEMU, this is very important, it >>>> greatly >>>> improves the usability of COLO feature! We hope it will gain more >>>> testers, >>>> users and developers. >>> >>> Is your block solution works for vhost? >> >> No, it can't works for vhost and dataplane, migration also won't work >> for dataplane IIRC. >> >>> >>>> 5. QEMU will gain a complete VM FT solution and the most advantage FT >>>> solution >>>> so far! >>>> >>>> Overall, usability is the most important factor that impact our choice. >>>> >>>> >>> >>> Usability will be improved if you can use exist primitives and decouple >>> unnecessary codes from qemu. >>> >>> Thanks >>> >>>>> >>>>> Thanks >>>>> . >>>>> >>>> >>> >>> >>> . >>> >> > > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 5:51 ` Yang Hongyang @ 2015-07-27 7:37 ` Jason Wang 2015-07-27 7:49 ` Yang Hongyang 2015-07-27 7:53 ` Jason Wang 1 sibling, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-07-27 7:37 UTC (permalink / raw) To: Yang Hongyang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 01:51 PM, Yang Hongyang wrote: > On 07/27/2015 12:49 PM, Jason Wang wrote: >> >> >> On 07/27/2015 11:54 AM, Yang Hongyang wrote: >>> >>> >>> On 07/27/2015 11:24 AM, Jason Wang wrote: >>>> >>>> >>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>>>> Hi Jason, >>>>> >>>>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>>>> >>>>>> >>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>>>> Hi Stefan: >>>>>>> Thanks for your comments! >>>>>>> >>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>>>> compare >>>>>>>> packets. >>>>>>>> >>>>>>>> I thought there is a kernel module to do that? >>>>>>> Yes, that is the previous solution the COLO sub-community >>>>>>> choose >>>>>>> to go, but we realized it might be not the best choices, and >>>>>>> thus we >>>>>>> want to bring discussion back here :) More comments are welcome. >>>>>>> >>>>>> >>>>>> Hi: >>>>>> >>>>>> Could you pls describe more details on this decision? What's the >>>>>> reason >>>>>> that you realize it was not the best choice? >>>>> >>>>> Below is my opinion: >>>>> >>>>> We realized that there're disadvantages do it in kernel spaces: >>>>> 1. We need to recompile kernel: the colo-proxy kernel module is >>>>> implemented as a nf conntrack extension. Adding a extension >>>>> need to >>>>> modify the extension struct in-kernel, so recompile kernel is >>>>> needed. >>>> >>>> There's no need to do all in kernel, you can use a separate process to >>>> do the comparing and trigger the state sync through monitor. >>> >>> I don't get it, colo-proxy kernel module using a kthread do the >>> comparing and >>> trigger the state sync. We implemented it as a nf conntrack extension >>> module, >>> so we need to extend the extension struct in-kernel, although it just >>> needs >>> few lines changes to kernel, but a recompile of kernel is needed. >>> Are you >>> talking about not implement it as a nf conntrack extension? >> >> Yes, I mean implement the comparing in userspace but not in qemu. > > Yes, it is an alternative, that requires other components such as > netfilter userspace tools, it will add the complexity I think, we > wanted to implement a simple solution in QEMU. Another reason is > that using other userspace tools will affect the performance, the > context switch between kernel and userspace may be an overhead. > >> >>> >>>> >>>>> 2. We need to recompile iptables/nftables to use together with the >>>>> colo-proxy >>>>> kernel module. >>>>> 3. Need to configure primary host to forward input packets to >>>>> secondary as >>>>> well as configure secondary to forward output packets to primary >>>>> host, the >>>>> network topology and configuration is too complex for a regular >>>>> user. >>>>> >>>> >>>> You can use current kernel primitives to mirror the traffic of both >>>> PVM >>>> and SVM to another process without any modification of kernel. And >>>> qemu >>>> can offload all network configuration to management in this case. And >>>> what's more import, this works for vhost. Filtering in qemu won't work >>>> for vhost. >>> >>> We are using tc to mirror/forward packets now. Implement in QEMU do >>> have some >>> limits, but there're also limits in kernel, if the packet do not pass >>> the host kernel TCP/IP stack, such as vhost-user. >> >> But the limits are much less than userspace, no? For vhost-user, maybe >> we could extend the backed to mirror the traffic also. > > IMO the limits are more or less. Besides, for mirror/forward packets, > using tc requires a separate physical nic or a vlan, the nic should not > be used for other purpose. if we implement it in QEMU, using an socket > connection to forward packets, we no longer need an separate nic, it will > reduce the network topology complexity. It depends on how do you design your user space. If you want using userspace to forward the packet, you can 1) use packet socket to capture all traffic on the tap that is used by VM 2) mirror the traffic to a new tap device, the user space can then read all traffic from this new tap device. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 7:37 ` Jason Wang @ 2015-07-27 7:49 ` Yang Hongyang 2015-07-27 8:06 ` Jason Wang 0 siblings, 1 reply; 63+ messages in thread From: Yang Hongyang @ 2015-07-27 7:49 UTC (permalink / raw) To: Jason Wang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 03:37 PM, Jason Wang wrote: > > > On 07/27/2015 01:51 PM, Yang Hongyang wrote: >> On 07/27/2015 12:49 PM, Jason Wang wrote: >>> >>> >>> On 07/27/2015 11:54 AM, Yang Hongyang wrote: >>>> >>>> >>>> On 07/27/2015 11:24 AM, Jason Wang wrote: >>>>> >>>>> >>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>>>>> Hi Jason, >>>>>> >>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>>>>> >>>>>>> >>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>>>>> Hi Stefan: >>>>>>>> Thanks for your comments! >>>>>>>> >>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>>>>> compare >>>>>>>>> packets. >>>>>>>>> >>>>>>>>> I thought there is a kernel module to do that? >>>>>>>> Yes, that is the previous solution the COLO sub-community >>>>>>>> choose >>>>>>>> to go, but we realized it might be not the best choices, and >>>>>>>> thus we >>>>>>>> want to bring discussion back here :) More comments are welcome. >>>>>>>> >>>>>>> >>>>>>> Hi: >>>>>>> >>>>>>> Could you pls describe more details on this decision? What's the >>>>>>> reason >>>>>>> that you realize it was not the best choice? >>>>>> >>>>>> Below is my opinion: >>>>>> >>>>>> We realized that there're disadvantages do it in kernel spaces: >>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is >>>>>> implemented as a nf conntrack extension. Adding a extension >>>>>> need to >>>>>> modify the extension struct in-kernel, so recompile kernel is >>>>>> needed. >>>>> >>>>> There's no need to do all in kernel, you can use a separate process to >>>>> do the comparing and trigger the state sync through monitor. >>>> >>>> I don't get it, colo-proxy kernel module using a kthread do the >>>> comparing and >>>> trigger the state sync. We implemented it as a nf conntrack extension >>>> module, >>>> so we need to extend the extension struct in-kernel, although it just >>>> needs >>>> few lines changes to kernel, but a recompile of kernel is needed. >>>> Are you >>>> talking about not implement it as a nf conntrack extension? >>> >>> Yes, I mean implement the comparing in userspace but not in qemu. >> >> Yes, it is an alternative, that requires other components such as >> netfilter userspace tools, it will add the complexity I think, we >> wanted to implement a simple solution in QEMU. Another reason is >> that using other userspace tools will affect the performance, the >> context switch between kernel and userspace may be an overhead. >> >>> >>>> >>>>> >>>>>> 2. We need to recompile iptables/nftables to use together with the >>>>>> colo-proxy >>>>>> kernel module. >>>>>> 3. Need to configure primary host to forward input packets to >>>>>> secondary as >>>>>> well as configure secondary to forward output packets to primary >>>>>> host, the >>>>>> network topology and configuration is too complex for a regular >>>>>> user. >>>>>> >>>>> >>>>> You can use current kernel primitives to mirror the traffic of both >>>>> PVM >>>>> and SVM to another process without any modification of kernel. And >>>>> qemu >>>>> can offload all network configuration to management in this case. And >>>>> what's more import, this works for vhost. Filtering in qemu won't work >>>>> for vhost. >>>> >>>> We are using tc to mirror/forward packets now. Implement in QEMU do >>>> have some >>>> limits, but there're also limits in kernel, if the packet do not pass >>>> the host kernel TCP/IP stack, such as vhost-user. >>> >>> But the limits are much less than userspace, no? For vhost-user, maybe >>> we could extend the backed to mirror the traffic also. >> >> IMO the limits are more or less. Besides, for mirror/forward packets, >> using tc requires a separate physical nic or a vlan, the nic should not >> be used for other purpose. if we implement it in QEMU, using an socket >> connection to forward packets, we no longer need an separate nic, it will >> reduce the network topology complexity. > > It depends on how do you design your user space. If you want using > userspace to forward the packet, you can 1) use packet socket to capture > all traffic on the tap that is used by VM 2) mirror the traffic to a new > tap device, the user space can then read all traffic from this new tap > device. Yes, but we can also do it in QEMU space, right? This will make life easier because we do all in one solution within QEMU. > > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 7:49 ` Yang Hongyang @ 2015-07-27 8:06 ` Jason Wang 2015-07-27 8:22 ` Yang Hongyang 0 siblings, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-07-27 8:06 UTC (permalink / raw) To: Yang Hongyang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, peter.huangpeng, qemu-devel@nongnu.org, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 03:49 PM, Yang Hongyang wrote: > On 07/27/2015 03:37 PM, Jason Wang wrote: >> >> >> On 07/27/2015 01:51 PM, Yang Hongyang wrote: >>> On 07/27/2015 12:49 PM, Jason Wang wrote: >>>> >>>> >>>> On 07/27/2015 11:54 AM, Yang Hongyang wrote: >>>>> >>>>> >>>>> On 07/27/2015 11:24 AM, Jason Wang wrote: >>>>>> >>>>>> >>>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>>>>>> Hi Jason, >>>>>>> >>>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>>>>>> Hi Stefan: >>>>>>>>> Thanks for your comments! >>>>>>>>> >>>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>>>>>> compare >>>>>>>>>> packets. >>>>>>>>>> >>>>>>>>>> I thought there is a kernel module to do that? >>>>>>>>> Yes, that is the previous solution the COLO sub-community >>>>>>>>> choose >>>>>>>>> to go, but we realized it might be not the best choices, and >>>>>>>>> thus we >>>>>>>>> want to bring discussion back here :) More comments are welcome. >>>>>>>>> >>>>>>>> >>>>>>>> Hi: >>>>>>>> >>>>>>>> Could you pls describe more details on this decision? What's the >>>>>>>> reason >>>>>>>> that you realize it was not the best choice? >>>>>>> >>>>>>> Below is my opinion: >>>>>>> >>>>>>> We realized that there're disadvantages do it in kernel spaces: >>>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is >>>>>>> implemented as a nf conntrack extension. Adding a extension >>>>>>> need to >>>>>>> modify the extension struct in-kernel, so recompile kernel is >>>>>>> needed. >>>>>> >>>>>> There's no need to do all in kernel, you can use a separate >>>>>> process to >>>>>> do the comparing and trigger the state sync through monitor. >>>>> >>>>> I don't get it, colo-proxy kernel module using a kthread do the >>>>> comparing and >>>>> trigger the state sync. We implemented it as a nf conntrack extension >>>>> module, >>>>> so we need to extend the extension struct in-kernel, although it just >>>>> needs >>>>> few lines changes to kernel, but a recompile of kernel is needed. >>>>> Are you >>>>> talking about not implement it as a nf conntrack extension? >>>> >>>> Yes, I mean implement the comparing in userspace but not in qemu. >>> >>> Yes, it is an alternative, that requires other components such as >>> netfilter userspace tools, it will add the complexity I think, we >>> wanted to implement a simple solution in QEMU. Another reason is >>> that using other userspace tools will affect the performance, the >>> context switch between kernel and userspace may be an overhead. >>> >>>> >>>>> >>>>>> >>>>>>> 2. We need to recompile iptables/nftables to use together with the >>>>>>> colo-proxy >>>>>>> kernel module. >>>>>>> 3. Need to configure primary host to forward input packets to >>>>>>> secondary as >>>>>>> well as configure secondary to forward output packets to >>>>>>> primary >>>>>>> host, the >>>>>>> network topology and configuration is too complex for a >>>>>>> regular >>>>>>> user. >>>>>>> >>>>>> >>>>>> You can use current kernel primitives to mirror the traffic of both >>>>>> PVM >>>>>> and SVM to another process without any modification of kernel. And >>>>>> qemu >>>>>> can offload all network configuration to management in this >>>>>> case. And >>>>>> what's more import, this works for vhost. Filtering in qemu won't >>>>>> work >>>>>> for vhost. >>>>> >>>>> We are using tc to mirror/forward packets now. Implement in QEMU do >>>>> have some >>>>> limits, but there're also limits in kernel, if the packet do not pass >>>>> the host kernel TCP/IP stack, such as vhost-user. >>>> >>>> But the limits are much less than userspace, no? For vhost-user, maybe >>>> we could extend the backed to mirror the traffic also. >>> >>> IMO the limits are more or less. Besides, for mirror/forward packets, >>> using tc requires a separate physical nic or a vlan, the nic should not >>> be used for other purpose. if we implement it in QEMU, using an socket >>> connection to forward packets, we no longer need an separate nic, it >>> will >>> reduce the network topology complexity. >> >> It depends on how do you design your user space. If you want using >> userspace to forward the packet, you can 1) use packet socket to capture >> all traffic on the tap that is used by VM 2) mirror the traffic to a new >> tap device, the user space can then read all traffic from this new tap >> device. > > Yes, but we can also do it in QEMU space, right? Right. > This will make life easier > because we do all in one solution within QEMU. But I'm not sure qemu is the right place to do this as you mention that it needs userspace protocol stack support. > >> >> . >> > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 8:06 ` Jason Wang @ 2015-07-27 8:22 ` Yang Hongyang 0 siblings, 0 replies; 63+ messages in thread From: Yang Hongyang @ 2015-07-27 8:22 UTC (permalink / raw) To: Jason Wang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, peter.huangpeng, qemu-devel@nongnu.org, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 04:06 PM, Jason Wang wrote: > > > On 07/27/2015 03:49 PM, Yang Hongyang wrote: >> On 07/27/2015 03:37 PM, Jason Wang wrote: >>> >>> >>> On 07/27/2015 01:51 PM, Yang Hongyang wrote: >>>> On 07/27/2015 12:49 PM, Jason Wang wrote: >>>>> >>>>> >>>>> On 07/27/2015 11:54 AM, Yang Hongyang wrote: >>>>>> >>>>>> >>>>>> On 07/27/2015 11:24 AM, Jason Wang wrote: >>>>>>> >>>>>>> >>>>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>>>>>>> Hi Jason, >>>>>>>> >>>>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>>>>>>> Hi Stefan: >>>>>>>>>> Thanks for your comments! >>>>>>>>>> >>>>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>>>>>>> compare >>>>>>>>>>> packets. >>>>>>>>>>> >>>>>>>>>>> I thought there is a kernel module to do that? >>>>>>>>>> Yes, that is the previous solution the COLO sub-community >>>>>>>>>> choose >>>>>>>>>> to go, but we realized it might be not the best choices, and >>>>>>>>>> thus we >>>>>>>>>> want to bring discussion back here :) More comments are welcome. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Hi: >>>>>>>>> >>>>>>>>> Could you pls describe more details on this decision? What's the >>>>>>>>> reason >>>>>>>>> that you realize it was not the best choice? >>>>>>>> >>>>>>>> Below is my opinion: >>>>>>>> >>>>>>>> We realized that there're disadvantages do it in kernel spaces: >>>>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is >>>>>>>> implemented as a nf conntrack extension. Adding a extension >>>>>>>> need to >>>>>>>> modify the extension struct in-kernel, so recompile kernel is >>>>>>>> needed. >>>>>>> >>>>>>> There's no need to do all in kernel, you can use a separate >>>>>>> process to >>>>>>> do the comparing and trigger the state sync through monitor. >>>>>> >>>>>> I don't get it, colo-proxy kernel module using a kthread do the >>>>>> comparing and >>>>>> trigger the state sync. We implemented it as a nf conntrack extension >>>>>> module, >>>>>> so we need to extend the extension struct in-kernel, although it just >>>>>> needs >>>>>> few lines changes to kernel, but a recompile of kernel is needed. >>>>>> Are you >>>>>> talking about not implement it as a nf conntrack extension? >>>>> >>>>> Yes, I mean implement the comparing in userspace but not in qemu. >>>> >>>> Yes, it is an alternative, that requires other components such as >>>> netfilter userspace tools, it will add the complexity I think, we >>>> wanted to implement a simple solution in QEMU. Another reason is >>>> that using other userspace tools will affect the performance, the >>>> context switch between kernel and userspace may be an overhead. >>>> >>>>> >>>>>> >>>>>>> >>>>>>>> 2. We need to recompile iptables/nftables to use together with the >>>>>>>> colo-proxy >>>>>>>> kernel module. >>>>>>>> 3. Need to configure primary host to forward input packets to >>>>>>>> secondary as >>>>>>>> well as configure secondary to forward output packets to >>>>>>>> primary >>>>>>>> host, the >>>>>>>> network topology and configuration is too complex for a >>>>>>>> regular >>>>>>>> user. >>>>>>>> >>>>>>> >>>>>>> You can use current kernel primitives to mirror the traffic of both >>>>>>> PVM >>>>>>> and SVM to another process without any modification of kernel. And >>>>>>> qemu >>>>>>> can offload all network configuration to management in this >>>>>>> case. And >>>>>>> what's more import, this works for vhost. Filtering in qemu won't >>>>>>> work >>>>>>> for vhost. >>>>>> >>>>>> We are using tc to mirror/forward packets now. Implement in QEMU do >>>>>> have some >>>>>> limits, but there're also limits in kernel, if the packet do not pass >>>>>> the host kernel TCP/IP stack, such as vhost-user. >>>>> >>>>> But the limits are much less than userspace, no? For vhost-user, maybe >>>>> we could extend the backed to mirror the traffic also. >>>> >>>> IMO the limits are more or less. Besides, for mirror/forward packets, >>>> using tc requires a separate physical nic or a vlan, the nic should not >>>> be used for other purpose. if we implement it in QEMU, using an socket >>>> connection to forward packets, we no longer need an separate nic, it >>>> will >>>> reduce the network topology complexity. >>> >>> It depends on how do you design your user space. If you want using >>> userspace to forward the packet, you can 1) use packet socket to capture >>> all traffic on the tap that is used by VM 2) mirror the traffic to a new >>> tap device, the user space can then read all traffic from this new tap >>> device. >> >> Yes, but we can also do it in QEMU space, right? > > Right. > >> This will make life easier >> because we do all in one solution within QEMU. > > But I'm not sure qemu is the right place to do this as you mention that > it needs userspace protocol stack support. We only need some simple features like defragment of TCP packets, analyze TCP headers, since QEMU has a slirp userspace protocol stack, that should not be a big deal. > >> >>> >>> . >>> >> > > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 5:51 ` Yang Hongyang 2015-07-27 7:37 ` Jason Wang @ 2015-07-27 7:53 ` Jason Wang 2015-07-27 8:17 ` Yang Hongyang 2015-07-27 18:33 ` Dr. David Alan Gilbert 1 sibling, 2 replies; 63+ messages in thread From: Jason Wang @ 2015-07-27 7:53 UTC (permalink / raw) To: Yang Hongyang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 01:51 PM, Yang Hongyang wrote: > On 07/27/2015 12:49 PM, Jason Wang wrote: >> >> >> On 07/27/2015 11:54 AM, Yang Hongyang wrote: >>> >>> >>> On 07/27/2015 11:24 AM, Jason Wang wrote: >>>> >>>> >>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>>>> Hi Jason, >>>>> >>>>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>>>> >>>>>> >>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>>>> Hi Stefan: >>>>>>> Thanks for your comments! >>>>>>> >>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>>>> compare >>>>>>>> packets. >>>>>>>> >>>>>>>> I thought there is a kernel module to do that? >>>>>>> Yes, that is the previous solution the COLO sub-community >>>>>>> choose >>>>>>> to go, but we realized it might be not the best choices, and >>>>>>> thus we >>>>>>> want to bring discussion back here :) More comments are welcome. >>>>>>> >>>>>> >>>>>> Hi: >>>>>> >>>>>> Could you pls describe more details on this decision? What's the >>>>>> reason >>>>>> that you realize it was not the best choice? >>>>> >>>>> Below is my opinion: >>>>> >>>>> We realized that there're disadvantages do it in kernel spaces: >>>>> 1. We need to recompile kernel: the colo-proxy kernel module is >>>>> implemented as a nf conntrack extension. Adding a extension >>>>> need to >>>>> modify the extension struct in-kernel, so recompile kernel is >>>>> needed. >>>> >>>> There's no need to do all in kernel, you can use a separate process to >>>> do the comparing and trigger the state sync through monitor. >>> >>> I don't get it, colo-proxy kernel module using a kthread do the >>> comparing and >>> trigger the state sync. We implemented it as a nf conntrack extension >>> module, >>> so we need to extend the extension struct in-kernel, although it just >>> needs >>> few lines changes to kernel, but a recompile of kernel is needed. >>> Are you >>> talking about not implement it as a nf conntrack extension? >> >> Yes, I mean implement the comparing in userspace but not in qemu. > > Yes, it is an alternative, that requires other components such as > netfilter userspace tools, it will add the complexity I think, we > wanted to implement a simple solution in QEMU. I didn't get the point that why netfilter is needed? Do you mean the packet comparing needs to be stateful? > Another reason is > that using other userspace tools will affect the performance, the > context switch between kernel and userspace may be an overhead. We can use 100% time of this process but looks like your RFC of filter just did it in iothread? ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 7:53 ` Jason Wang @ 2015-07-27 8:17 ` Yang Hongyang 2015-07-27 18:33 ` Dr. David Alan Gilbert 1 sibling, 0 replies; 63+ messages in thread From: Yang Hongyang @ 2015-07-27 8:17 UTC (permalink / raw) To: Jason Wang, Dong, Eddie, Stefan Hajnoczi, Li Zhijian Cc: zhanghailiang, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei (Arei), stefanha@redhat.com, dgilbert@redhat.com On 07/27/2015 03:53 PM, Jason Wang wrote: > > > On 07/27/2015 01:51 PM, Yang Hongyang wrote: >> On 07/27/2015 12:49 PM, Jason Wang wrote: >>> >>> >>> On 07/27/2015 11:54 AM, Yang Hongyang wrote: >>>> >>>> >>>> On 07/27/2015 11:24 AM, Jason Wang wrote: >>>>> >>>>> >>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: >>>>>> Hi Jason, >>>>>> >>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote: >>>>>>> >>>>>>> >>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>>>>>> Hi Stefan: >>>>>>>> Thanks for your comments! >>>>>>>> >>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and >>>>>>>>>> compare >>>>>>>>> packets. >>>>>>>>> >>>>>>>>> I thought there is a kernel module to do that? >>>>>>>> Yes, that is the previous solution the COLO sub-community >>>>>>>> choose >>>>>>>> to go, but we realized it might be not the best choices, and >>>>>>>> thus we >>>>>>>> want to bring discussion back here :) More comments are welcome. >>>>>>>> >>>>>>> >>>>>>> Hi: >>>>>>> >>>>>>> Could you pls describe more details on this decision? What's the >>>>>>> reason >>>>>>> that you realize it was not the best choice? >>>>>> >>>>>> Below is my opinion: >>>>>> >>>>>> We realized that there're disadvantages do it in kernel spaces: >>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is >>>>>> implemented as a nf conntrack extension. Adding a extension >>>>>> need to >>>>>> modify the extension struct in-kernel, so recompile kernel is >>>>>> needed. >>>>> >>>>> There's no need to do all in kernel, you can use a separate process to >>>>> do the comparing and trigger the state sync through monitor. >>>> >>>> I don't get it, colo-proxy kernel module using a kthread do the >>>> comparing and >>>> trigger the state sync. We implemented it as a nf conntrack extension >>>> module, >>>> so we need to extend the extension struct in-kernel, although it just >>>> needs >>>> few lines changes to kernel, but a recompile of kernel is needed. >>>> Are you >>>> talking about not implement it as a nf conntrack extension? >>> >>> Yes, I mean implement the comparing in userspace but not in qemu. >> >> Yes, it is an alternative, that requires other components such as >> netfilter userspace tools, it will add the complexity I think, we >> wanted to implement a simple solution in QEMU. > > I didn't get the point that why netfilter is needed? Do you mean the > packet comparing needs to be stateful? Yes. > >> Another reason is >> that using other userspace tools will affect the performance, the >> context switch between kernel and userspace may be an overhead. > > We can use 100% time of this process but looks like your RFC of filter > just did it in iothread? That's not the colo-proxy case, colo-proxy will require a separate thread to do the comparing work. > > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 7:53 ` Jason Wang 2015-07-27 8:17 ` Yang Hongyang @ 2015-07-27 18:33 ` Dr. David Alan Gilbert 1 sibling, 0 replies; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-07-27 18:33 UTC (permalink / raw) To: Jason Wang Cc: zhanghailiang, Li Zhijian, Stefan Hajnoczi, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei (Arei), stefanha@redhat.com, jan.kiszka@siemens.com, Yang Hongyang, dgilbert@redhat.com * Jason Wang (jasowang@redhat.com) wrote: > > > On 07/27/2015 01:51 PM, Yang Hongyang wrote: > > On 07/27/2015 12:49 PM, Jason Wang wrote: > >> > >> > >> On 07/27/2015 11:54 AM, Yang Hongyang wrote: > >>> > >>> > >>> On 07/27/2015 11:24 AM, Jason Wang wrote: > >>>> > >>>> > >>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote: > >>>>> Hi Jason, > >>>>> > >>>>> On 07/24/2015 10:12 AM, Jason Wang wrote: > >>>>>> > >>>>>> > >>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: > >>>>>>> Hi Stefan: > >>>>>>> Thanks for your comments! > >>>>>>> > >>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: > >>>>>>>>> We are planning to implement colo-proxy in qemu to cache and > >>>>>>>>> compare > >>>>>>>> packets. > >>>>>>>> > >>>>>>>> I thought there is a kernel module to do that? > >>>>>>> Yes, that is the previous solution the COLO sub-community > >>>>>>> choose > >>>>>>> to go, but we realized it might be not the best choices, and > >>>>>>> thus we > >>>>>>> want to bring discussion back here :) More comments are welcome. > >>>>>>> > >>>>>> > >>>>>> Hi: > >>>>>> > >>>>>> Could you pls describe more details on this decision? What's the > >>>>>> reason > >>>>>> that you realize it was not the best choice? > >>>>> > >>>>> Below is my opinion: > >>>>> > >>>>> We realized that there're disadvantages do it in kernel spaces: > >>>>> 1. We need to recompile kernel: the colo-proxy kernel module is > >>>>> implemented as a nf conntrack extension. Adding a extension > >>>>> need to > >>>>> modify the extension struct in-kernel, so recompile kernel is > >>>>> needed. > >>>> > >>>> There's no need to do all in kernel, you can use a separate process to > >>>> do the comparing and trigger the state sync through monitor. > >>> > >>> I don't get it, colo-proxy kernel module using a kthread do the > >>> comparing and > >>> trigger the state sync. We implemented it as a nf conntrack extension > >>> module, > >>> so we need to extend the extension struct in-kernel, although it just > >>> needs > >>> few lines changes to kernel, but a recompile of kernel is needed. > >>> Are you > >>> talking about not implement it as a nf conntrack extension? > >> > >> Yes, I mean implement the comparing in userspace but not in qemu. > > > > Yes, it is an alternative, that requires other components such as > > netfilter userspace tools, it will add the complexity I think, we > > wanted to implement a simple solution in QEMU. > > I didn't get the point that why netfilter is needed? Do you mean the > packet comparing needs to be stateful? The current kernel world does a few things that take advantage of the netfilter code: 1) It's stateful hanging state off conntrack 2) It modifies sequence numbers off the secondary to match what the primary did when it created the stream. 3) Comparison is on a per-stream basis so that the order of unrelated packets doesn't cause a miscompare. Dave > > Another reason is > > that using other userspace tools will affect the performance, the > > context switch between kernel and userspace may be an overhead. > > We can use 100% time of this process but looks like your RFC of filter > just did it in iothread? > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-24 8:04 ` Yang Hongyang 2015-07-27 3:24 ` Jason Wang @ 2015-07-27 10:40 ` Dr. David Alan Gilbert 2015-07-27 13:39 ` Yang Hongyang 1 sibling, 1 reply; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-07-27 10:40 UTC (permalink / raw) To: Yang Hongyang Cc: zhanghailiang, Li Zhijian, Stefan Hajnoczi, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei (Arei), stefanha@redhat.com, jan.kiszka@siemens.com * Yang Hongyang (yanghy@cn.fujitsu.com) wrote: > Hi Jason, > > On 07/24/2015 10:12 AM, Jason Wang wrote: > > > > > >On 07/24/2015 10:04 AM, Dong, Eddie wrote: > >>Hi Stefan: > >> Thanks for your comments! > >> > >>>On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: > >>>>We are planning to implement colo-proxy in qemu to cache and compare > >>>packets. > >>> > >>>I thought there is a kernel module to do that? > >> Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :) More comments are welcome. > >> > > > >Hi: > > > >Could you pls describe more details on this decision? What's the reason > >that you realize it was not the best choice? > > Below is my opinion: > > We realized that there're disadvantages do it in kernel spaces: > 1. We need to recompile kernel: the colo-proxy kernel module is > implemented as a nf conntrack extension. Adding a extension need to > modify the extension struct in-kernel, so recompile kernel is needed. That change is tiny though, so I don't think the change to the kernel is a big issue (but I'm not a netfilter guy). (For those following, the patch is: https://github.com/coloft/colo-proxy/blob/master/patch4kernel/0001-colo-patch-for-kernel.patch ) The comparison modules are bigger though, but still not massive. > 2. We need to recompile iptables/nftables to use together with the colo-proxy > kernel module. Again, the changes to iptables are small; so I don't think this should influence it too much. The bigger problem shown by 1&2 is that these changes are single-use - just for COLO, which does make it a little harder to justify. > 3. Need to configure primary host to forward input packets to secondary as > well as configure secondary to forward output packets to primary host, the > network topology and configuration is too complex for a regular user. Yes, and that bit is HARD - it took me quite a while to get it right; however, we'll still need to forward packets between primary and secondary, and all that hard setup should get rolled into something like libvirt, so perhaps it's not really that bad for the user in the end. > You can refer to http://wiki.qemu.org/Features/COLO > to see the network topology and the steps to setup an env. > > Setup a test env is too complex. The usability is so important to a feature > like COLO which provide VM FT solution, if fewer people can/willing to > setup the env, the feature is useless. So we decide to develop user space > colo-proxy. > > The advantage is obvious, > 1. we do not need to recompile kernel. > 2. No need to recompile iptables/nftables. > 3. we do not need to deal with the network configuration, we just using a > socket connection between 2 QEMUs to forward packets. > 4. A complete VM FT solution in one go, we have already developed the block > replication in QEMU, so with the network replication in QEMU, all > components we needed are within QEMU, this is very important, it greatly > improves the usability of COLO feature! We hope it will gain more testers, > users and developers. > 5. QEMU will gain a complete VM FT solution and the most advantage FT solution > so far! > > Overall, usability is the most important factor that impact our choice. My biggest worry is your reliance on SLIRP for the TCP/IP stack; it doesn't get much work done on it and I worry about it's reliability for using it for the level of complexity you need. Your current kernel implementation gets all the nf_conntrack stuff for free which is very powerful. However, I can see some advantages from doing it in user space; it would be easier to debug, and possibly easier to configure, and might also be easier to handle continuous FT (i.e. transferring the state of the proxy to a new COLO connection). I think at the moment I'd still prefer kernel space (especially since your kernel code now works pretty reliably!) Another thought; if you're main worry is to do with the complexity of kernel changes, had you considered looking at the bpf-jit - I'm not sure if it can do what you need, but perhaps it's worth a look? Dave P.S. I think 'proxy' is still the right word to describe it rather than 'agency'. > > > > > >Thanks > >. > > > > -- > Thanks, > Yang. -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-27 10:40 ` Dr. David Alan Gilbert @ 2015-07-27 13:39 ` Yang Hongyang 0 siblings, 0 replies; 63+ messages in thread From: Yang Hongyang @ 2015-07-27 13:39 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: zhanghailiang, Li Zhijian, Stefan Hajnoczi, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei (Arei), stefanha@redhat.com, jan.kiszka@siemens.com Hi Dave, Thanks for the comments! On 07/27/2015 06:40 PM, Dr. David Alan Gilbert wrote: > * Yang Hongyang (yanghy@cn.fujitsu.com) wrote: >> Hi Jason, >> >> On 07/24/2015 10:12 AM, Jason Wang wrote: >>> >>> >>> On 07/24/2015 10:04 AM, Dong, Eddie wrote: >>>> Hi Stefan: >>>> Thanks for your comments! >>>> >>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote: >>>>>> We are planning to implement colo-proxy in qemu to cache and compare >>>>> packets. >>>>> >>>>> I thought there is a kernel module to do that? >>>> Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :) More comments are welcome. >>>> >>> >>> Hi: >>> >>> Could you pls describe more details on this decision? What's the reason >>> that you realize it was not the best choice? >> >> Below is my opinion: >> >> We realized that there're disadvantages do it in kernel spaces: >> 1. We need to recompile kernel: the colo-proxy kernel module is >> implemented as a nf conntrack extension. Adding a extension need to >> modify the extension struct in-kernel, so recompile kernel is needed. > > That change is tiny though, so I don't think the change to the kernel > is a big issue (but I'm not a netfilter guy). > > (For those following, the patch is: > https://github.com/coloft/colo-proxy/blob/master/patch4kernel/0001-colo-patch-for-kernel.patch > ) > The comparison modules are bigger though, but still not massive. > >> 2. We need to recompile iptables/nftables to use together with the colo-proxy >> kernel module. > > Again, the changes to iptables are small; so I don't think this should > influence it too much. Yes, these changes are small, but even a small change needs to recompile the component and reinstall it, for user, it is not friendly... > > The bigger problem shown by 1&2 is that these changes are single-use - just for > COLO, which does make it a little harder to justify. That's true. > >> 3. Need to configure primary host to forward input packets to secondary as >> well as configure secondary to forward output packets to primary host, the >> network topology and configuration is too complex for a regular user. > > Yes, and that bit is HARD - it took me quite a while to get it right; however, > we'll still need to forward packets between primary and secondary, If we forward in qemu using a socket connection, a separate forward nic will not be needed, and all tc stuff will not needed, will make configuration easier I think. > and all that > hard setup should get rolled into something like libvirt, so perhaps it's not really > that bad for the user in the end. > >> You can refer to http://wiki.qemu.org/Features/COLO >> to see the network topology and the steps to setup an env. >> >> Setup a test env is too complex. The usability is so important to a feature >> like COLO which provide VM FT solution, if fewer people can/willing to >> setup the env, the feature is useless. So we decide to develop user space >> colo-proxy. >> >> The advantage is obvious, >> 1. we do not need to recompile kernel. >> 2. No need to recompile iptables/nftables. >> 3. we do not need to deal with the network configuration, we just using a >> socket connection between 2 QEMUs to forward packets. >> 4. A complete VM FT solution in one go, we have already developed the block >> replication in QEMU, so with the network replication in QEMU, all >> components we needed are within QEMU, this is very important, it greatly >> improves the usability of COLO feature! We hope it will gain more testers, >> users and developers. >> 5. QEMU will gain a complete VM FT solution and the most advantage FT solution >> so far! >> >> Overall, usability is the most important factor that impact our choice. > > My biggest worry is your reliance on SLIRP for the TCP/IP stack; it > doesn't get much work done on it and I worry about it's reliability for > using it for the level of complexity you need. > > Your current kernel implementation gets all the nf_conntrack stuff for free > which is very powerful. > > However, I can see some advantages from doing it in user space; it would > be easier to debug, and possibly easier to configure, and might also be easier > to handle continuous FT (i.e. transferring the state of the proxy to a new COLO > connection). > > I think at the moment I'd still prefer kernel space (especially since your kernel > code now works pretty reliably!) > > Another thought; if you're main worry is to do with the complexity of kernel > changes, had you considered looking at the bpf-jit - I'm not sure if it can > do what you need, but perhaps it's worth a look? Will have a look, thank you! > > Dave > P.S. I think 'proxy' is still the right word to describe it rather than 'agency'. > >> >> >>> >>> Thanks >>> . >>> >> >> -- >> Thanks, >> Yang. > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 6:42 [Qemu-devel] [POC] colo-proxy " Li Zhijian 2015-07-20 10:32 ` Stefan Hajnoczi @ 2015-07-24 2:05 ` Dong, Eddie 2015-07-30 4:23 ` Jason Wang 2 siblings, 0 replies; 63+ messages in thread From: Dong, Eddie @ 2015-07-24 2:05 UTC (permalink / raw) To: Li Zhijian, qemu-devel@nongnu.org, stefanha@redhat.com, jasowang@redhat.com Cc: zhanghailiang, jan.kiszka@siemens.com, Dong, Eddie, dgilbert@redhat.com, peter.huangpeng, Gonglei (Arei), Yang Hongyang BTW, I felt it is better to be called as an agency, rather than a proxy. Any comments from native speaker? Thx Eddie > Hi, all > > We are planning to implement colo-proxy in qemu to cache and compare > packets. > This module is one of the important component of COLO project and now it is > still in early stage, so any comments and feedback are warmly welcomed, thanks > in advance. > > ## Background > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service) > project is a high availability solution. Both Primary VM (PVM) and Secondary > VM > (SVM) run in parallel. They receive the same request from client, and generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint (on > demand) is conducted. > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and finding out > whether they are identical, we introduce a new module to qemu networking > called colo-proxy. > > This document describes the design of the colo-proxy module > > ## Glossary > PVM - Primary VM, which provides services to clients. > SVM - Secondary VM, a hot standby and replication of PVM. > PN - Primary Node, the host which PVM runs on > SN - Secondary Node, the host which SVM runs on > > ## Workflow ## > The following image show the qemu networking packet datapath between > guest's NIC and qemu's backend in colo-proxy. > > +---+ +---+ > |PN | |SN | > +---+--------------------------+ > +---+--------------------------+ +------------------------------+ > | +-------+ | | +-------+ | > +--------+ |chkpoint<--------[socket]------->chkpoint +--------+ > |PVM | +---^---+ | | +---+---+ |SVM | > | | +proxy--v--------+ | | | | | > | | | | | | | | | > | +---+ | | +TCP/IP stack+ | | | +-----v-------proxy | +---+ | > +-|NIC|--+ | | | | | | | | +-|NIC|--+ > | +^-++ | | +--------+ | | | | | +TCP/IP stack-+ | +^--+ | > | | +------> | | compare| | <-[socket]-forward- | +--------+ | | | | > | | | | +---+----+ | | | | | | |seq&ack | | <----+ | > | | | +-----|------+ | | | | | |adjust | | | | > | | | | | | | | | +--------+ | | | > | +-----------<+>-----copy&forward-[socket]---> +-------------+ | | > | +---|---|--------+ | | +------------^----+ | > | | | | | | | > | | | | | x | > | +--+---v----+ | | +-v---------+ | > | QEMU | backend | | | QEMU | backend | | > +------------+ (tap) +-----+ +------------+ (tap) +-----+ > +-----------+ +-----------+ > > ## Our Idea ## > > ### Net filter > In current QEMU, a packet is transported between networking backend(tap) > and qemu network adapter(NIC) directly. Backend and adapter is linked by > NetClientState->peer in qemu as following > +----------------------------------------+ > v | > +NetClientState+ +------->+NetClientState+ | > |info->type=TAP| | |info->type=NIC| | > +--------------+ | +--------------+ | > | *peer +---+ | *peer +----+ > +--------------+ +--------------+ > |name="tap0" | |name="e1000" | > +--------------+ +--------------+ > | ... | | ... | > +--------------+ +--------------+ > > In COLO QEMU, we insert a net filter named colo-proxy between backend and > adapter like below: > typedef struct COLOState { > NetClientState nc; > NetClientState *peer; > } COLOState; > +------->+NetClientState+ +NetClientState+<--------+ > | |info->type=TAP| |info->type=NIC| | > | +--------------+ +--------------+ | > +-----------+ *peer | | *peer +------------+ > | | +--------------+ +--------------+ | | > | | |name="tap0" | |name="e1000" | | | > | | +--------------+ +--------------+ | | > | | | ... | | ... | | | > | | +--------------+ +--------------+ | | > | | | | > | | +-COLOState------------+ +-COLOState------------+ | | > +--------->+NetClientState+<- - + +---->+NetClientState+<---------+ > | | |info->type=COLO | | | | |info->type=COLO | | > | | +--------------+ | | | | +--------------+ | | > +-------+ *peer | | | | | | *peer +-------+ > | +--------------+ | | | | +--------------+ | > | |name="colo1" | | | | | |name="colo2" | | > | +--------------+ | | | | +--------------+ | > | | | | | | > | +--------------+---------+ | +--------------+ | > | | *peer | | | | | *peer | | > | +--------------+ | +---------+--------------+ | > +----------------------+ +----------------------+ > > After we insert colo-proxy filter, all packets will pass by this filter and more > important thing is that we can analysis packet by ourselves. > > ### QEMU space TCP/IP stack(re-use SLIRP) ### We need a QEMU space TCP/IP > stack to help us to analysis packet. After looking into QEMU, we found that > SLIRP > http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP. > 29 > is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it can > help use to handle the packet written to/read from backend(tap) device which is > just like a link layer(L2) packet. > > ### packet enqueue and compare ### > Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM > and SVM on Primary QEMU, and then compare the packet payload for each > connection. > > ### Net filter Usage ### > On both Primary/Secondary host, invoke QEMU with the following parameters > to insert a net filter(colo-proxy): > "-netdev tap,id=hn0 -device e1000,netdev=hn0 \ > -netdev colo,id=colo,backend=hn0" > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-20 6:42 [Qemu-devel] [POC] colo-proxy " Li Zhijian 2015-07-20 10:32 ` Stefan Hajnoczi 2015-07-24 2:05 ` Dong, Eddie @ 2015-07-30 4:23 ` Jason Wang 2015-07-30 7:16 ` Gonglei 2 siblings, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-07-30 4:23 UTC (permalink / raw) To: Li Zhijian, qemu-devel, stefanha Cc: zhanghailiang, jan.kiszka, dgilbert, peter.huangpeng, Gonglei (Arei), Yang Hongyang On 07/20/2015 02:42 PM, Li Zhijian wrote: > Hi, all > > We are planning to implement colo-proxy in qemu to cache and compare > packets. > This module is one of the important component of COLO project and now > it is > still in early stage, so any comments and feedback are warmly welcomed, > thanks in advance. > > ## Background > COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop > Service) > project is a high availability solution. Both Primary VM (PVM) and > Secondary VM > (SVM) run in parallel. They receive the same request from client, and > generate > responses in parallel too. If the response packets from PVM and SVM are > identical, they are released immediately. Otherwise, a VM checkpoint > (on demand) > is conducted. > Paper: > http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 > COLO on Xen: > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping > COLO on Qemu/KVM: > http://wiki.qemu.org/Features/COLO > > By the needs of capturing response packets from PVM and SVM and > finding out > whether they are identical, we introduce a new module to qemu > networking called > colo-proxy. A question here, the packet comparing may be very tricky. For example, some protocol use random data to generate unpredictable id or something else. One example is ipv6_select_ident() in Linux. So COLO needs a mechanism to make sure PVM and SVM can generate same random data? ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 4:23 ` Jason Wang @ 2015-07-30 7:16 ` Gonglei 2015-07-30 7:47 ` Dong, Eddie 0 siblings, 1 reply; 63+ messages in thread From: Gonglei @ 2015-07-30 7:16 UTC (permalink / raw) To: Jason Wang, Li Zhijian, qemu-devel, stefanha Cc: jan.kiszka, Yang Hongyang, zhanghailiang, dgilbert, peter.huangpeng On 2015/7/30 12:23, Jason Wang wrote: > > > On 07/20/2015 02:42 PM, Li Zhijian wrote: >> Hi, all >> >> We are planning to implement colo-proxy in qemu to cache and compare >> packets. >> This module is one of the important component of COLO project and now >> it is >> still in early stage, so any comments and feedback are warmly welcomed, >> thanks in advance. >> >> ## Background >> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop >> Service) >> project is a high availability solution. Both Primary VM (PVM) and >> Secondary VM >> (SVM) run in parallel. They receive the same request from client, and >> generate >> responses in parallel too. If the response packets from PVM and SVM are >> identical, they are released immediately. Otherwise, a VM checkpoint >> (on demand) >> is conducted. >> Paper: >> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0 >> COLO on Xen: >> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping >> COLO on Qemu/KVM: >> http://wiki.qemu.org/Features/COLO >> >> By the needs of capturing response packets from PVM and SVM and >> finding out >> whether they are identical, we introduce a new module to qemu >> networking called >> colo-proxy. > > A question here, the packet comparing may be very tricky. For example, > some protocol use random data to generate unpredictable id or something > else. One example is ipv6_select_ident() in Linux. So COLO needs a > mechanism to make sure PVM and SVM can generate same random data? > Good question, the random data connection is a big problem for COLO. At present, it will trigger checkpoint processing because of the different random data. I don't think any mechanisms can assure two different machines generate the same random data. If you have any ideas, pls tell us :) Frequent checkpoint can handle this scenario, but maybe will cause the performance poor. :( Regards, -Gonglei ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 7:16 ` Gonglei @ 2015-07-30 7:47 ` Dong, Eddie 2015-07-30 8:03 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 63+ messages in thread From: Dong, Eddie @ 2015-07-30 7:47 UTC (permalink / raw) To: Gonglei, Jason Wang, Li Zhijian, qemu-devel@nongnu.org, stefanha@redhat.com Cc: zhanghailiang, jan.kiszka@siemens.com, Dong, Eddie, peter.huangpeng, dgilbert@redhat.com, Yang Hongyang > > > > A question here, the packet comparing may be very tricky. For example, > > some protocol use random data to generate unpredictable id or > > something else. One example is ipv6_select_ident() in Linux. So COLO > > needs a mechanism to make sure PVM and SVM can generate same random > data? > > > Good question, the random data connection is a big problem for COLO. At > present, it will trigger checkpoint processing because of the different random > data. > I don't think any mechanisms can assure two different machines generate the > same random data. If you have any ideas, pls tell us :) > > Frequent checkpoint can handle this scenario, but maybe will cause the > performance poor. :( > The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... Thx Eddie ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 7:47 ` Dong, Eddie @ 2015-07-30 8:03 ` Dr. David Alan Gilbert 2015-07-30 8:15 ` Jason Wang 0 siblings, 1 reply; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-07-30 8:03 UTC (permalink / raw) To: Dong, Eddie Cc: zhanghailiang, Li Zhijian, jan.kiszka@siemens.com, Jason Wang, qemu-devel@nongnu.org, peter.huangpeng, Gonglei, stefanha@redhat.com, Yang Hongyang * Dong, Eddie (eddie.dong@intel.com) wrote: > > > > > > A question here, the packet comparing may be very tricky. For example, > > > some protocol use random data to generate unpredictable id or > > > something else. One example is ipv6_select_ident() in Linux. So COLO > > > needs a mechanism to make sure PVM and SVM can generate same random > > data? > > > > > Good question, the random data connection is a big problem for COLO. At > > present, it will trigger checkpoint processing because of the different random > > data. > > I don't think any mechanisms can assure two different machines generate the > > same random data. If you have any ideas, pls tell us :) > > > > Frequent checkpoint can handle this scenario, but maybe will cause the > > performance poor. :( > > > The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... They do diverge pretty quickly though; I have simple examples which reliably cause a checkpoint because of simple randomness in applications. Dave > Thx Eddie > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 8:03 ` Dr. David Alan Gilbert @ 2015-07-30 8:15 ` Jason Wang 2015-07-30 11:56 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 63+ messages in thread From: Jason Wang @ 2015-07-30 8:15 UTC (permalink / raw) To: Dr. David Alan Gilbert, Dong, Eddie Cc: zhanghailiang, Li Zhijian, jan.kiszka@siemens.com, qemu-devel@nongnu.org, peter.huangpeng, Gonglei, stefanha@redhat.com, Yang Hongyang On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: > * Dong, Eddie (eddie.dong@intel.com) wrote: >>>> A question here, the packet comparing may be very tricky. For example, >>>> some protocol use random data to generate unpredictable id or >>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>> needs a mechanism to make sure PVM and SVM can generate same random >>> data? >>> Good question, the random data connection is a big problem for COLO. At >>> present, it will trigger checkpoint processing because of the different random >>> data. >>> I don't think any mechanisms can assure two different machines generate the >>> same random data. If you have any ideas, pls tell us :) >>> >>> Frequent checkpoint can handle this scenario, but maybe will cause the >>> performance poor. :( >>> >> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... > They do diverge pretty quickly though; I have simple examples which > reliably cause a checkpoint because of simple randomness in applications. > > Dave > And it will become even worse if hwrng is used in guest. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 8:15 ` Jason Wang @ 2015-07-30 11:56 ` Dr. David Alan Gilbert 2015-07-30 12:10 ` Gonglei 0 siblings, 1 reply; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-07-30 11:56 UTC (permalink / raw) To: Jason Wang Cc: zhanghailiang, Li Zhijian, jan.kiszka@siemens.com, Dong, Eddie, qemu-devel@nongnu.org, peter.huangpeng, Gonglei, stefanha@redhat.com, Yang Hongyang * Jason Wang (jasowang@redhat.com) wrote: > > > On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: > > * Dong, Eddie (eddie.dong@intel.com) wrote: > >>>> A question here, the packet comparing may be very tricky. For example, > >>>> some protocol use random data to generate unpredictable id or > >>>> something else. One example is ipv6_select_ident() in Linux. So COLO > >>>> needs a mechanism to make sure PVM and SVM can generate same random > >>> data? > >>> Good question, the random data connection is a big problem for COLO. At > >>> present, it will trigger checkpoint processing because of the different random > >>> data. > >>> I don't think any mechanisms can assure two different machines generate the > >>> same random data. If you have any ideas, pls tell us :) > >>> > >>> Frequent checkpoint can handle this scenario, but maybe will cause the > >>> performance poor. :( > >>> > >> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... > > They do diverge pretty quickly though; I have simple examples which > > reliably cause a checkpoint because of simple randomness in applications. > > > > Dave > > > > And it will become even worse if hwrng is used in guest. Yes; it seems quite application dependent; (on IPv4) an ssh connection, once established, tends to work well without triggering checkpoints; and static web pages also work well. Examples of things that do cause more checkpoints are, displaying guest statistics (e.g. running top in that ssh) which is timing dependent, and dynamically generated web pages that include a unique ID (bugzilla's password reset link in it's front page was a fun one), I think also establishing new encrypted connections cause the same randomness. However, it's worth remembering that COLO is trying to reduce the number of checkpoints compared to a simple checkpointing world which would be aiming to do a checkpoint ~100 times a second, and for compute bound workloads, or ones that don't expose the randomness that much, it can get checkpoints of a few seconds in length which greatly reduces the overhead. Dave -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 11:56 ` Dr. David Alan Gilbert @ 2015-07-30 12:10 ` Gonglei 2015-07-30 12:30 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 63+ messages in thread From: Gonglei @ 2015-07-30 12:10 UTC (permalink / raw) To: Dr. David Alan Gilbert, Jason Wang Cc: zhanghailiang, Li Zhijian, jan.kiszka@siemens.com, Dong, Eddie, qemu-devel@nongnu.org, peter.huangpeng, stefanha@redhat.com, Yang Hongyang On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: > * Jason Wang (jasowang@redhat.com) wrote: >> >> >> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: >>> * Dong, Eddie (eddie.dong@intel.com) wrote: >>>>>> A question here, the packet comparing may be very tricky. For example, >>>>>> some protocol use random data to generate unpredictable id or >>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>>>> needs a mechanism to make sure PVM and SVM can generate same random >>>>> data? >>>>> Good question, the random data connection is a big problem for COLO. At >>>>> present, it will trigger checkpoint processing because of the different random >>>>> data. >>>>> I don't think any mechanisms can assure two different machines generate the >>>>> same random data. If you have any ideas, pls tell us :) >>>>> >>>>> Frequent checkpoint can handle this scenario, but maybe will cause the >>>>> performance poor. :( >>>>> >>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... >>> They do diverge pretty quickly though; I have simple examples which >>> reliably cause a checkpoint because of simple randomness in applications. >>> >>> Dave >>> >> >> And it will become even worse if hwrng is used in guest. > > Yes; it seems quite application dependent; (on IPv4) an ssh connection, > once established, tends to work well without triggering checkpoints; > and static web pages also work well. Examples of things that do cause > more checkpoints are, displaying guest statistics (e.g. running top > in that ssh) which is timing dependent, and dynamically generated > web pages that include a unique ID (bugzilla's password reset link in > it's front page was a fun one), I think also establishing > new encrypted connections cause the same randomness. > > However, it's worth remembering that COLO is trying to reduce the > number of checkpoints compared to a simple checkpointing world > which would be aiming to do a checkpoint ~100 times a second, > and for compute bound workloads, or ones that don't expose > the randomness that much, it can get checkpoints of a few seconds > in length which greatly reduces the overhead. > Yes. That's the truth. We can set two different modes for different scenarios. Maybe Named 1) frequent checkpoint mode for multi-connections and randomness scenarios and 2) non-frequent checkpoint mode for other scenarios. But that's the next plan, we are thinking about that. Regards, -Gonglei ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 12:10 ` Gonglei @ 2015-07-30 12:30 ` Dr. David Alan Gilbert 2015-07-30 12:42 ` zhanghailiang 0 siblings, 1 reply; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-07-30 12:30 UTC (permalink / raw) To: Gonglei Cc: zhanghailiang, Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, qemu-devel@nongnu.org, peter.huangpeng, stefanha@redhat.com, Yang Hongyang * Gonglei (arei.gonglei@huawei.com) wrote: > On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: > > * Jason Wang (jasowang@redhat.com) wrote: > >> > >> > >> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: > >>> * Dong, Eddie (eddie.dong@intel.com) wrote: > >>>>>> A question here, the packet comparing may be very tricky. For example, > >>>>>> some protocol use random data to generate unpredictable id or > >>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO > >>>>>> needs a mechanism to make sure PVM and SVM can generate same random > >>>>> data? > >>>>> Good question, the random data connection is a big problem for COLO. At > >>>>> present, it will trigger checkpoint processing because of the different random > >>>>> data. > >>>>> I don't think any mechanisms can assure two different machines generate the > >>>>> same random data. If you have any ideas, pls tell us :) > >>>>> > >>>>> Frequent checkpoint can handle this scenario, but maybe will cause the > >>>>> performance poor. :( > >>>>> > >>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... > >>> They do diverge pretty quickly though; I have simple examples which > >>> reliably cause a checkpoint because of simple randomness in applications. > >>> > >>> Dave > >>> > >> > >> And it will become even worse if hwrng is used in guest. > > > > Yes; it seems quite application dependent; (on IPv4) an ssh connection, > > once established, tends to work well without triggering checkpoints; > > and static web pages also work well. Examples of things that do cause > > more checkpoints are, displaying guest statistics (e.g. running top > > in that ssh) which is timing dependent, and dynamically generated > > web pages that include a unique ID (bugzilla's password reset link in > > it's front page was a fun one), I think also establishing > > new encrypted connections cause the same randomness. > > > > However, it's worth remembering that COLO is trying to reduce the > > number of checkpoints compared to a simple checkpointing world > > which would be aiming to do a checkpoint ~100 times a second, > > and for compute bound workloads, or ones that don't expose > > the randomness that much, it can get checkpoints of a few seconds > > in length which greatly reduces the overhead. > > > > Yes. That's the truth. > We can set two different modes for different scenarios. Maybe Named > 1) frequent checkpoint mode for multi-connections and randomness scenarios > and 2) non-frequent checkpoint mode for other scenarios. > > But that's the next plan, we are thinking about that. I have some code that tries to automatically switch between those; it measures the checkpoint lengths, and if they're consistently short it sends a different message byte to the secondary at the start of the checkpoint, so that it doesn't bother running. Every so often it then flips back to a COLO checkpoint to see if the checkpoints are still really fast. Dave > > Regards, > -Gonglei > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 12:30 ` Dr. David Alan Gilbert @ 2015-07-30 12:42 ` zhanghailiang 2015-07-30 13:59 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 63+ messages in thread From: zhanghailiang @ 2015-07-30 12:42 UTC (permalink / raw) To: Dr. David Alan Gilbert, Gonglei Cc: Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, stefanha@redhat.com, Yang Hongyang On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: > * Gonglei (arei.gonglei@huawei.com) wrote: >> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: >>> * Jason Wang (jasowang@redhat.com) wrote: >>>> >>>> >>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: >>>>> * Dong, Eddie (eddie.dong@intel.com) wrote: >>>>>>>> A question here, the packet comparing may be very tricky. For example, >>>>>>>> some protocol use random data to generate unpredictable id or >>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random >>>>>>> data? >>>>>>> Good question, the random data connection is a big problem for COLO. At >>>>>>> present, it will trigger checkpoint processing because of the different random >>>>>>> data. >>>>>>> I don't think any mechanisms can assure two different machines generate the >>>>>>> same random data. If you have any ideas, pls tell us :) >>>>>>> >>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the >>>>>>> performance poor. :( >>>>>>> >>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... >>>>> They do diverge pretty quickly though; I have simple examples which >>>>> reliably cause a checkpoint because of simple randomness in applications. >>>>> >>>>> Dave >>>>> >>>> >>>> And it will become even worse if hwrng is used in guest. >>> >>> Yes; it seems quite application dependent; (on IPv4) an ssh connection, >>> once established, tends to work well without triggering checkpoints; >>> and static web pages also work well. Examples of things that do cause >>> more checkpoints are, displaying guest statistics (e.g. running top >>> in that ssh) which is timing dependent, and dynamically generated >>> web pages that include a unique ID (bugzilla's password reset link in >>> it's front page was a fun one), I think also establishing >>> new encrypted connections cause the same randomness. >>> >>> However, it's worth remembering that COLO is trying to reduce the >>> number of checkpoints compared to a simple checkpointing world >>> which would be aiming to do a checkpoint ~100 times a second, >>> and for compute bound workloads, or ones that don't expose >>> the randomness that much, it can get checkpoints of a few seconds >>> in length which greatly reduces the overhead. >>> >> >> Yes. That's the truth. >> We can set two different modes for different scenarios. Maybe Named >> 1) frequent checkpoint mode for multi-connections and randomness scenarios >> and 2) non-frequent checkpoint mode for other scenarios. >> >> But that's the next plan, we are thinking about that. > > I have some code that tries to automatically switch between those; > it measures the checkpoint lengths, and if they're consistently short > it sends a different message byte to the secondary at the start of the > checkpoint, so that it doesn't bother running. Every so often it > then flips back to a COLO checkpoint to see if the checkpoints > are still really fast. > Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM? Resume to common COLO mode until the checkpoint lengths is so not short ? Thanks. > Dave > >> >> Regards, >> -Gonglei >> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > . > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 12:42 ` zhanghailiang @ 2015-07-30 13:59 ` Dr. David Alan Gilbert 2015-07-30 15:17 ` Yang Hongyang 0 siblings, 1 reply; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-07-30 13:59 UTC (permalink / raw) To: zhanghailiang Cc: Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei, stefanha@redhat.com, Yang Hongyang * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: > On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: > >* Gonglei (arei.gonglei@huawei.com) wrote: > >>On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: > >>>* Jason Wang (jasowang@redhat.com) wrote: > >>>> > >>>> > >>>>On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: > >>>>>* Dong, Eddie (eddie.dong@intel.com) wrote: > >>>>>>>>A question here, the packet comparing may be very tricky. For example, > >>>>>>>>some protocol use random data to generate unpredictable id or > >>>>>>>>something else. One example is ipv6_select_ident() in Linux. So COLO > >>>>>>>>needs a mechanism to make sure PVM and SVM can generate same random > >>>>>>>data? > >>>>>>>Good question, the random data connection is a big problem for COLO. At > >>>>>>>present, it will trigger checkpoint processing because of the different random > >>>>>>>data. > >>>>>>>I don't think any mechanisms can assure two different machines generate the > >>>>>>>same random data. If you have any ideas, pls tell us :) > >>>>>>> > >>>>>>>Frequent checkpoint can handle this scenario, but maybe will cause the > >>>>>>>performance poor. :( > >>>>>>> > >>>>>>The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... > >>>>>They do diverge pretty quickly though; I have simple examples which > >>>>>reliably cause a checkpoint because of simple randomness in applications. > >>>>> > >>>>>Dave > >>>>> > >>>> > >>>>And it will become even worse if hwrng is used in guest. > >>> > >>>Yes; it seems quite application dependent; (on IPv4) an ssh connection, > >>>once established, tends to work well without triggering checkpoints; > >>>and static web pages also work well. Examples of things that do cause > >>>more checkpoints are, displaying guest statistics (e.g. running top > >>>in that ssh) which is timing dependent, and dynamically generated > >>>web pages that include a unique ID (bugzilla's password reset link in > >>>it's front page was a fun one), I think also establishing > >>>new encrypted connections cause the same randomness. > >>> > >>>However, it's worth remembering that COLO is trying to reduce the > >>>number of checkpoints compared to a simple checkpointing world > >>>which would be aiming to do a checkpoint ~100 times a second, > >>>and for compute bound workloads, or ones that don't expose > >>>the randomness that much, it can get checkpoints of a few seconds > >>>in length which greatly reduces the overhead. > >>> > >> > >>Yes. That's the truth. > >>We can set two different modes for different scenarios. Maybe Named > >>1) frequent checkpoint mode for multi-connections and randomness scenarios > >>and 2) non-frequent checkpoint mode for other scenarios. > >> > >>But that's the next plan, we are thinking about that. > > > >I have some code that tries to automatically switch between those; > >it measures the checkpoint lengths, and if they're consistently short > >it sends a different message byte to the secondary at the start of the > >checkpoint, so that it doesn't bother running. Every so often it > >then flips back to a COLO checkpoint to see if the checkpoints > >are still really fast. > > > > Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM? > Resume to common COLO mode until the checkpoint lengths is so not short ? We still have to do checkpoints, but we send a special message to the SVM so that the SVM just takes the checkpoint but does not run. I'll send the code after I've updated it to your current version; but it's quite rough/experimental. It works something like -----------run PVM run SVM COLO <long gap> mode miscompare checkpoint -----------run PVM run SVM COLO <short gap> mode miscompare checkpoint -----------run PVM run SVM COLO <short gap> mode miscompare < After a few short runs checkpoint -----------run PVM SVM idle \ Passive <fixed delay> | - repeat 'n' times mode checkpoint / -----------run PVM run SVM COLO <short gap> < Still a short gap mode miscompare -----------run PVM SVM idle \ Passive <fixed delay> | - repeat 'n' times mode checkpoint / -----------run PVM run SVM COLO <long gap> < long gap now, stay in COLO mode miscompare checkpoint -----------run PVM run SVM COLO <long gap> mode miscompare checkpoint So it saves the CPU time on the SVM, and the comparison traffic, and is automatic at switching into the passive mode. It used to be more useful, but your minimum COLO run time that you added a few versions ago helps a lot in the cases where there are miscompares, and the delay after the miscompare before you take the checkpoint also helps in the case where the data is very random. Dave > > Thanks. > > >Dave > > > >> > >>Regards, > >>-Gonglei > >> > >-- > >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > > >. > > > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 13:59 ` Dr. David Alan Gilbert @ 2015-07-30 15:17 ` Yang Hongyang 2015-07-30 17:53 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 63+ messages in thread From: Yang Hongyang @ 2015-07-30 15:17 UTC (permalink / raw) To: Dr. David Alan Gilbert, zhanghailiang Cc: Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei, stefanha@redhat.com On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote: > * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: >> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: >>> * Gonglei (arei.gonglei@huawei.com) wrote: >>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: >>>>> * Jason Wang (jasowang@redhat.com) wrote: >>>>>> >>>>>> >>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: >>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote: >>>>>>>>>> A question here, the packet comparing may be very tricky. For example, >>>>>>>>>> some protocol use random data to generate unpredictable id or >>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random >>>>>>>>> data? >>>>>>>>> Good question, the random data connection is a big problem for COLO. At >>>>>>>>> present, it will trigger checkpoint processing because of the different random >>>>>>>>> data. >>>>>>>>> I don't think any mechanisms can assure two different machines generate the >>>>>>>>> same random data. If you have any ideas, pls tell us :) >>>>>>>>> >>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the >>>>>>>>> performance poor. :( >>>>>>>>> >>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... >>>>>>> They do diverge pretty quickly though; I have simple examples which >>>>>>> reliably cause a checkpoint because of simple randomness in applications. >>>>>>> >>>>>>> Dave >>>>>>> >>>>>> >>>>>> And it will become even worse if hwrng is used in guest. >>>>> >>>>> Yes; it seems quite application dependent; (on IPv4) an ssh connection, >>>>> once established, tends to work well without triggering checkpoints; >>>>> and static web pages also work well. Examples of things that do cause >>>>> more checkpoints are, displaying guest statistics (e.g. running top >>>>> in that ssh) which is timing dependent, and dynamically generated >>>>> web pages that include a unique ID (bugzilla's password reset link in >>>>> it's front page was a fun one), I think also establishing >>>>> new encrypted connections cause the same randomness. >>>>> >>>>> However, it's worth remembering that COLO is trying to reduce the >>>>> number of checkpoints compared to a simple checkpointing world >>>>> which would be aiming to do a checkpoint ~100 times a second, >>>>> and for compute bound workloads, or ones that don't expose >>>>> the randomness that much, it can get checkpoints of a few seconds >>>>> in length which greatly reduces the overhead. >>>>> >>>> >>>> Yes. That's the truth. >>>> We can set two different modes for different scenarios. Maybe Named >>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios >>>> and 2) non-frequent checkpoint mode for other scenarios. >>>> >>>> But that's the next plan, we are thinking about that. >>> >>> I have some code that tries to automatically switch between those; >>> it measures the checkpoint lengths, and if they're consistently short >>> it sends a different message byte to the secondary at the start of the >>> checkpoint, so that it doesn't bother running. Every so often it >>> then flips back to a COLO checkpoint to see if the checkpoints >>> are still really fast. >>> >> >> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM? >> Resume to common COLO mode until the checkpoint lengths is so not short ? > > We still have to do checkpoints, but we send a special message to the SVM so that > the SVM just takes the checkpoint but does not run. > > I'll send the code after I've updated it to your current version; but it's > quite rough/experimental. > > It works something like > > -----------run PVM run SVM > COLO <long gap> > mode miscompare > checkpoint > -----------run PVM run SVM > COLO <short gap> > mode miscompare > checkpoint > -----------run PVM run SVM > COLO <short gap> > mode miscompare < After a few short runs > checkpoint > -----------run PVM SVM idle \ > Passive <fixed delay> | - repeat 'n' times > mode checkpoint / > -----------run PVM run SVM > COLO <short gap> < Still a short gap > mode miscompare > -----------run PVM SVM idle \ > Passive <fixed delay> | - repeat 'n' times > mode checkpoint / > -----------run PVM run SVM > COLO <long gap> < long gap now, stay in COLO > mode miscompare > checkpoint > -----------run PVM run SVM > COLO <long gap> > mode miscompare > checkpoint > > So it saves the CPU time on the SVM, and the comparison traffic, and is > automatic at switching into the passive mode. > > It used to be more useful, but your minimum COLO run time that you > added a few versions ago helps a lot in the cases where there are miscompares, > and the delay after the miscompare before you take the checkpoint also helps > in the case where the data is very random. This is great! This is exactly what we were thinking about, when random scenario will fallback to MC/Remus like FT. Thank you very much! I have a question, do you also modify colo-proxy kernel module? because in the fixed checkpoint mode, I think we need to buffer the network packets, and release them at checkpoint. > > Dave > >> >> Thanks. >> >>> Dave >>> >>>> >>>> Regards, >>>> -Gonglei >>>> >>> -- >>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>> >>> . >>> >> >> > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 15:17 ` Yang Hongyang @ 2015-07-30 17:53 ` Dr. David Alan Gilbert 2015-07-31 1:08 ` Yang Hongyang 2015-07-31 1:26 ` zhanghailiang 0 siblings, 2 replies; 63+ messages in thread From: Dr. David Alan Gilbert @ 2015-07-30 17:53 UTC (permalink / raw) To: Yang Hongyang Cc: zhanghailiang, Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei, stefanha@redhat.com * Yang Hongyang (yanghy@cn.fujitsu.com) wrote: > > > On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote: > >* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: > >>On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: > >>>* Gonglei (arei.gonglei@huawei.com) wrote: > >>>>On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: > >>>>>* Jason Wang (jasowang@redhat.com) wrote: > >>>>>> > >>>>>> > >>>>>>On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: > >>>>>>>* Dong, Eddie (eddie.dong@intel.com) wrote: > >>>>>>>>>>A question here, the packet comparing may be very tricky. For example, > >>>>>>>>>>some protocol use random data to generate unpredictable id or > >>>>>>>>>>something else. One example is ipv6_select_ident() in Linux. So COLO > >>>>>>>>>>needs a mechanism to make sure PVM and SVM can generate same random > >>>>>>>>>data? > >>>>>>>>>Good question, the random data connection is a big problem for COLO. At > >>>>>>>>>present, it will trigger checkpoint processing because of the different random > >>>>>>>>>data. > >>>>>>>>>I don't think any mechanisms can assure two different machines generate the > >>>>>>>>>same random data. If you have any ideas, pls tell us :) > >>>>>>>>> > >>>>>>>>>Frequent checkpoint can handle this scenario, but maybe will cause the > >>>>>>>>>performance poor. :( > >>>>>>>>> > >>>>>>>>The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... > >>>>>>>They do diverge pretty quickly though; I have simple examples which > >>>>>>>reliably cause a checkpoint because of simple randomness in applications. > >>>>>>> > >>>>>>>Dave > >>>>>>> > >>>>>> > >>>>>>And it will become even worse if hwrng is used in guest. > >>>>> > >>>>>Yes; it seems quite application dependent; (on IPv4) an ssh connection, > >>>>>once established, tends to work well without triggering checkpoints; > >>>>>and static web pages also work well. Examples of things that do cause > >>>>>more checkpoints are, displaying guest statistics (e.g. running top > >>>>>in that ssh) which is timing dependent, and dynamically generated > >>>>>web pages that include a unique ID (bugzilla's password reset link in > >>>>>it's front page was a fun one), I think also establishing > >>>>>new encrypted connections cause the same randomness. > >>>>> > >>>>>However, it's worth remembering that COLO is trying to reduce the > >>>>>number of checkpoints compared to a simple checkpointing world > >>>>>which would be aiming to do a checkpoint ~100 times a second, > >>>>>and for compute bound workloads, or ones that don't expose > >>>>>the randomness that much, it can get checkpoints of a few seconds > >>>>>in length which greatly reduces the overhead. > >>>>> > >>>> > >>>>Yes. That's the truth. > >>>>We can set two different modes for different scenarios. Maybe Named > >>>>1) frequent checkpoint mode for multi-connections and randomness scenarios > >>>>and 2) non-frequent checkpoint mode for other scenarios. > >>>> > >>>>But that's the next plan, we are thinking about that. > >>> > >>>I have some code that tries to automatically switch between those; > >>>it measures the checkpoint lengths, and if they're consistently short > >>>it sends a different message byte to the secondary at the start of the > >>>checkpoint, so that it doesn't bother running. Every so often it > >>>then flips back to a COLO checkpoint to see if the checkpoints > >>>are still really fast. > >>> > >> > >>Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM? > >>Resume to common COLO mode until the checkpoint lengths is so not short ? > > > > We still have to do checkpoints, but we send a special message to the SVM so that > >the SVM just takes the checkpoint but does not run. > > > > I'll send the code after I've updated it to your current version; but it's > >quite rough/experimental. > > > >It works something like > > > > -----------run PVM run SVM > > COLO <long gap> > > mode miscompare > > checkpoint > > -----------run PVM run SVM > > COLO <short gap> > > mode miscompare > > checkpoint > > -----------run PVM run SVM > > COLO <short gap> > > mode miscompare < After a few short runs > > checkpoint > > -----------run PVM SVM idle \ > > Passive <fixed delay> | - repeat 'n' times > > mode checkpoint / > > -----------run PVM run SVM > > COLO <short gap> < Still a short gap > > mode miscompare > > -----------run PVM SVM idle \ > > Passive <fixed delay> | - repeat 'n' times > > mode checkpoint / > > -----------run PVM run SVM > > COLO <long gap> < long gap now, stay in COLO > > mode miscompare > > checkpoint > > -----------run PVM run SVM > > COLO <long gap> > > mode miscompare > > checkpoint > > > >So it saves the CPU time on the SVM, and the comparison traffic, and is > >automatic at switching into the passive mode. > > > >It used to be more useful, but your minimum COLO run time that you > >added a few versions ago helps a lot in the cases where there are miscompares, > >and the delay after the miscompare before you take the checkpoint also helps > >in the case where the data is very random. > > This is great! This is exactly what we were thinking about, when random > scenario will fallback to MC/Remus like FT. Thank you very much! > I have a question, do you also modify colo-proxy kernel module? because > in the fixed checkpoint mode, I think we need to buffer the network > packets, and release them at checkpoint. Yes, we do need to buffer and release them at the end, but I've not modified colo-proxy so far. Doesn't the current code on PMY already need to buffer packets that are generated after the first miscompare and before the checkpoint and then release them at the checkpoint? Dave > > > > >Dave > > > >> > >>Thanks. > >> > >>>Dave > >>> > >>>> > >>>>Regards, > >>>>-Gonglei > >>>> > >>>-- > >>>Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >>> > >>>. > >>> > >> > >> > >-- > >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > >. > > > > -- > Thanks, > Yang. -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 17:53 ` Dr. David Alan Gilbert @ 2015-07-31 1:08 ` Yang Hongyang 2015-07-31 1:28 ` zhanghailiang 2015-07-31 1:26 ` zhanghailiang 1 sibling, 1 reply; 63+ messages in thread From: Yang Hongyang @ 2015-07-31 1:08 UTC (permalink / raw) To: Dr. David Alan Gilbert Cc: zhanghailiang, Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei, stefanha@redhat.com On 07/31/2015 01:53 AM, Dr. David Alan Gilbert wrote: > * Yang Hongyang (yanghy@cn.fujitsu.com) wrote: >> >> >> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote: >>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: >>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: >>>>> * Gonglei (arei.gonglei@huawei.com) wrote: >>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: >>>>>>> * Jason Wang (jasowang@redhat.com) wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: >>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote: >>>>>>>>>>>> A question here, the packet comparing may be very tricky. For example, >>>>>>>>>>>> some protocol use random data to generate unpredictable id or >>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random >>>>>>>>>>> data? >>>>>>>>>>> Good question, the random data connection is a big problem for COLO. At >>>>>>>>>>> present, it will trigger checkpoint processing because of the different random >>>>>>>>>>> data. >>>>>>>>>>> I don't think any mechanisms can assure two different machines generate the >>>>>>>>>>> same random data. If you have any ideas, pls tell us :) >>>>>>>>>>> >>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the >>>>>>>>>>> performance poor. :( >>>>>>>>>>> >>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... >>>>>>>>> They do diverge pretty quickly though; I have simple examples which >>>>>>>>> reliably cause a checkpoint because of simple randomness in applications. >>>>>>>>> >>>>>>>>> Dave >>>>>>>>> >>>>>>>> >>>>>>>> And it will become even worse if hwrng is used in guest. >>>>>>> >>>>>>> Yes; it seems quite application dependent; (on IPv4) an ssh connection, >>>>>>> once established, tends to work well without triggering checkpoints; >>>>>>> and static web pages also work well. Examples of things that do cause >>>>>>> more checkpoints are, displaying guest statistics (e.g. running top >>>>>>> in that ssh) which is timing dependent, and dynamically generated >>>>>>> web pages that include a unique ID (bugzilla's password reset link in >>>>>>> it's front page was a fun one), I think also establishing >>>>>>> new encrypted connections cause the same randomness. >>>>>>> >>>>>>> However, it's worth remembering that COLO is trying to reduce the >>>>>>> number of checkpoints compared to a simple checkpointing world >>>>>>> which would be aiming to do a checkpoint ~100 times a second, >>>>>>> and for compute bound workloads, or ones that don't expose >>>>>>> the randomness that much, it can get checkpoints of a few seconds >>>>>>> in length which greatly reduces the overhead. >>>>>>> >>>>>> >>>>>> Yes. That's the truth. >>>>>> We can set two different modes for different scenarios. Maybe Named >>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios >>>>>> and 2) non-frequent checkpoint mode for other scenarios. >>>>>> >>>>>> But that's the next plan, we are thinking about that. >>>>> >>>>> I have some code that tries to automatically switch between those; >>>>> it measures the checkpoint lengths, and if they're consistently short >>>>> it sends a different message byte to the secondary at the start of the >>>>> checkpoint, so that it doesn't bother running. Every so often it >>>>> then flips back to a COLO checkpoint to see if the checkpoints >>>>> are still really fast. >>>>> >>>> >>>> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM? >>>> Resume to common COLO mode until the checkpoint lengths is so not short ? >>> >>> We still have to do checkpoints, but we send a special message to the SVM so that >>> the SVM just takes the checkpoint but does not run. >>> >>> I'll send the code after I've updated it to your current version; but it's >>> quite rough/experimental. >>> >>> It works something like >>> >>> -----------run PVM run SVM >>> COLO <long gap> >>> mode miscompare >>> checkpoint >>> -----------run PVM run SVM >>> COLO <short gap> >>> mode miscompare >>> checkpoint >>> -----------run PVM run SVM >>> COLO <short gap> >>> mode miscompare < After a few short runs >>> checkpoint >>> -----------run PVM SVM idle \ >>> Passive <fixed delay> | - repeat 'n' times >>> mode checkpoint / >>> -----------run PVM run SVM >>> COLO <short gap> < Still a short gap >>> mode miscompare >>> -----------run PVM SVM idle \ >>> Passive <fixed delay> | - repeat 'n' times >>> mode checkpoint / >>> -----------run PVM run SVM >>> COLO <long gap> < long gap now, stay in COLO >>> mode miscompare >>> checkpoint >>> -----------run PVM run SVM >>> COLO <long gap> >>> mode miscompare >>> checkpoint >>> >>> So it saves the CPU time on the SVM, and the comparison traffic, and is >>> automatic at switching into the passive mode. >>> >>> It used to be more useful, but your minimum COLO run time that you >>> added a few versions ago helps a lot in the cases where there are miscompares, >>> and the delay after the miscompare before you take the checkpoint also helps >>> in the case where the data is very random. >> >> This is great! This is exactly what we were thinking about, when random >> scenario will fallback to MC/Remus like FT. Thank you very much! >> I have a question, do you also modify colo-proxy kernel module? because >> in the fixed checkpoint mode, I think we need to buffer the network >> packets, and release them at checkpoint. > > Yes, we do need to buffer and release them at the end, but I've not modified > colo-proxy so far. Doesn't the current code on PMY already need to buffer packets > that are generated after the first miscompare Yes, they are buffered, > and before the checkpoint and > then release them at the checkpoint? but will be release only if the packets compare returns identical. so in order to support this fallback mode, we need to modify it to release the packets at the checkpoint, there won't be too much code though. > > Dave > >> >>> >>> Dave >>> >>>> >>>> Thanks. >>>> >>>>> Dave >>>>> >>>>>> >>>>>> Regards, >>>>>> -Gonglei >>>>>> >>>>> -- >>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>>> >>>>> . >>>>> >>>> >>>> >>> -- >>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>> . >>> >> >> -- >> Thanks, >> Yang. > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-31 1:08 ` Yang Hongyang @ 2015-07-31 1:28 ` zhanghailiang 2015-07-31 1:31 ` Yang Hongyang 0 siblings, 1 reply; 63+ messages in thread From: zhanghailiang @ 2015-07-31 1:28 UTC (permalink / raw) To: Yang Hongyang, Dr. David Alan Gilbert Cc: Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei, stefanha@redhat.com On 2015/7/31 9:08, Yang Hongyang wrote: > > > On 07/31/2015 01:53 AM, Dr. David Alan Gilbert wrote: >> * Yang Hongyang (yanghy@cn.fujitsu.com) wrote: >>> >>> >>> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote: >>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: >>>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: >>>>>> * Gonglei (arei.gonglei@huawei.com) wrote: >>>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: >>>>>>>> * Jason Wang (jasowang@redhat.com) wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: >>>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote: >>>>>>>>>>>>> A question here, the packet comparing may be very tricky. For example, >>>>>>>>>>>>> some protocol use random data to generate unpredictable id or >>>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random >>>>>>>>>>>> data? >>>>>>>>>>>> Good question, the random data connection is a big problem for COLO. At >>>>>>>>>>>> present, it will trigger checkpoint processing because of the different random >>>>>>>>>>>> data. >>>>>>>>>>>> I don't think any mechanisms can assure two different machines generate the >>>>>>>>>>>> same random data. If you have any ideas, pls tell us :) >>>>>>>>>>>> >>>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the >>>>>>>>>>>> performance poor. :( >>>>>>>>>>>> >>>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... >>>>>>>>>> They do diverge pretty quickly though; I have simple examples which >>>>>>>>>> reliably cause a checkpoint because of simple randomness in applications. >>>>>>>>>> >>>>>>>>>> Dave >>>>>>>>>> >>>>>>>>> >>>>>>>>> And it will become even worse if hwrng is used in guest. >>>>>>>> >>>>>>>> Yes; it seems quite application dependent; (on IPv4) an ssh connection, >>>>>>>> once established, tends to work well without triggering checkpoints; >>>>>>>> and static web pages also work well. Examples of things that do cause >>>>>>>> more checkpoints are, displaying guest statistics (e.g. running top >>>>>>>> in that ssh) which is timing dependent, and dynamically generated >>>>>>>> web pages that include a unique ID (bugzilla's password reset link in >>>>>>>> it's front page was a fun one), I think also establishing >>>>>>>> new encrypted connections cause the same randomness. >>>>>>>> >>>>>>>> However, it's worth remembering that COLO is trying to reduce the >>>>>>>> number of checkpoints compared to a simple checkpointing world >>>>>>>> which would be aiming to do a checkpoint ~100 times a second, >>>>>>>> and for compute bound workloads, or ones that don't expose >>>>>>>> the randomness that much, it can get checkpoints of a few seconds >>>>>>>> in length which greatly reduces the overhead. >>>>>>>> >>>>>>> >>>>>>> Yes. That's the truth. >>>>>>> We can set two different modes for different scenarios. Maybe Named >>>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios >>>>>>> and 2) non-frequent checkpoint mode for other scenarios. >>>>>>> >>>>>>> But that's the next plan, we are thinking about that. >>>>>> >>>>>> I have some code that tries to automatically switch between those; >>>>>> it measures the checkpoint lengths, and if they're consistently short >>>>>> it sends a different message byte to the secondary at the start of the >>>>>> checkpoint, so that it doesn't bother running. Every so often it >>>>>> then flips back to a COLO checkpoint to see if the checkpoints >>>>>> are still really fast. >>>>>> >>>>> >>>>> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM? >>>>> Resume to common COLO mode until the checkpoint lengths is so not short ? >>>> >>>> We still have to do checkpoints, but we send a special message to the SVM so that >>>> the SVM just takes the checkpoint but does not run. >>>> >>>> I'll send the code after I've updated it to your current version; but it's >>>> quite rough/experimental. >>>> >>>> It works something like >>>> >>>> -----------run PVM run SVM >>>> COLO <long gap> >>>> mode miscompare >>>> checkpoint >>>> -----------run PVM run SVM >>>> COLO <short gap> >>>> mode miscompare >>>> checkpoint >>>> -----------run PVM run SVM >>>> COLO <short gap> >>>> mode miscompare < After a few short runs >>>> checkpoint >>>> -----------run PVM SVM idle \ >>>> Passive <fixed delay> | - repeat 'n' times >>>> mode checkpoint / >>>> -----------run PVM run SVM >>>> COLO <short gap> < Still a short gap >>>> mode miscompare >>>> -----------run PVM SVM idle \ >>>> Passive <fixed delay> | - repeat 'n' times >>>> mode checkpoint / >>>> -----------run PVM run SVM >>>> COLO <long gap> < long gap now, stay in COLO >>>> mode miscompare >>>> checkpoint >>>> -----------run PVM run SVM >>>> COLO <long gap> >>>> mode miscompare >>>> checkpoint >>>> >>>> So it saves the CPU time on the SVM, and the comparison traffic, and is >>>> automatic at switching into the passive mode. >>>> >>>> It used to be more useful, but your minimum COLO run time that you >>>> added a few versions ago helps a lot in the cases where there are miscompares, >>>> and the delay after the miscompare before you take the checkpoint also helps >>>> in the case where the data is very random. >>> >>> This is great! This is exactly what we were thinking about, when random >>> scenario will fallback to MC/Remus like FT. Thank you very much! >>> I have a question, do you also modify colo-proxy kernel module? because >>> in the fixed checkpoint mode, I think we need to buffer the network >>> packets, and release them at checkpoint. >> >> Yes, we do need to buffer and release them at the end, but I've not modified >> colo-proxy so far. Doesn't the current code on PMY already need to buffer packets >> that are generated after the first miscompare > > Yes, they are buffered, > >> and before the checkpoint and >> then release them at the checkpoint? > > but will be release only if the packets compare returns identical. so in order > to support this fallback mode, we need to modify it to release the packets at > the checkpoint, there won't be too much code though. > No, when do checkpoint, we will send all the residual queued packets. So it is already supported. >> >> Dave >> >>> >>>> >>>> Dave >>>> >>>>> >>>>> Thanks. >>>>> >>>>>> Dave >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> -Gonglei >>>>>>> >>>>>> -- >>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>>>> >>>>>> . >>>>>> >>>>> >>>>> >>>> -- >>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>> . >>>> >>> >>> -- >>> Thanks, >>> Yang. >> -- >> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >> . >> > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-31 1:28 ` zhanghailiang @ 2015-07-31 1:31 ` Yang Hongyang 0 siblings, 0 replies; 63+ messages in thread From: Yang Hongyang @ 2015-07-31 1:31 UTC (permalink / raw) To: zhanghailiang, Dr. David Alan Gilbert Cc: Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei, stefanha@redhat.com On 07/31/2015 09:28 AM, zhanghailiang wrote: > On 2015/7/31 9:08, Yang Hongyang wrote: >> >> >> On 07/31/2015 01:53 AM, Dr. David Alan Gilbert wrote: >>> * Yang Hongyang (yanghy@cn.fujitsu.com) wrote: >>>> >>>> >>>> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote: >>>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: >>>>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: >>>>>>> * Gonglei (arei.gonglei@huawei.com) wrote: >>>>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: >>>>>>>>> * Jason Wang (jasowang@redhat.com) wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: >>>>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote: >>>>>>>>>>>>>> A question here, the packet comparing may be very tricky. For >>>>>>>>>>>>>> example, >>>>>>>>>>>>>> some protocol use random data to generate unpredictable id or >>>>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random >>>>>>>>>>>>> data? >>>>>>>>>>>>> Good question, the random data connection is a big problem for >>>>>>>>>>>>> COLO. At >>>>>>>>>>>>> present, it will trigger checkpoint processing because of the >>>>>>>>>>>>> different random >>>>>>>>>>>>> data. >>>>>>>>>>>>> I don't think any mechanisms can assure two different machines >>>>>>>>>>>>> generate the >>>>>>>>>>>>> same random data. If you have any ideas, pls tell us :) >>>>>>>>>>>>> >>>>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the >>>>>>>>>>>>> performance poor. :( >>>>>>>>>>>>> >>>>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have >>>>>>>>>>>> identical internal state, so the pattern used to generate random >>>>>>>>>>>> data has high possibility to generate identical data at short time, >>>>>>>>>>>> at least... >>>>>>>>>>> They do diverge pretty quickly though; I have simple examples which >>>>>>>>>>> reliably cause a checkpoint because of simple randomness in >>>>>>>>>>> applications. >>>>>>>>>>> >>>>>>>>>>> Dave >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> And it will become even worse if hwrng is used in guest. >>>>>>>>> >>>>>>>>> Yes; it seems quite application dependent; (on IPv4) an ssh connection, >>>>>>>>> once established, tends to work well without triggering checkpoints; >>>>>>>>> and static web pages also work well. Examples of things that do cause >>>>>>>>> more checkpoints are, displaying guest statistics (e.g. running top >>>>>>>>> in that ssh) which is timing dependent, and dynamically generated >>>>>>>>> web pages that include a unique ID (bugzilla's password reset link in >>>>>>>>> it's front page was a fun one), I think also establishing >>>>>>>>> new encrypted connections cause the same randomness. >>>>>>>>> >>>>>>>>> However, it's worth remembering that COLO is trying to reduce the >>>>>>>>> number of checkpoints compared to a simple checkpointing world >>>>>>>>> which would be aiming to do a checkpoint ~100 times a second, >>>>>>>>> and for compute bound workloads, or ones that don't expose >>>>>>>>> the randomness that much, it can get checkpoints of a few seconds >>>>>>>>> in length which greatly reduces the overhead. >>>>>>>>> >>>>>>>> >>>>>>>> Yes. That's the truth. >>>>>>>> We can set two different modes for different scenarios. Maybe Named >>>>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios >>>>>>>> and 2) non-frequent checkpoint mode for other scenarios. >>>>>>>> >>>>>>>> But that's the next plan, we are thinking about that. >>>>>>> >>>>>>> I have some code that tries to automatically switch between those; >>>>>>> it measures the checkpoint lengths, and if they're consistently short >>>>>>> it sends a different message byte to the secondary at the start of the >>>>>>> checkpoint, so that it doesn't bother running. Every so often it >>>>>>> then flips back to a COLO checkpoint to see if the checkpoints >>>>>>> are still really fast. >>>>>>> >>>>>> >>>>>> Do you mean if there are consistent checkpoint requests, not do checkpoint >>>>>> but just send a special message to SVM? >>>>>> Resume to common COLO mode until the checkpoint lengths is so not short ? >>>>> >>>>> We still have to do checkpoints, but we send a special message to the >>>>> SVM so that >>>>> the SVM just takes the checkpoint but does not run. >>>>> >>>>> I'll send the code after I've updated it to your current version; but it's >>>>> quite rough/experimental. >>>>> >>>>> It works something like >>>>> >>>>> -----------run PVM run SVM >>>>> COLO <long gap> >>>>> mode miscompare >>>>> checkpoint >>>>> -----------run PVM run SVM >>>>> COLO <short gap> >>>>> mode miscompare >>>>> checkpoint >>>>> -----------run PVM run SVM >>>>> COLO <short gap> >>>>> mode miscompare < After a few short runs >>>>> checkpoint >>>>> -----------run PVM SVM idle \ >>>>> Passive <fixed delay> | - repeat 'n' times >>>>> mode checkpoint / >>>>> -----------run PVM run SVM >>>>> COLO <short gap> < Still a short gap >>>>> mode miscompare >>>>> -----------run PVM SVM idle \ >>>>> Passive <fixed delay> | - repeat 'n' times >>>>> mode checkpoint / >>>>> -----------run PVM run SVM >>>>> COLO <long gap> < long gap now, stay in COLO >>>>> mode miscompare >>>>> checkpoint >>>>> -----------run PVM run SVM >>>>> COLO <long gap> >>>>> mode miscompare >>>>> checkpoint >>>>> >>>>> So it saves the CPU time on the SVM, and the comparison traffic, and is >>>>> automatic at switching into the passive mode. >>>>> >>>>> It used to be more useful, but your minimum COLO run time that you >>>>> added a few versions ago helps a lot in the cases where there are miscompares, >>>>> and the delay after the miscompare before you take the checkpoint also helps >>>>> in the case where the data is very random. >>>> >>>> This is great! This is exactly what we were thinking about, when random >>>> scenario will fallback to MC/Remus like FT. Thank you very much! >>>> I have a question, do you also modify colo-proxy kernel module? because >>>> in the fixed checkpoint mode, I think we need to buffer the network >>>> packets, and release them at checkpoint. >>> >>> Yes, we do need to buffer and release them at the end, but I've not modified >>> colo-proxy so far. Doesn't the current code on PMY already need to buffer >>> packets >>> that are generated after the first miscompare >> >> Yes, they are buffered, >> >>> and before the checkpoint and >>> then release them at the checkpoint? >> >> but will be release only if the packets compare returns identical. so in order >> to support this fallback mode, we need to modify it to release the packets at >> the checkpoint, there won't be too much code though. >> > > No, when do checkpoint, we will send all the residual queued packets. > So it is already supported. Great, my memory is wrong, sorry... > >>> >>> Dave >>> >>>> >>>>> >>>>> Dave >>>>> >>>>>> >>>>>> Thanks. >>>>>> >>>>>>> Dave >>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> -Gonglei >>>>>>>> >>>>>>> -- >>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>>>>> >>>>>>> . >>>>>>> >>>>>> >>>>>> >>>>> -- >>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>>> . >>>>> >>>> >>>> -- >>>> Thanks, >>>> Yang. >>> -- >>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>> . >>> >> > > > . > -- Thanks, Yang. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [Qemu-devel] [POC] colo-proxy in qemu 2015-07-30 17:53 ` Dr. David Alan Gilbert 2015-07-31 1:08 ` Yang Hongyang @ 2015-07-31 1:26 ` zhanghailiang 1 sibling, 0 replies; 63+ messages in thread From: zhanghailiang @ 2015-07-31 1:26 UTC (permalink / raw) To: Dr. David Alan Gilbert, Yang Hongyang Cc: Li Zhijian, jan.kiszka@siemens.com, Jason Wang, Dong, Eddie, peter.huangpeng, qemu-devel@nongnu.org, Gonglei, stefanha@redhat.com On 2015/7/31 1:53, Dr. David Alan Gilbert wrote: > * Yang Hongyang (yanghy@cn.fujitsu.com) wrote: >> >> >> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote: >>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote: >>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote: >>>>> * Gonglei (arei.gonglei@huawei.com) wrote: >>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote: >>>>>>> * Jason Wang (jasowang@redhat.com) wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote: >>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote: >>>>>>>>>>>> A question here, the packet comparing may be very tricky. For example, >>>>>>>>>>>> some protocol use random data to generate unpredictable id or >>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO >>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random >>>>>>>>>>> data? >>>>>>>>>>> Good question, the random data connection is a big problem for COLO. At >>>>>>>>>>> present, it will trigger checkpoint processing because of the different random >>>>>>>>>>> data. >>>>>>>>>>> I don't think any mechanisms can assure two different machines generate the >>>>>>>>>>> same random data. If you have any ideas, pls tell us :) >>>>>>>>>>> >>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the >>>>>>>>>>> performance poor. :( >>>>>>>>>>> >>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least... >>>>>>>>> They do diverge pretty quickly though; I have simple examples which >>>>>>>>> reliably cause a checkpoint because of simple randomness in applications. >>>>>>>>> >>>>>>>>> Dave >>>>>>>>> >>>>>>>> >>>>>>>> And it will become even worse if hwrng is used in guest. >>>>>>> >>>>>>> Yes; it seems quite application dependent; (on IPv4) an ssh connection, >>>>>>> once established, tends to work well without triggering checkpoints; >>>>>>> and static web pages also work well. Examples of things that do cause >>>>>>> more checkpoints are, displaying guest statistics (e.g. running top >>>>>>> in that ssh) which is timing dependent, and dynamically generated >>>>>>> web pages that include a unique ID (bugzilla's password reset link in >>>>>>> it's front page was a fun one), I think also establishing >>>>>>> new encrypted connections cause the same randomness. >>>>>>> >>>>>>> However, it's worth remembering that COLO is trying to reduce the >>>>>>> number of checkpoints compared to a simple checkpointing world >>>>>>> which would be aiming to do a checkpoint ~100 times a second, >>>>>>> and for compute bound workloads, or ones that don't expose >>>>>>> the randomness that much, it can get checkpoints of a few seconds >>>>>>> in length which greatly reduces the overhead. >>>>>>> >>>>>> >>>>>> Yes. That's the truth. >>>>>> We can set two different modes for different scenarios. Maybe Named >>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios >>>>>> and 2) non-frequent checkpoint mode for other scenarios. >>>>>> >>>>>> But that's the next plan, we are thinking about that. >>>>> >>>>> I have some code that tries to automatically switch between those; >>>>> it measures the checkpoint lengths, and if they're consistently short >>>>> it sends a different message byte to the secondary at the start of the >>>>> checkpoint, so that it doesn't bother running. Every so often it >>>>> then flips back to a COLO checkpoint to see if the checkpoints >>>>> are still really fast. >>>>> >>>> >>>> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM? >>>> Resume to common COLO mode until the checkpoint lengths is so not short ? >>> >>> We still have to do checkpoints, but we send a special message to the SVM so that >>> the SVM just takes the checkpoint but does not run. >>> >>> I'll send the code after I've updated it to your current version; but it's >>> quite rough/experimental. >>> Yes, please, we can merge them into our branch. ;) >>> It works something like >>> >>> -----------run PVM run SVM >>> COLO <long gap> >>> mode miscompare >>> checkpoint >>> -----------run PVM run SVM >>> COLO <short gap> >>> mode miscompare >>> checkpoint >>> -----------run PVM run SVM >>> COLO <short gap> >>> mode miscompare < After a few short runs >>> checkpoint >>> -----------run PVM SVM idle \ >>> Passive <fixed delay> | - repeat 'n' times >>> mode checkpoint / >>> -----------run PVM run SVM >>> COLO <short gap> < Still a short gap >>> mode miscompare >>> -----------run PVM SVM idle \ >>> Passive <fixed delay> | - repeat 'n' times >>> mode checkpoint / >>> -----------run PVM run SVM >>> COLO <long gap> < long gap now, stay in COLO >>> mode miscompare >>> checkpoint >>> -----------run PVM run SVM >>> COLO <long gap> >>> mode miscompare >>> checkpoint >>> >>> So it saves the CPU time on the SVM, and the comparison traffic, and is >>> automatic at switching into the passive mode. >>> That's a good solution, actually, we have a plan to realize the checkpoint strategy, which can automatically adapt to different situation, including period checkpoint (MC/Remus mode), COLO mode, mix mode (just like your above method), also, it it a good idea to apply command for users to choose the mode~ >>> It used to be more useful, but your minimum COLO run time that you >>> added a few versions ago helps a lot in the cases where there are miscompares, >>> and the delay after the miscompare before you take the checkpoint also helps >>> in the case where the data is very random. >> >> This is great! This is exactly what we were thinking about, when random >> scenario will fallback to MC/Remus like FT. Thank you very much! >> I have a question, do you also modify colo-proxy kernel module? because >> in the fixed checkpoint mode, I think we need to buffer the network >> packets, and release them at checkpoint. > > Yes, we do need to buffer and release them at the end, but I've not modified > colo-proxy so far. Doesn't the current code on PMY already need to buffer packets > that are generated after the first miscompare and before the checkpoint and > then release them at the checkpoint? > Yes, we have support this already, for PMY, it will queue all packets sent from PVM, and waits for corresponding net packets sent from SVM, after comparing, if it is consistent it will be send out, if not, there will be a mis-compare checkpoint request. After a checkpoint, we will release all packets that queued. So if the SVM is not running, we will queued all packets, without any comparison, and just release them after a new checkpoint. Thanks, zhanghailiang >> >>> >>> Dave >>> >>>> >>>> Thanks. >>>> >>>>> Dave >>>>> >>>>>> >>>>>> Regards, >>>>>> -Gonglei >>>>>> >>>>> -- >>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>>>> >>>>> . >>>>> >>>> >>>> >>> -- >>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK >>> . >>> >> >> -- >> Thanks, >> Yang. > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > . > ^ permalink raw reply [flat|nested] 63+ messages in thread
end of thread, other threads:[~2015-11-13 12:33 UTC | newest] Thread overview: 63+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-11-10 5:26 [Qemu-devel] [POC]colo-proxy in qemu Tkid 2015-11-10 7:35 ` Jason Wang 2015-11-10 8:30 ` zhanghailiang 2015-11-11 2:28 ` Jason Wang 2015-11-10 9:35 ` Tkid 2015-11-11 3:04 ` Jason Wang 2015-11-10 9:41 ` Dr. David Alan Gilbert 2015-11-11 3:09 ` Jason Wang 2015-11-11 9:03 ` Dr. David Alan Gilbert 2015-11-11 1:23 ` Dong, Eddie 2015-11-11 3:26 ` Jason Wang 2015-11-10 10:54 ` Dr. David Alan Gilbert 2015-11-11 2:46 ` Zhang Chen 2015-11-13 12:33 ` Dr. David Alan Gilbert -- strict thread matches above, loose matches on Subject: below -- 2015-07-20 6:42 [Qemu-devel] [POC] colo-proxy " Li Zhijian 2015-07-20 10:32 ` Stefan Hajnoczi 2015-07-20 11:55 ` zhanghailiang 2015-07-20 13:12 ` Vasiliy Tolstov 2015-07-20 15:01 ` Stefan Hajnoczi 2015-07-21 1:59 ` zhanghailiang 2015-07-28 22:13 ` Samuel Thibault 2015-07-21 6:13 ` Jan Kiszka 2015-07-21 9:49 ` Stefan Hajnoczi 2015-07-27 10:13 ` Stefan Hajnoczi 2015-07-27 11:24 ` zhanghailiang 2015-07-27 11:31 ` Samuel Thibault 2015-07-27 13:33 ` Jan Kiszka 2015-07-28 22:12 ` Samuel Thibault 2015-07-29 7:36 ` Jan Kiszka 2015-07-20 12:02 ` Li Zhijian 2015-07-24 2:04 ` Dong, Eddie 2015-07-24 2:12 ` Jason Wang 2015-07-24 8:04 ` Yang Hongyang 2015-07-27 3:24 ` Jason Wang 2015-07-27 3:54 ` Yang Hongyang 2015-07-27 4:49 ` Jason Wang 2015-07-27 5:51 ` Yang Hongyang 2015-07-27 7:37 ` Jason Wang 2015-07-27 7:49 ` Yang Hongyang 2015-07-27 8:06 ` Jason Wang 2015-07-27 8:22 ` Yang Hongyang 2015-07-27 7:53 ` Jason Wang 2015-07-27 8:17 ` Yang Hongyang 2015-07-27 18:33 ` Dr. David Alan Gilbert 2015-07-27 10:40 ` Dr. David Alan Gilbert 2015-07-27 13:39 ` Yang Hongyang 2015-07-24 2:05 ` Dong, Eddie 2015-07-30 4:23 ` Jason Wang 2015-07-30 7:16 ` Gonglei 2015-07-30 7:47 ` Dong, Eddie 2015-07-30 8:03 ` Dr. David Alan Gilbert 2015-07-30 8:15 ` Jason Wang 2015-07-30 11:56 ` Dr. David Alan Gilbert 2015-07-30 12:10 ` Gonglei 2015-07-30 12:30 ` Dr. David Alan Gilbert 2015-07-30 12:42 ` zhanghailiang 2015-07-30 13:59 ` Dr. David Alan Gilbert 2015-07-30 15:17 ` Yang Hongyang 2015-07-30 17:53 ` Dr. David Alan Gilbert 2015-07-31 1:08 ` Yang Hongyang 2015-07-31 1:28 ` zhanghailiang 2015-07-31 1:31 ` Yang Hongyang 2015-07-31 1:26 ` zhanghailiang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).