[Qemu-devel] [POC] colo-proxy in qemu

[Qemu-devel] [POC] colo-proxy in qemu

[Li Zhijian]

Hi, all

We are planning to implement colo-proxy in qemu to cache and compare packets.
This module is one of the important component of COLO project and now it is
still in early stage, so any comments and feedback are warmly welcomed,
thanks in advance.

## Background
COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
project is a high availability solution. Both Primary VM (PVM) and Secondary VM
(SVM) run in parallel. They receive the same request from client, and generate
responses in parallel too. If the response packets from PVM and SVM are
identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
is conducted.
Paper:
http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
COLO on Xen:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
COLO on Qemu/KVM:
http://wiki.qemu.org/Features/COLO

By the needs of capturing response packets from PVM and SVM and finding out
whether they are identical, we introduce a new module to qemu networking called
colo-proxy.

This document describes the design of the colo-proxy module

## Glossary
   PVM - Primary VM, which provides services to clients.
   SVM - Secondary VM, a hot standby and replication of PVM.
   PN - Primary Node, the host which PVM runs on
   SN - Secondary Node, the host which SVM runs on

## Workflow ##
The following image show the qemu networking packet datapath between
guest's NIC and qemu's backend in colo-proxy.

+---+                                        +---+
|PN |                                        |SN |
+---+--------------------------+             +------------------------------+
|               +-------+      |             |   +-------+                  |
+--------+      |chkpoint<--------[socket]------->chkpoint         +--------+
|PVM     |      +---^---+      |             |   +---+---+         |SVM     |
|        |  +proxy--v--------+ |             |       |             |        |
|        |  |                | |             |       |             |        |
| +---+  |  | +TCP/IP stack+ | |             | +-----v-------proxy | +---+  |
+-|NIC|--+  | |            | | |             | |                 | +-|NIC|--+
| +^-++     | | +--------+ | | |             | | +TCP/IP stack-+ |   +^--+  |
|  | +------> | | compare| | <-[socket]-forward- | +--------+  | |    |     |
|  |        | | +---+----+ | | |             | | | |seq&ack |  | <----+     |
|  |        | +-----|------+ | |             | | | |adjust  |  | |          |
|  |        |       |        | |             | | | +--------+  | |          |
|  +-----------<+>-----copy&forward-[socket]---> +-------------+ |          |
|           +---|---|--------+ |             | +------------^----+          |
|               |   |          |             |              |               |
|               |   |          |             |              x               |
|            +--+---v----+     |             |            +-v---------+     |
| QEMU       |  backend  |     |             | QEMU       |  backend  |     |
+------------+  (tap)    +-----+             +------------+  (tap)    +-----+
              +-----------+                                +-----------+

## Our Idea ##

### Net filter
In current QEMU, a packet is transported between networking backend(tap) and
qemu network adapter(NIC) directly. Backend and adapter is linked by
NetClientState->peer in qemu as following
        +----------------------------------------+
        v                                        |
+NetClientState+   +------->+NetClientState+    |
|info->type=TAP|   |        |info->type=NIC|    |
+--------------+   |        +--------------+    |
|   *peer      +---+        |   *peer      +----+
+--------------+            +--------------+
|name="tap0"   |            |name="e1000"  |
+--------------+            +--------------+
| ...          |            | ...          |
+--------------+            +--------------+

In COLO QEMU, we insert a net filter named colo-proxy between backend and
adapter like below:
typedef struct COLOState {
     NetClientState nc;
     NetClientState *peer;
} COLOState;
    +------->+NetClientState+            +NetClientState+<--------+
    |        |info->type=TAP|            |info->type=NIC|         |
    |        +--------------+            +--------------+         |
+-----------+   *peer      |            |   *peer      +------------+
|  |        +--------------+            +--------------+         |  |
|  |        |name="tap0"   |            |name="e1000"  |         |  |
|  |        +--------------+            +--------------+         |  |
|  |        | ...          |            | ...          |         |  |
|  |        +--------------+            +--------------+         |  |
|  |                                                             |  |
|  |   +-COLOState------------+       +-COLOState------------+   |  |
+--------->+NetClientState+<- - +   +---->+NetClientState+<---------+
    |   |   |info->type=COLO   | |   | |   |info->type=COLO   |   |
    |   |   +--------------+   | |   | |   +--------------+   |   |
    +-------+   *peer      |   | |   | |   |   *peer      +-------+
        |   +--------------+   | |   | |   +--------------+   |
        |   |name="colo1"  |   | |   | |   |name="colo2"  |   |
        |   +--------------+   | |   | |   +--------------+   |
        |                      | |   | |                      |
        |   +--------------+---------+ |   +--------------+   |
        |   |   *peer      |   | |     |   |   *peer      |   |
        |   +--------------+   | +---------+--------------+   |
        +----------------------+       +----------------------+

After we insert colo-proxy filter, all packets will pass by this filter
and more important thing is that we can analysis packet by ourselves.

### QEMU space TCP/IP stack(re-use SLIRP) ###
We need a QEMU space TCP/IP stack to help us to analysis packet. After looking
into QEMU, we found that SLIRP
http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it can
help use to handle the packet written to/read from backend(tap) device which is
just like a link layer(L2) packet.

### packet enqueue and compare ###
Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM and
SVM on Primary QEMU, and then compare the packet payload for each connection.

### Net filter Usage ###
On both Primary/Secondary host, invoke QEMU with the following parameters to insert
a net filter(colo-proxy):
   "-netdev tap,id=hn0 -device e1000,netdev=hn0 \
    -netdev colo,id=colo,backend=hn0"

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
> We are planning to implement colo-proxy in qemu to cache and compare packets.

I thought there is a kernel module to do that?

Why does the proxy need to be part of the QEMU process?  -netdev socket
or host network stack features allow you to process packets in a
separate process.

Without details on what the proxy does it's hard to discuss this.  What
happens in the non-TCP case?  What happens in the TCP case?

Does the proxy need to perform privileged operations, create sockets,
open files, etc?

The slirp code is not actively developed or used much in production.  It
might be a good idea to audit the code for bugs if you want to use it.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [POC] colo-proxy in qemu

[zhanghailiang]

On 2015/7/20 18:32, Stefan Hajnoczi wrote:
> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>> We are planning to implement colo-proxy in qemu to cache and compare packets.
>
> I thought there is a kernel module to do that?
>

Yes, but we decided to re-implement it in userspace (Here is in qemu),
there are mainly two reasons that we made this change. One is the colo-proxy in kernel is
narrowly used, which can only be used for COLO FT, besides, we have to modify iptables and nftables
to support this capability. IMHO, it is hardly been accepted by the kernel community.
The other reason is that the kernel proxy scenario can't been used in all situations, for example,
evs + vhost-user + dpdk, it can't work if VM's network packets don't go through host's network
stack. (For the new userspace colo proxy scheme, we also can't use it with vhost-net,
we have to use virtio-net instead).

> Why does the proxy need to be part of the QEMU process?  -netdev socket
> or host network stack features allow you to process packets in a
> separate process.
>
> Without details on what the proxy does it's hard to discuss this.  What
> happens in the non-TCP case?  What happens in the TCP case?
>
> Does the proxy need to perform privileged operations, create sockets,
> open files, etc?
>
> The slirp code is not actively developed or used much in production.  It
> might be a good idea to audit the code for bugs if you want to use it.
>

Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
have to supplement it.

Thanks,
zhanghailiang

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Vasiliy Tolstov]

2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>:
> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
> have to supplement it.


patch for ipv6 slirp support some times ago sended to qemu list, but i
don't know why in not accepted.

-- 
Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Stefan Hajnoczi]

On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote:
> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>:
>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
>> have to supplement it.
>
>
> patch for ipv6 slirp support some times ago sended to qemu list, but i
> don't know why in not accepted.

I think no one reviewed it but there was no objection against IPv6
support in principle.

Jan: Can we merge slirp IPv6 support for QEMU 2.5?

Stefan

Re: [Qemu-devel] [POC] colo-proxy in qemu

[zhanghailiang]

On 2015/7/20 23:01, Stefan Hajnoczi wrote:
> On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote:
>> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>:
>>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
>>> have to supplement it.
>>
>>
>> patch for ipv6 slirp support some times ago sended to qemu list, but i
>> don't know why in not accepted.
>
> I think no one reviewed it but there was no objection against IPv6
> support in principle.
>
> Jan: Can we merge slirp IPv6 support for QEMU 2.5?
>

I have found the corresponding patch series:
https://lists.gnu.org/archive/html/qemu-devel/2014-02/msg01832.html

Cc: Samuel Thibault <samuel.thibault@ens-lyon.org>

Hi Samuel,

What's the status of the 'slirp: Adding IPv6 support to Qemu -net use' series ?
I didn't find any news since that version, are you still trying to push them to qemu upstream ?

Thanks,
zhanghailiang

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Samuel Thibault]

zhanghailiang, le Tue 21 Jul 2015 09:59:22 +0800, a écrit :
> I didn't find any news since that version, are you still trying to
> push them to qemu upstream ?

I'd still be trying if I had any actual answer other than "we need to
find time to deal about it" :)

I can rebase the patch series over the current master and submit again
the patches.

Samuel

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jan Kiszka]

On 2015-07-20 17:01, Stefan Hajnoczi wrote:
> On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote:
>> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>:
>>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
>>> have to supplement it.
>>
>>
>> patch for ipv6 slirp support some times ago sended to qemu list, but i
>> don't know why in not accepted.
> 
> I think no one reviewed it but there was no objection against IPv6
> support in principle.
> 
> Jan: Can we merge slirp IPv6 support for QEMU 2.5?

Sorry, as I pointed out some time back, I don't have the bandwidth to
look into slirp. Someone need to do a review, then send a pull request.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote:
> On 2015-07-20 17:01, Stefan Hajnoczi wrote:
> > On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote:
> >> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>:
> >>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
> >>> have to supplement it.
> >>
> >>
> >> patch for ipv6 slirp support some times ago sended to qemu list, but i
> >> don't know why in not accepted.
> > 
> > I think no one reviewed it but there was no objection against IPv6
> > support in principle.
> > 
> > Jan: Can we merge slirp IPv6 support for QEMU 2.5?
> 
> Sorry, as I pointed out some time back, I don't have the bandwidth to
> look into slirp. Someone need to do a review, then send a pull request.

Do you want to remove yourself from the slirp section of the MAINTAINERS
file?

Going forward we'll need to find someone familiar with the QEMU
development process and with enough time to review slirp patches.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1324 bytes --]

On Tue, Jul 21, 2015 at 10:49:29AM +0100, Stefan Hajnoczi wrote:
> On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote:
> > On 2015-07-20 17:01, Stefan Hajnoczi wrote:
> > > On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote:
> > >> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>:
> > >>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
> > >>> have to supplement it.
> > >>
> > >>
> > >> patch for ipv6 slirp support some times ago sended to qemu list, but i
> > >> don't know why in not accepted.
> > > 
> > > I think no one reviewed it but there was no objection against IPv6
> > > support in principle.
> > > 
> > > Jan: Can we merge slirp IPv6 support for QEMU 2.5?
> > 
> > Sorry, as I pointed out some time back, I don't have the bandwidth to
> > look into slirp. Someone need to do a review, then send a pull request.
> 
> Do you want to remove yourself from the slirp section of the MAINTAINERS
> file?
> 
> Going forward we'll need to find someone familiar with the QEMU
> development process and with enough time to review slirp patches.

Ping?

I hoped this would raise some discussion and that maybe we could find a
new maintainer or co-maintainer to get slirp moving.

Any thoughts?

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [POC] colo-proxy in qemu

[zhanghailiang]

On 2015/7/27 18:13, Stefan Hajnoczi wrote:
> On Tue, Jul 21, 2015 at 10:49:29AM +0100, Stefan Hajnoczi wrote:
>> On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote:
>>> On 2015-07-20 17:01, Stefan Hajnoczi wrote:
>>>> On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov <v.tolstov@selfip.ru> wrote:
>>>>> 2015-07-20 14:55 GMT+03:00 zhanghailiang <zhang.zhanghailiang@huawei.com>:
>>>>>> Agreed, besides, it is seemed that slirp is not supporting ipv6, we also
>>>>>> have to supplement it.
>>>>>
>>>>>
>>>>> patch for ipv6 slirp support some times ago sended to qemu list, but i
>>>>> don't know why in not accepted.
>>>>
>>>> I think no one reviewed it but there was no objection against IPv6
>>>> support in principle.
>>>>
>>>> Jan: Can we merge slirp IPv6 support for QEMU 2.5?
>>>
>>> Sorry, as I pointed out some time back, I don't have the bandwidth to
>>> look into slirp. Someone need to do a review, then send a pull request.
>>
>> Do you want to remove yourself from the slirp section of the MAINTAINERS
>> file?
>>
>> Going forward we'll need to find someone familiar with the QEMU
>> development process and with enough time to review slirp patches.
>
> Ping?
>
> I hoped this would raise some discussion and that maybe we could find a
> new maintainer or co-maintainer to get slirp moving.
>

Yes, please, this is important, we need slirp's maintainer to help reviewing the COLO proxy
patches that will be implemented based on slirp. (If we finally come to an agreement on realizing it in qemu)

We also need to support ipv6 for slirp, i have emailed Samuel who have sent ipv6 patches for slirp before,
but got no response. (I would like to respin and test Samuel's ipv6 slirp patch if he don't have time to do this,
but firstly, i need to get his permission ：） ）

Cc: Samuel Thibault <samuel.thibault@ens-lyon.org> <samuel.thibault@gnu.org>.

Thanks,
zhanghailiang

> Any thoughts?
>
> Stefan
>

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Samuel Thibault]

Hello,

I'm just back from vacancy with no Internet access, so will answer
shortly :)

Samuel

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jan Kiszka]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2015-07-27 12:13, Stefan Hajnoczi wrote:
> On Tue, Jul 21, 2015 at 10:49:29AM +0100, Stefan Hajnoczi wrote:
>> On Tue, Jul 21, 2015 at 08:13:42AM +0200, Jan Kiszka wrote:
>>> On 2015-07-20 17:01, Stefan Hajnoczi wrote:
>>>> On Mon, Jul 20, 2015 at 2:12 PM, Vasiliy Tolstov
>>>> <v.tolstov@selfip.ru> wrote:
>>>>> 2015-07-20 14:55 GMT+03:00 zhanghailiang
>>>>> <zhang.zhanghailiang@huawei.com>:
>>>>>> Agreed, besides, it is seemed that slirp is not
>>>>>> supporting ipv6, we also have to supplement it.
>>>>> 
>>>>> 
>>>>> patch for ipv6 slirp support some times ago sended to qemu
>>>>> list, but i don't know why in not accepted.
>>>> 
>>>> I think no one reviewed it but there was no objection against
>>>> IPv6 support in principle.
>>>> 
>>>> Jan: Can we merge slirp IPv6 support for QEMU 2.5?
>>> 
>>> Sorry, as I pointed out some time back, I don't have the
>>> bandwidth to look into slirp. Someone need to do a review, then
>>> send a pull request.
>> 
>> Do you want to remove yourself from the slirp section of the
>> MAINTAINERS file?
>> 
>> Going forward we'll need to find someone familiar with the QEMU 
>> development process and with enough time to review slirp
>> patches.
> 
> Ping?
> 
> I hoped this would raise some discussion and that maybe we could
> find a new maintainer or co-maintainer to get slirp moving.
> 
> Any thoughts?

Of course, I'm fine with handing this over to someone who'd like to
pick up. Do we have volunteers?

Samuel, would you like to do this? As a subsystem maintainer, you are
already familiar with QEMU processes. Well, this still wouldn't
resolve the independent review need for slirp-ipv6.

Jan

- -- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlW2MycACgkQitSsb3rl5xSCaACePNubKPBkrdxQkcThUGD7w56B
Q6oAoIgCzT9qVRzDf5IhY2eKFXgTZ+Ul
=yk6R
-----END PGP SIGNATURE-----

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Samuel Thibault]

Hello,

Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
> Of course, I'm fine with handing this over to someone who'd like to
> pick up. Do we have volunteers?
> 
> Samuel, would you like to do this? As a subsystem maintainer, you are
> already familiar with QEMU processes.

I can help with maintenance, yes.

> Well, this still wouldn't resolve the independent review need for
> slirp-ipv6.

Well, actually I didn't write slirp-ipv6, Guillaume Subiron did, and I
reviewed it (and we iterated quite a bit) before we submit the patch
series to qemu-devel.

Samuel

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jan Kiszka]

On 2015-07-29 00:12, Samuel Thibault wrote:
> Hello,
> 
> Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
>> Of course, I'm fine with handing this over to someone who'd like to
>> pick up. Do we have volunteers?
>>
>> Samuel, would you like to do this? As a subsystem maintainer, you are
>> already familiar with QEMU processes.
> 
> I can help with maintenance, yes.

"Help with" will easily mean "be the one and only". ;) If you prefer,
send a patch which only adds you as a maintainer, but I would also ack
one that drops me from the list as well.

> 
>> Well, this still wouldn't resolve the independent review need for
>> slirp-ipv6.
> 
> Well, actually I didn't write slirp-ipv6, Guillaume Subiron did, and I
> reviewed it (and we iterated quite a bit) before we submit the patch
> series to qemu-devel.

Perfekt!

Thanks,
Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

[Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Samuel Thibault]

Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>

---
Jan Kiszka, le Wed 29 Jul 2015 09:36:15 +0200, a écrit :
> On 2015-07-29 00:12, Samuel Thibault wrote:
> > Hello,
> > 
> > Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
> >> Of course, I'm fine with handing this over to someone who'd like to
> >> pick up. Do we have volunteers?
> >>
> >> Samuel, would you like to do this? As a subsystem maintainer, you are
> >> already familiar with QEMU processes.
> > 
> > I can help with maintenance, yes.
> 
> "Help with" will easily mean "be the one and only". ;)

Right, I understand that :) That's still preferable to have two contact
people.

diff --git a/MAINTAINERS b/MAINTAINERS
index 978b717..b506df5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1018,6 +1018,7 @@ F: scripts/qmp/
 T: git git://repo.or.cz/qemu/armbru.git qapi-next
 
 SLIRP
+M: Samuel Thibault <samuel.thibault@ens-lyon.org>
 M: Jan Kiszka <jan.kiszka@siemens.com>
 S: Maintained
 F: slirp/

Re: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

On Wed, Jul 29, 2015 at 11:33:26AM +0200, Samuel Thibault wrote:
> Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
> 
> ---
> Jan Kiszka, le Wed 29 Jul 2015 09:36:15 +0200, a écrit :
> > On 2015-07-29 00:12, Samuel Thibault wrote:
> > > Hello,
> > > 
> > > Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
> > >> Of course, I'm fine with handing this over to someone who'd like to
> > >> pick up. Do we have volunteers?
> > >>
> > >> Samuel, would you like to do this? As a subsystem maintainer, you are
> > >> already familiar with QEMU processes.
> > > 
> > > I can help with maintenance, yes.
> > 
> > "Help with" will easily mean "be the one and only". ;)
> 
> Right, I understand that :) That's still preferable to have two contact
> people.
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 978b717..b506df5 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1018,6 +1018,7 @@ F: scripts/qmp/
>  T: git git://repo.or.cz/qemu/armbru.git qapi-next
>  
>  SLIRP
> +M: Samuel Thibault <samuel.thibault@ens-lyon.org>
>  M: Jan Kiszka <jan.kiszka@siemens.com>
>  S: Maintained
>  F: slirp/

Please resend as a top-level email.

If you send patches as a reply to a thread they are easy to miss.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Fam Zheng]

On Thu, 08/06 11:10, Stefan Hajnoczi wrote:
> On Wed, Jul 29, 2015 at 11:33:26AM +0200, Samuel Thibault wrote:
> > Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
> > 
> > ---
> > Jan Kiszka, le Wed 29 Jul 2015 09:36:15 +0200, a écrit :
> > > On 2015-07-29 00:12, Samuel Thibault wrote:
> > > > Hello,
> > > > 
> > > > Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
> > > >> Of course, I'm fine with handing this over to someone who'd like to
> > > >> pick up. Do we have volunteers?
> > > >>
> > > >> Samuel, would you like to do this? As a subsystem maintainer, you are
> > > >> already familiar with QEMU processes.
> > > > 
> > > > I can help with maintenance, yes.
> > > 
> > > "Help with" will easily mean "be the one and only". ;)
> > 
> > Right, I understand that :) That's still preferable to have two contact
> > people.
> > 
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 978b717..b506df5 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -1018,6 +1018,7 @@ F: scripts/qmp/
> >  T: git git://repo.or.cz/qemu/armbru.git qapi-next
> >  
> >  SLIRP
> > +M: Samuel Thibault <samuel.thibault@ens-lyon.org>
> >  M: Jan Kiszka <jan.kiszka@siemens.com>
> >  S: Maintained
> >  F: slirp/
> 
> Please resend as a top-level email.

This email has proper subject, sign-off line and patch format, and the header
doesn't have "In-reply-to". So it makes a legit patch. The quotes are below the
"---" line so it should be read as complementary information and I think it is
totally fine. At least patchew treats it so :)

http://qemu.patchew.org/series/%3C20150806111014.GA6432%40type.bordeaux.inria.fr%3E

Therefore,

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Stefan Hajnoczi]

On Thu, Aug 6, 2015 at 1:29 PM, Fam Zheng <famz@redhat.com> wrote:
> On Thu, 08/06 11:10, Stefan Hajnoczi wrote:
>> On Wed, Jul 29, 2015 at 11:33:26AM +0200, Samuel Thibault wrote:
>> > Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
>> >
>> > ---
>> > Jan Kiszka, le Wed 29 Jul 2015 09:36:15 +0200, a écrit :
>> > > On 2015-07-29 00:12, Samuel Thibault wrote:
>> > > > Hello,
>> > > >
>> > > > Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
>> > > >> Of course, I'm fine with handing this over to someone who'd like to
>> > > >> pick up. Do we have volunteers?
>> > > >>
>> > > >> Samuel, would you like to do this? As a subsystem maintainer, you are
>> > > >> already familiar with QEMU processes.
>> > > >
>> > > > I can help with maintenance, yes.
>> > >
>> > > "Help with" will easily mean "be the one and only". ;)
>> >
>> > Right, I understand that :) That's still preferable to have two contact
>> > people.
>> >
>> > diff --git a/MAINTAINERS b/MAINTAINERS
>> > index 978b717..b506df5 100644
>> > --- a/MAINTAINERS
>> > +++ b/MAINTAINERS
>> > @@ -1018,6 +1018,7 @@ F: scripts/qmp/
>> >  T: git git://repo.or.cz/qemu/armbru.git qapi-next
>> >
>> >  SLIRP
>> > +M: Samuel Thibault <samuel.thibault@ens-lyon.org>
>> >  M: Jan Kiszka <jan.kiszka@siemens.com>
>> >  S: Maintained
>> >  F: slirp/
>>
>> Please resend as a top-level email.
>
> This email has proper subject, sign-off line and patch format, and the header
> doesn't have "In-reply-to".

It is definitely a reply rather than a top-level post.  Here is what
the headers look like here:

Delivered-To: stefanha@gmail.com
Date: Wed, 29 Jul 2015 11:33:26 +0200
From: Samuel Thibault <samuel.thibault@gnu.org>
To: qemu-devel <qemu-devel@nongnu.org>
Message-ID: <20150729093326.GG2979@type.bordeaux.inria.fr>
References: <20150720103208.GA12675@stefanha-thinkpad.redhat.com>
<55ACE1B4.4010109@huawei.com>
<CACaajQsMGGuxTrOYb0AYF_Vv7kzOztn_Phi4TEKtfWPEEkfRyg@mail.gmail.com>
<CAJSP0QWw8SyENL=eMpFstxr9kG038CVs5GKY3qZ+V1foxDmykg@mail.gmail.com>
<55ADE316.1060704@siemens.com>
<20150721094929.GD13993@stefanha-thinkpad.redhat.com>
<20150727101305.GD30018@stefanha-thinkpad.redhat.com>
<55B63327.5080205@siemens.com> <20150728221234.GI3467@type.home>
<55B8826F.1090305@siemens.com>
In-Reply-To: <55B8826F.1090305@siemens.com>
Subject: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp
maintainer
...

Re: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Fam Zheng]

On Fri, 08/07 11:19, Stefan Hajnoczi wrote:
> It is definitely a reply rather than a top-level post.  Here is what
> the headers look like here:
> 
> Delivered-To: stefanha@gmail.com
> Date: Wed, 29 Jul 2015 11:33:26 +0200
> From: Samuel Thibault <samuel.thibault@gnu.org>
> To: qemu-devel <qemu-devel@nongnu.org>
> Message-ID: <20150729093326.GG2979@type.bordeaux.inria.fr>
> References: <20150720103208.GA12675@stefanha-thinkpad.redhat.com>
> <55ACE1B4.4010109@huawei.com>
> <CACaajQsMGGuxTrOYb0AYF_Vv7kzOztn_Phi4TEKtfWPEEkfRyg@mail.gmail.com>
> <CAJSP0QWw8SyENL=eMpFstxr9kG038CVs5GKY3qZ+V1foxDmykg@mail.gmail.com>
> <55ADE316.1060704@siemens.com>
> <20150721094929.GD13993@stefanha-thinkpad.redhat.com>
> <20150727101305.GD30018@stefanha-thinkpad.redhat.com>
> <55B63327.5080205@siemens.com> <20150728221234.GI3467@type.home>
> <55B8826F.1090305@siemens.com>
> In-Reply-To: <55B8826F.1090305@siemens.com>
> Subject: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp
> maintainer
> ...
> 

Oh you're right! I misread it and my email client displayed as a top level post
too, no idea why...

Fam

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Li Zhijian]

CC Wen Congyang

On 07/20/2015 06:32 PM, Stefan Hajnoczi wrote:
> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>> We are planning to implement colo-proxy in qemu to cache and compare packets.
> I thought there is a kernel module to do that?
>
> Why does the proxy need to be part of the QEMU process?  -netdev socket
> or host network stack features allow you to process packets in a
> separate process.
yes, it used to be a kernel module.
we plan to re-implement a QEMU space colo-proxy by the following reasons:
1. colo-proxy in kernel was based on netfilter, it was
impletmented by add a new nf_ct_ext_id, but this will
touch the existed kernel code and we must re-build the kernel before
we install the colo-proxy modules. For this reason, less people
is like to test colo-proxy and it become harder to post to kenel
2. COLO is the only scene of colo-proxy in kernel
3. colo-proxy in kernel only works on the case where packet will deliver to
kernel tcp/ip stack.

COLO project is mainly including 3 components, COLO-Frame COLO-Block and 
COLO-Proxy.
The first tow components is being post to QEMU, if we integrate proxy 
into QEMU,
it will become convenienter to manage the whole COLO project.
further more, COLO will become easier to configure without depending on 
kernel


> Without details on what the proxy does it's hard to discuss this.  What
> happens in the non-TCP case?  What happens in the TCP case?
more details will be post soon


> Does the proxy need to perform privileged operations, create sockets,
> open files, etc?
IMO, we just need to create a new socket like the migration socket to
forward packet between PVM and SVM.

Best regards.
Li Zhijian

> The slirp code is not actively developed or used much in production.  It
> might be a good idea to audit the code for bugs if you want to use it.
>
> Stefan

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dong, Eddie]

Hi Stefan:
	Thanks for your comments!

> 
> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
> > We are planning to implement colo-proxy in qemu to cache and compare
> packets.
> 
> I thought there is a kernel module to do that?

	Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :)  More comments are welcome.

> 
> Why does the proxy need to be part of the QEMU process?  -netdev socket or
> host network stack features allow you to process packets in a separate process.

	Hailiang did a very good summary, and we don't need privilege ops so far. The main thing that motivated us to revisit is because the former kernel land driver would be a pure virtualization (and for high availability) component. It may have limited user only at very beginning and thus slower to be accepted. And, people typically like to put component to be in user land as if it can. 

	In addition, as a pure virtualization feature, we guess people in Qemu mailing list may be much more interested to support all the VMMs, such as pure Qemu VMM, KVM, etc. 


Thx Eddie 

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/24/2015 10:04 AM, Dong, Eddie wrote:
> Hi Stefan:
> 	Thanks for your comments!
>
>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>> We are planning to implement colo-proxy in qemu to cache and compare
>> packets.
>>
>> I thought there is a kernel module to do that?
> 	Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :)  More comments are welcome.
>

Hi:

Could you pls describe more details on this decision? What's the reason
that you realize it was not the best choice?

Thanks

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

Hi Jason,

On 07/24/2015 10:12 AM, Jason Wang wrote:
>
>
> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>> Hi Stefan:
>> 	Thanks for your comments!
>>
>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>> We are planning to implement colo-proxy in qemu to cache and compare
>>> packets.
>>>
>>> I thought there is a kernel module to do that?
>> 	Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :)  More comments are welcome.
>>
>
> Hi:
>
> Could you pls describe more details on this decision? What's the reason
> that you realize it was not the best choice?

Below is my opinion:

We realized that there're disadvantages do it in kernel spaces:
1. We need to recompile kernel: the colo-proxy kernel module is
    implemented as a nf conntrack extension. Adding a extension need to
    modify the extension struct in-kernel, so recompile kernel is needed.
2. We need to recompile iptables/nftables to use together with the colo-proxy
    kernel module.
3. Need to configure primary host to forward input packets to secondary as
    well as configure secondary to forward output packets to primary host, the
    network topology and configuration is too complex for a regular user.

You can refer to http://wiki.qemu.org/Features/COLO
to see the network topology and the steps to setup an env.

Setup a test env is too complex. The usability is so important to a feature
like COLO which provide VM FT solution, if fewer people can/willing to
setup the env, the feature is useless. So we decide to develop user space
colo-proxy.

The advantage is obvious,
1. we do not need to recompile kernel.
2. No need to recompile iptables/nftables.
3. we do not need to deal with the network configuration, we just using a
    socket connection between 2 QEMUs to forward packets.
4. A complete VM FT solution in one go, we have already developed the block
    replication in QEMU, so with the network replication in QEMU, all
    components we needed are within QEMU, this is very important, it greatly
    improves the usability of COLO feature! We hope it will gain more testers,
    users and developers.
5. QEMU will gain a complete VM FT solution and the most advantage FT solution
    so far!

Overall, usability is the most important factor that impact our choice.


>
> Thanks
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/24/2015 04:04 PM, Yang Hongyang wrote:
> Hi Jason,
>
> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>
>>
>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>> Hi Stefan:
>>>     Thanks for your comments!
>>>
>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>> We are planning to implement colo-proxy in qemu to cache and compare
>>>> packets.
>>>>
>>>> I thought there is a kernel module to do that?
>>>     Yes, that is the previous solution the COLO sub-community choose
>>> to go, but we realized it might be not the best choices, and thus we
>>> want to bring discussion back here :)  More comments are welcome.
>>>
>>
>> Hi:
>>
>> Could you pls describe more details on this decision? What's the reason
>> that you realize it was not the best choice?
>
> Below is my opinion:
>
> We realized that there're disadvantages do it in kernel spaces:
> 1. We need to recompile kernel: the colo-proxy kernel module is
>    implemented as a nf conntrack extension. Adding a extension need to
>    modify the extension struct in-kernel, so recompile kernel is needed.

There's no need to do all in kernel, you can use a separate process to
do the comparing and trigger the state sync through monitor.

> 2. We need to recompile iptables/nftables to use together with the
> colo-proxy
>    kernel module.
> 3. Need to configure primary host to forward input packets to
> secondary as
>    well as configure secondary to forward output packets to primary
> host, the
>    network topology and configuration is too complex for a regular user.
>

You can use current kernel primitives to mirror the traffic of both PVM
and SVM to another process without any modification of kernel. And qemu
can offload all network configuration to management in this case.  And
what's more import, this works for vhost. Filtering in qemu won't work
for vhost.


> You can refer to http://wiki.qemu.org/Features/COLO
> to see the network topology and the steps to setup an env.

The figure "COLO Framework" shows there's a proxy kernel module in
primary node but in secondary node this is done through a process? This
will complicate the environment a bit more.

>
> Setup a test env is too complex. The usability is so important to a
> feature
> like COLO which provide VM FT solution, if fewer people can/willing to
> setup the env, the feature is useless. So we decide to develop user space
> colo-proxy.

If the setup is too complex, need to consider to simplify or reuse codes
and designs. Otherwise you probably introduce something new that needs
fault tolerance.

>
> The advantage is obvious,
> 1. we do not need to recompile kernel.
> 2. No need to recompile iptables/nftables.

As I descried above, looks like there's no need to modify kernel.

> 3. we do not need to deal with the network configuration, we just using a
>    socket connection between 2 QEMUs to forward packets.

All network configurations should be offloaded to management. And you
still need a dedicated topology according to the wiki.

> 4. A complete VM FT solution in one go, we have already developed the
> block
>    replication in QEMU, so with the network replication in QEMU, all
>    components we needed are within QEMU, this is very important, it
> greatly
>    improves the usability of COLO feature! We hope it will gain more
> testers,
>    users and developers.

Is your block solution works for vhost?

> 5. QEMU will gain a complete VM FT solution and the most advantage FT
> solution
>    so far!
>
> Overall, usability is the most important factor that impact our choice.
>
>

Usability will be improved if you can use exist primitives and decouple
unnecessary codes from qemu.

Thanks

>>
>> Thanks
>> .
>>
>

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/27/2015 11:24 AM, Jason Wang wrote:
>
>
> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>> Hi Jason,
>>
>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>
>>>
>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>> Hi Stefan:
>>>>      Thanks for your comments!
>>>>
>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>> We are planning to implement colo-proxy in qemu to cache and compare
>>>>> packets.
>>>>>
>>>>> I thought there is a kernel module to do that?
>>>>      Yes, that is the previous solution the COLO sub-community choose
>>>> to go, but we realized it might be not the best choices, and thus we
>>>> want to bring discussion back here :)  More comments are welcome.
>>>>
>>>
>>> Hi:
>>>
>>> Could you pls describe more details on this decision? What's the reason
>>> that you realize it was not the best choice?
>>
>> Below is my opinion:
>>
>> We realized that there're disadvantages do it in kernel spaces:
>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>     implemented as a nf conntrack extension. Adding a extension need to
>>     modify the extension struct in-kernel, so recompile kernel is needed.
>
> There's no need to do all in kernel, you can use a separate process to
> do the comparing and trigger the state sync through monitor.

I don't get it, colo-proxy kernel module using a kthread do the comparing and
trigger the state sync. We implemented it as a nf conntrack extension module,
so we need to extend the extension struct in-kernel, although it just needs
few lines changes to kernel, but a recompile of kernel is needed. Are you
talking about not implement it as a nf conntrack extension?

>
>> 2. We need to recompile iptables/nftables to use together with the
>> colo-proxy
>>     kernel module.
>> 3. Need to configure primary host to forward input packets to
>> secondary as
>>     well as configure secondary to forward output packets to primary
>> host, the
>>     network topology and configuration is too complex for a regular user.
>>
>
> You can use current kernel primitives to mirror the traffic of both PVM
> and SVM to another process without any modification of kernel. And qemu
> can offload all network configuration to management in this case.  And
> what's more import, this works for vhost. Filtering in qemu won't work
> for vhost.

We are using tc to mirror/forward packets now. Implement in QEMU do have some
limits, but there're also limits in kernel, if the packet do not pass
the host kernel TCP/IP stack, such as vhost-user.

>
>
>> You can refer to http://wiki.qemu.org/Features/COLO
>> to see the network topology and the steps to setup an env.
>
> The figure "COLO Framework" shows there's a proxy kernel module in
> primary node but in secondary node this is done through a process? This
> will complicate the environment a bit more.

proxy kernel module also works for secondary node.

>
>>
>> Setup a test env is too complex. The usability is so important to a
>> feature
>> like COLO which provide VM FT solution, if fewer people can/willing to
>> setup the env, the feature is useless. So we decide to develop user space
>> colo-proxy.
>
> If the setup is too complex, need to consider to simplify or reuse codes
> and designs. Otherwise you probably introduce something new that needs
> fault tolerance.
>
>>
>> The advantage is obvious,
>> 1. we do not need to recompile kernel.
>> 2. No need to recompile iptables/nftables.
>
> As I descried above, looks like there's no need to modify kernel.
>
>> 3. we do not need to deal with the network configuration, we just using a
>>     socket connection between 2 QEMUs to forward packets.
>
> All network configurations should be offloaded to management. And you
> still need a dedicated topology according to the wiki.
>
>> 4. A complete VM FT solution in one go, we have already developed the
>> block
>>     replication in QEMU, so with the network replication in QEMU, all
>>     components we needed are within QEMU, this is very important, it
>> greatly
>>     improves the usability of COLO feature! We hope it will gain more
>> testers,
>>     users and developers.
>
> Is your block solution works for vhost?

No, it can't works for vhost and dataplane, migration also won't work
for dataplane IIRC.

>
>> 5. QEMU will gain a complete VM FT solution and the most advantage FT
>> solution
>>     so far!
>>
>> Overall, usability is the most important factor that impact our choice.
>>
>>
>
> Usability will be improved if you can use exist primitives and decouple
> unnecessary codes from qemu.
>
> Thanks
>
>>>
>>> Thanks
>>> .
>>>
>>
>
>
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>
>
> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>
>>
>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>> Hi Jason,
>>>
>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>
>>>>
>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>> Hi Stefan:
>>>>>      Thanks for your comments!
>>>>>
>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>> compare
>>>>>> packets.
>>>>>>
>>>>>> I thought there is a kernel module to do that?
>>>>>      Yes, that is the previous solution the COLO sub-community choose
>>>>> to go, but we realized it might be not the best choices, and thus we
>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>
>>>>
>>>> Hi:
>>>>
>>>> Could you pls describe more details on this decision? What's the
>>>> reason
>>>> that you realize it was not the best choice?
>>>
>>> Below is my opinion:
>>>
>>> We realized that there're disadvantages do it in kernel spaces:
>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>     implemented as a nf conntrack extension. Adding a extension need to
>>>     modify the extension struct in-kernel, so recompile kernel is
>>> needed.
>>
>> There's no need to do all in kernel, you can use a separate process to
>> do the comparing and trigger the state sync through monitor.
>
> I don't get it, colo-proxy kernel module using a kthread do the
> comparing and
> trigger the state sync. We implemented it as a nf conntrack extension
> module,
> so we need to extend the extension struct in-kernel, although it just
> needs
> few lines changes to kernel, but a recompile of kernel is needed. Are you
> talking about not implement it as a nf conntrack extension?

Yes, I mean implement the comparing in userspace but not in qemu.

>
>>
>>> 2. We need to recompile iptables/nftables to use together with the
>>> colo-proxy
>>>     kernel module.
>>> 3. Need to configure primary host to forward input packets to
>>> secondary as
>>>     well as configure secondary to forward output packets to primary
>>> host, the
>>>     network topology and configuration is too complex for a regular
>>> user.
>>>
>>
>> You can use current kernel primitives to mirror the traffic of both PVM
>> and SVM to another process without any modification of kernel. And qemu
>> can offload all network configuration to management in this case.  And
>> what's more import, this works for vhost. Filtering in qemu won't work
>> for vhost.
>
> We are using tc to mirror/forward packets now. Implement in QEMU do
> have some
> limits, but there're also limits in kernel, if the packet do not pass
> the host kernel TCP/IP stack, such as vhost-user.

But the limits are much less than userspace, no? For vhost-user, maybe
we could extend the backed to mirror the traffic also.

>
>>
>>
>>> You can refer to http://wiki.qemu.org/Features/COLO
>>> to see the network topology and the steps to setup an env.
>>
>> The figure "COLO Framework" shows there's a proxy kernel module in
>> primary node but in secondary node this is done through a process? This
>> will complicate the environment a bit more.
>
> proxy kernel module also works for secondary node.
>
>>
>>>
>>> Setup a test env is too complex. The usability is so important to a
>>> feature
>>> like COLO which provide VM FT solution, if fewer people can/willing to
>>> setup the env, the feature is useless. So we decide to develop user
>>> space
>>> colo-proxy.
>>
>> If the setup is too complex, need to consider to simplify or reuse codes
>> and designs. Otherwise you probably introduce something new that needs
>> fault tolerance.
>>
>>>
>>> The advantage is obvious,
>>> 1. we do not need to recompile kernel.
>>> 2. No need to recompile iptables/nftables.
>>
>> As I descried above, looks like there's no need to modify kernel.
>>
>>> 3. we do not need to deal with the network configuration, we just
>>> using a
>>>     socket connection between 2 QEMUs to forward packets.
>>
>> All network configurations should be offloaded to management. And you
>> still need a dedicated topology according to the wiki.
>>
>>> 4. A complete VM FT solution in one go, we have already developed the
>>> block
>>>     replication in QEMU, so with the network replication in QEMU, all
>>>     components we needed are within QEMU, this is very important, it
>>> greatly
>>>     improves the usability of COLO feature! We hope it will gain more
>>> testers,
>>>     users and developers.
>>
>> Is your block solution works for vhost?
>
> No, it can't works for vhost and dataplane, migration also won't work
> for dataplane IIRC.
>
>>
>>> 5. QEMU will gain a complete VM FT solution and the most advantage FT
>>> solution
>>>     so far!
>>>
>>> Overall, usability is the most important factor that impact our choice.
>>>
>>>
>>
>> Usability will be improved if you can use exist primitives and decouple
>> unnecessary codes from qemu.
>>
>> Thanks
>>
>>>>
>>>> Thanks
>>>> .
>>>>
>>>
>>
>>
>> .
>>
>

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/27/2015 12:49 PM, Jason Wang wrote:
>
>
> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>>
>>
>> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>>
>>>
>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>>> Hi Jason,
>>>>
>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>>
>>>>>
>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>>> Hi Stefan:
>>>>>>       Thanks for your comments!
>>>>>>
>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>>> compare
>>>>>>> packets.
>>>>>>>
>>>>>>> I thought there is a kernel module to do that?
>>>>>>       Yes, that is the previous solution the COLO sub-community choose
>>>>>> to go, but we realized it might be not the best choices, and thus we
>>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>>
>>>>>
>>>>> Hi:
>>>>>
>>>>> Could you pls describe more details on this decision? What's the
>>>>> reason
>>>>> that you realize it was not the best choice?
>>>>
>>>> Below is my opinion:
>>>>
>>>> We realized that there're disadvantages do it in kernel spaces:
>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>>      implemented as a nf conntrack extension. Adding a extension need to
>>>>      modify the extension struct in-kernel, so recompile kernel is
>>>> needed.
>>>
>>> There's no need to do all in kernel, you can use a separate process to
>>> do the comparing and trigger the state sync through monitor.
>>
>> I don't get it, colo-proxy kernel module using a kthread do the
>> comparing and
>> trigger the state sync. We implemented it as a nf conntrack extension
>> module,
>> so we need to extend the extension struct in-kernel, although it just
>> needs
>> few lines changes to kernel, but a recompile of kernel is needed. Are you
>> talking about not implement it as a nf conntrack extension?
>
> Yes, I mean implement the comparing in userspace but not in qemu.

Yes, it is an alternative, that requires other components such as
netfilter userspace tools, it will add the complexity I think, we
wanted to implement a simple solution in QEMU. Another reason is
that using other userspace tools will affect the performance, the
context switch between kernel and userspace may be an overhead.

>
>>
>>>
>>>> 2. We need to recompile iptables/nftables to use together with the
>>>> colo-proxy
>>>>      kernel module.
>>>> 3. Need to configure primary host to forward input packets to
>>>> secondary as
>>>>      well as configure secondary to forward output packets to primary
>>>> host, the
>>>>      network topology and configuration is too complex for a regular
>>>> user.
>>>>
>>>
>>> You can use current kernel primitives to mirror the traffic of both PVM
>>> and SVM to another process without any modification of kernel. And qemu
>>> can offload all network configuration to management in this case.  And
>>> what's more import, this works for vhost. Filtering in qemu won't work
>>> for vhost.
>>
>> We are using tc to mirror/forward packets now. Implement in QEMU do
>> have some
>> limits, but there're also limits in kernel, if the packet do not pass
>> the host kernel TCP/IP stack, such as vhost-user.
>
> But the limits are much less than userspace, no? For vhost-user, maybe
> we could extend the backed to mirror the traffic also.

IMO the limits are more or less. Besides, for mirror/forward packets,
using tc requires a separate physical nic or a vlan, the nic should not
be used for other purpose. if we implement it in QEMU, using an socket
connection to forward packets, we no longer need an separate nic, it will
reduce the network topology complexity.


>
>>
>>>
>>>
>>>> You can refer to http://wiki.qemu.org/Features/COLO
>>>> to see the network topology and the steps to setup an env.
>>>
>>> The figure "COLO Framework" shows there's a proxy kernel module in
>>> primary node but in secondary node this is done through a process? This
>>> will complicate the environment a bit more.
>>
>> proxy kernel module also works for secondary node.
>>
>>>
>>>>
>>>> Setup a test env is too complex. The usability is so important to a
>>>> feature
>>>> like COLO which provide VM FT solution, if fewer people can/willing to
>>>> setup the env, the feature is useless. So we decide to develop user
>>>> space
>>>> colo-proxy.
>>>
>>> If the setup is too complex, need to consider to simplify or reuse codes
>>> and designs. Otherwise you probably introduce something new that needs
>>> fault tolerance.
>>>
>>>>
>>>> The advantage is obvious,
>>>> 1. we do not need to recompile kernel.
>>>> 2. No need to recompile iptables/nftables.
>>>
>>> As I descried above, looks like there's no need to modify kernel.
>>>
>>>> 3. we do not need to deal with the network configuration, we just
>>>> using a
>>>>      socket connection between 2 QEMUs to forward packets.
>>>
>>> All network configurations should be offloaded to management. And you
>>> still need a dedicated topology according to the wiki.
>>>
>>>> 4. A complete VM FT solution in one go, we have already developed the
>>>> block
>>>>      replication in QEMU, so with the network replication in QEMU, all
>>>>      components we needed are within QEMU, this is very important, it
>>>> greatly
>>>>      improves the usability of COLO feature! We hope it will gain more
>>>> testers,
>>>>      users and developers.
>>>
>>> Is your block solution works for vhost?
>>
>> No, it can't works for vhost and dataplane, migration also won't work
>> for dataplane IIRC.
>>
>>>
>>>> 5. QEMU will gain a complete VM FT solution and the most advantage FT
>>>> solution
>>>>      so far!
>>>>
>>>> Overall, usability is the most important factor that impact our choice.
>>>>
>>>>
>>>
>>> Usability will be improved if you can use exist primitives and decouple
>>> unnecessary codes from qemu.
>>>
>>> Thanks
>>>
>>>>>
>>>>> Thanks
>>>>> .
>>>>>
>>>>
>>>
>>>
>>> .
>>>
>>
>
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/27/2015 01:51 PM, Yang Hongyang wrote:
> On 07/27/2015 12:49 PM, Jason Wang wrote:
>>
>>
>> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>>>
>>>
>>> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>>>
>>>>
>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>>>> Hi Jason,
>>>>>
>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>>>
>>>>>>
>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>>>> Hi Stefan:
>>>>>>>       Thanks for your comments!
>>>>>>>
>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>>>> compare
>>>>>>>> packets.
>>>>>>>>
>>>>>>>> I thought there is a kernel module to do that?
>>>>>>>       Yes, that is the previous solution the COLO sub-community
>>>>>>> choose
>>>>>>> to go, but we realized it might be not the best choices, and
>>>>>>> thus we
>>>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>>>
>>>>>>
>>>>>> Hi:
>>>>>>
>>>>>> Could you pls describe more details on this decision? What's the
>>>>>> reason
>>>>>> that you realize it was not the best choice?
>>>>>
>>>>> Below is my opinion:
>>>>>
>>>>> We realized that there're disadvantages do it in kernel spaces:
>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>>>      implemented as a nf conntrack extension. Adding a extension
>>>>> need to
>>>>>      modify the extension struct in-kernel, so recompile kernel is
>>>>> needed.
>>>>
>>>> There's no need to do all in kernel, you can use a separate process to
>>>> do the comparing and trigger the state sync through monitor.
>>>
>>> I don't get it, colo-proxy kernel module using a kthread do the
>>> comparing and
>>> trigger the state sync. We implemented it as a nf conntrack extension
>>> module,
>>> so we need to extend the extension struct in-kernel, although it just
>>> needs
>>> few lines changes to kernel, but a recompile of kernel is needed.
>>> Are you
>>> talking about not implement it as a nf conntrack extension?
>>
>> Yes, I mean implement the comparing in userspace but not in qemu.
>
> Yes, it is an alternative, that requires other components such as
> netfilter userspace tools, it will add the complexity I think, we
> wanted to implement a simple solution in QEMU. Another reason is
> that using other userspace tools will affect the performance, the
> context switch between kernel and userspace may be an overhead.
>
>>
>>>
>>>>
>>>>> 2. We need to recompile iptables/nftables to use together with the
>>>>> colo-proxy
>>>>>      kernel module.
>>>>> 3. Need to configure primary host to forward input packets to
>>>>> secondary as
>>>>>      well as configure secondary to forward output packets to primary
>>>>> host, the
>>>>>      network topology and configuration is too complex for a regular
>>>>> user.
>>>>>
>>>>
>>>> You can use current kernel primitives to mirror the traffic of both
>>>> PVM
>>>> and SVM to another process without any modification of kernel. And
>>>> qemu
>>>> can offload all network configuration to management in this case.  And
>>>> what's more import, this works for vhost. Filtering in qemu won't work
>>>> for vhost.
>>>
>>> We are using tc to mirror/forward packets now. Implement in QEMU do
>>> have some
>>> limits, but there're also limits in kernel, if the packet do not pass
>>> the host kernel TCP/IP stack, such as vhost-user.
>>
>> But the limits are much less than userspace, no? For vhost-user, maybe
>> we could extend the backed to mirror the traffic also.
>
> IMO the limits are more or less. Besides, for mirror/forward packets,
> using tc requires a separate physical nic or a vlan, the nic should not
> be used for other purpose. if we implement it in QEMU, using an socket
> connection to forward packets, we no longer need an separate nic, it will
> reduce the network topology complexity.

It depends on how do you design your user space. If you want using
userspace to forward the packet, you can 1) use packet socket to capture
all traffic on the tap that is used by VM 2) mirror the traffic to a new
tap device, the user space can then read all traffic from this new tap
device.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/27/2015 03:37 PM, Jason Wang wrote:
>
>
> On 07/27/2015 01:51 PM, Yang Hongyang wrote:
>> On 07/27/2015 12:49 PM, Jason Wang wrote:
>>>
>>>
>>> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>>>>
>>>>
>>>> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>>>>
>>>>>
>>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>>>>> Hi Jason,
>>>>>>
>>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>>>>> Hi Stefan:
>>>>>>>>        Thanks for your comments!
>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>>>>> compare
>>>>>>>>> packets.
>>>>>>>>>
>>>>>>>>> I thought there is a kernel module to do that?
>>>>>>>>        Yes, that is the previous solution the COLO sub-community
>>>>>>>> choose
>>>>>>>> to go, but we realized it might be not the best choices, and
>>>>>>>> thus we
>>>>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>>>>
>>>>>>>
>>>>>>> Hi:
>>>>>>>
>>>>>>> Could you pls describe more details on this decision? What's the
>>>>>>> reason
>>>>>>> that you realize it was not the best choice?
>>>>>>
>>>>>> Below is my opinion:
>>>>>>
>>>>>> We realized that there're disadvantages do it in kernel spaces:
>>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>>>>       implemented as a nf conntrack extension. Adding a extension
>>>>>> need to
>>>>>>       modify the extension struct in-kernel, so recompile kernel is
>>>>>> needed.
>>>>>
>>>>> There's no need to do all in kernel, you can use a separate process to
>>>>> do the comparing and trigger the state sync through monitor.
>>>>
>>>> I don't get it, colo-proxy kernel module using a kthread do the
>>>> comparing and
>>>> trigger the state sync. We implemented it as a nf conntrack extension
>>>> module,
>>>> so we need to extend the extension struct in-kernel, although it just
>>>> needs
>>>> few lines changes to kernel, but a recompile of kernel is needed.
>>>> Are you
>>>> talking about not implement it as a nf conntrack extension?
>>>
>>> Yes, I mean implement the comparing in userspace but not in qemu.
>>
>> Yes, it is an alternative, that requires other components such as
>> netfilter userspace tools, it will add the complexity I think, we
>> wanted to implement a simple solution in QEMU. Another reason is
>> that using other userspace tools will affect the performance, the
>> context switch between kernel and userspace may be an overhead.
>>
>>>
>>>>
>>>>>
>>>>>> 2. We need to recompile iptables/nftables to use together with the
>>>>>> colo-proxy
>>>>>>       kernel module.
>>>>>> 3. Need to configure primary host to forward input packets to
>>>>>> secondary as
>>>>>>       well as configure secondary to forward output packets to primary
>>>>>> host, the
>>>>>>       network topology and configuration is too complex for a regular
>>>>>> user.
>>>>>>
>>>>>
>>>>> You can use current kernel primitives to mirror the traffic of both
>>>>> PVM
>>>>> and SVM to another process without any modification of kernel. And
>>>>> qemu
>>>>> can offload all network configuration to management in this case.  And
>>>>> what's more import, this works for vhost. Filtering in qemu won't work
>>>>> for vhost.
>>>>
>>>> We are using tc to mirror/forward packets now. Implement in QEMU do
>>>> have some
>>>> limits, but there're also limits in kernel, if the packet do not pass
>>>> the host kernel TCP/IP stack, such as vhost-user.
>>>
>>> But the limits are much less than userspace, no? For vhost-user, maybe
>>> we could extend the backed to mirror the traffic also.
>>
>> IMO the limits are more or less. Besides, for mirror/forward packets,
>> using tc requires a separate physical nic or a vlan, the nic should not
>> be used for other purpose. if we implement it in QEMU, using an socket
>> connection to forward packets, we no longer need an separate nic, it will
>> reduce the network topology complexity.
>
> It depends on how do you design your user space. If you want using
> userspace to forward the packet, you can 1) use packet socket to capture
> all traffic on the tap that is used by VM 2) mirror the traffic to a new
> tap device, the user space can then read all traffic from this new tap
> device.

Yes, but we can also do it in QEMU space, right? This will make life easier
because we do all in one solution within QEMU.

>
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/27/2015 03:49 PM, Yang Hongyang wrote:
> On 07/27/2015 03:37 PM, Jason Wang wrote:
>>
>>
>> On 07/27/2015 01:51 PM, Yang Hongyang wrote:
>>> On 07/27/2015 12:49 PM, Jason Wang wrote:
>>>>
>>>>
>>>> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>>>>>
>>>>>
>>>>> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>>>>>
>>>>>>
>>>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>>>>>> Hi Jason,
>>>>>>>
>>>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>>>>>> Hi Stefan:
>>>>>>>>>        Thanks for your comments!
>>>>>>>>>
>>>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>>>>>> compare
>>>>>>>>>> packets.
>>>>>>>>>>
>>>>>>>>>> I thought there is a kernel module to do that?
>>>>>>>>>        Yes, that is the previous solution the COLO sub-community
>>>>>>>>> choose
>>>>>>>>> to go, but we realized it might be not the best choices, and
>>>>>>>>> thus we
>>>>>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi:
>>>>>>>>
>>>>>>>> Could you pls describe more details on this decision? What's the
>>>>>>>> reason
>>>>>>>> that you realize it was not the best choice?
>>>>>>>
>>>>>>> Below is my opinion:
>>>>>>>
>>>>>>> We realized that there're disadvantages do it in kernel spaces:
>>>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>>>>>       implemented as a nf conntrack extension. Adding a extension
>>>>>>> need to
>>>>>>>       modify the extension struct in-kernel, so recompile kernel is
>>>>>>> needed.
>>>>>>
>>>>>> There's no need to do all in kernel, you can use a separate
>>>>>> process to
>>>>>> do the comparing and trigger the state sync through monitor.
>>>>>
>>>>> I don't get it, colo-proxy kernel module using a kthread do the
>>>>> comparing and
>>>>> trigger the state sync. We implemented it as a nf conntrack extension
>>>>> module,
>>>>> so we need to extend the extension struct in-kernel, although it just
>>>>> needs
>>>>> few lines changes to kernel, but a recompile of kernel is needed.
>>>>> Are you
>>>>> talking about not implement it as a nf conntrack extension?
>>>>
>>>> Yes, I mean implement the comparing in userspace but not in qemu.
>>>
>>> Yes, it is an alternative, that requires other components such as
>>> netfilter userspace tools, it will add the complexity I think, we
>>> wanted to implement a simple solution in QEMU. Another reason is
>>> that using other userspace tools will affect the performance, the
>>> context switch between kernel and userspace may be an overhead.
>>>
>>>>
>>>>>
>>>>>>
>>>>>>> 2. We need to recompile iptables/nftables to use together with the
>>>>>>> colo-proxy
>>>>>>>       kernel module.
>>>>>>> 3. Need to configure primary host to forward input packets to
>>>>>>> secondary as
>>>>>>>       well as configure secondary to forward output packets to
>>>>>>> primary
>>>>>>> host, the
>>>>>>>       network topology and configuration is too complex for a
>>>>>>> regular
>>>>>>> user.
>>>>>>>
>>>>>>
>>>>>> You can use current kernel primitives to mirror the traffic of both
>>>>>> PVM
>>>>>> and SVM to another process without any modification of kernel. And
>>>>>> qemu
>>>>>> can offload all network configuration to management in this
>>>>>> case.  And
>>>>>> what's more import, this works for vhost. Filtering in qemu won't
>>>>>> work
>>>>>> for vhost.
>>>>>
>>>>> We are using tc to mirror/forward packets now. Implement in QEMU do
>>>>> have some
>>>>> limits, but there're also limits in kernel, if the packet do not pass
>>>>> the host kernel TCP/IP stack, such as vhost-user.
>>>>
>>>> But the limits are much less than userspace, no? For vhost-user, maybe
>>>> we could extend the backed to mirror the traffic also.
>>>
>>> IMO the limits are more or less. Besides, for mirror/forward packets,
>>> using tc requires a separate physical nic or a vlan, the nic should not
>>> be used for other purpose. if we implement it in QEMU, using an socket
>>> connection to forward packets, we no longer need an separate nic, it
>>> will
>>> reduce the network topology complexity.
>>
>> It depends on how do you design your user space. If you want using
>> userspace to forward the packet, you can 1) use packet socket to capture
>> all traffic on the tap that is used by VM 2) mirror the traffic to a new
>> tap device, the user space can then read all traffic from this new tap
>> device.
>
> Yes, but we can also do it in QEMU space, right? 

Right.

> This will make life easier
> because we do all in one solution within QEMU.

But I'm not sure qemu is the right place to do this as you mention that
it needs userspace protocol stack support.

>
>>
>> .
>>
>

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/27/2015 04:06 PM, Jason Wang wrote:
>
>
> On 07/27/2015 03:49 PM, Yang Hongyang wrote:
>> On 07/27/2015 03:37 PM, Jason Wang wrote:
>>>
>>>
>>> On 07/27/2015 01:51 PM, Yang Hongyang wrote:
>>>> On 07/27/2015 12:49 PM, Jason Wang wrote:
>>>>>
>>>>>
>>>>> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>>>>>>
>>>>>>
>>>>>> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>>>>>>> Hi Jason,
>>>>>>>>
>>>>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>>>>>>> Hi Stefan:
>>>>>>>>>>         Thanks for your comments!
>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>>>>>>> compare
>>>>>>>>>>> packets.
>>>>>>>>>>>
>>>>>>>>>>> I thought there is a kernel module to do that?
>>>>>>>>>>         Yes, that is the previous solution the COLO sub-community
>>>>>>>>>> choose
>>>>>>>>>> to go, but we realized it might be not the best choices, and
>>>>>>>>>> thus we
>>>>>>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi:
>>>>>>>>>
>>>>>>>>> Could you pls describe more details on this decision? What's the
>>>>>>>>> reason
>>>>>>>>> that you realize it was not the best choice?
>>>>>>>>
>>>>>>>> Below is my opinion:
>>>>>>>>
>>>>>>>> We realized that there're disadvantages do it in kernel spaces:
>>>>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>>>>>>        implemented as a nf conntrack extension. Adding a extension
>>>>>>>> need to
>>>>>>>>        modify the extension struct in-kernel, so recompile kernel is
>>>>>>>> needed.
>>>>>>>
>>>>>>> There's no need to do all in kernel, you can use a separate
>>>>>>> process to
>>>>>>> do the comparing and trigger the state sync through monitor.
>>>>>>
>>>>>> I don't get it, colo-proxy kernel module using a kthread do the
>>>>>> comparing and
>>>>>> trigger the state sync. We implemented it as a nf conntrack extension
>>>>>> module,
>>>>>> so we need to extend the extension struct in-kernel, although it just
>>>>>> needs
>>>>>> few lines changes to kernel, but a recompile of kernel is needed.
>>>>>> Are you
>>>>>> talking about not implement it as a nf conntrack extension?
>>>>>
>>>>> Yes, I mean implement the comparing in userspace but not in qemu.
>>>>
>>>> Yes, it is an alternative, that requires other components such as
>>>> netfilter userspace tools, it will add the complexity I think, we
>>>> wanted to implement a simple solution in QEMU. Another reason is
>>>> that using other userspace tools will affect the performance, the
>>>> context switch between kernel and userspace may be an overhead.
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>> 2. We need to recompile iptables/nftables to use together with the
>>>>>>>> colo-proxy
>>>>>>>>        kernel module.
>>>>>>>> 3. Need to configure primary host to forward input packets to
>>>>>>>> secondary as
>>>>>>>>        well as configure secondary to forward output packets to
>>>>>>>> primary
>>>>>>>> host, the
>>>>>>>>        network topology and configuration is too complex for a
>>>>>>>> regular
>>>>>>>> user.
>>>>>>>>
>>>>>>>
>>>>>>> You can use current kernel primitives to mirror the traffic of both
>>>>>>> PVM
>>>>>>> and SVM to another process without any modification of kernel. And
>>>>>>> qemu
>>>>>>> can offload all network configuration to management in this
>>>>>>> case.  And
>>>>>>> what's more import, this works for vhost. Filtering in qemu won't
>>>>>>> work
>>>>>>> for vhost.
>>>>>>
>>>>>> We are using tc to mirror/forward packets now. Implement in QEMU do
>>>>>> have some
>>>>>> limits, but there're also limits in kernel, if the packet do not pass
>>>>>> the host kernel TCP/IP stack, such as vhost-user.
>>>>>
>>>>> But the limits are much less than userspace, no? For vhost-user, maybe
>>>>> we could extend the backed to mirror the traffic also.
>>>>
>>>> IMO the limits are more or less. Besides, for mirror/forward packets,
>>>> using tc requires a separate physical nic or a vlan, the nic should not
>>>> be used for other purpose. if we implement it in QEMU, using an socket
>>>> connection to forward packets, we no longer need an separate nic, it
>>>> will
>>>> reduce the network topology complexity.
>>>
>>> It depends on how do you design your user space. If you want using
>>> userspace to forward the packet, you can 1) use packet socket to capture
>>> all traffic on the tap that is used by VM 2) mirror the traffic to a new
>>> tap device, the user space can then read all traffic from this new tap
>>> device.
>>
>> Yes, but we can also do it in QEMU space, right?
>
> Right.
>
>> This will make life easier
>> because we do all in one solution within QEMU.
>
> But I'm not sure qemu is the right place to do this as you mention that
> it needs userspace protocol stack support.

We only need some simple features like defragment of TCP packets, analyze
TCP headers, since QEMU has a slirp userspace protocol stack, that should
not be a big deal.

>
>>
>>>
>>> .
>>>
>>
>
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/27/2015 01:51 PM, Yang Hongyang wrote:
> On 07/27/2015 12:49 PM, Jason Wang wrote:
>>
>>
>> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>>>
>>>
>>> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>>>
>>>>
>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>>>> Hi Jason,
>>>>>
>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>>>
>>>>>>
>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>>>> Hi Stefan:
>>>>>>>       Thanks for your comments!
>>>>>>>
>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>>>> compare
>>>>>>>> packets.
>>>>>>>>
>>>>>>>> I thought there is a kernel module to do that?
>>>>>>>       Yes, that is the previous solution the COLO sub-community
>>>>>>> choose
>>>>>>> to go, but we realized it might be not the best choices, and
>>>>>>> thus we
>>>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>>>
>>>>>>
>>>>>> Hi:
>>>>>>
>>>>>> Could you pls describe more details on this decision? What's the
>>>>>> reason
>>>>>> that you realize it was not the best choice?
>>>>>
>>>>> Below is my opinion:
>>>>>
>>>>> We realized that there're disadvantages do it in kernel spaces:
>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>>>      implemented as a nf conntrack extension. Adding a extension
>>>>> need to
>>>>>      modify the extension struct in-kernel, so recompile kernel is
>>>>> needed.
>>>>
>>>> There's no need to do all in kernel, you can use a separate process to
>>>> do the comparing and trigger the state sync through monitor.
>>>
>>> I don't get it, colo-proxy kernel module using a kthread do the
>>> comparing and
>>> trigger the state sync. We implemented it as a nf conntrack extension
>>> module,
>>> so we need to extend the extension struct in-kernel, although it just
>>> needs
>>> few lines changes to kernel, but a recompile of kernel is needed.
>>> Are you
>>> talking about not implement it as a nf conntrack extension?
>>
>> Yes, I mean implement the comparing in userspace but not in qemu.
>
> Yes, it is an alternative, that requires other components such as
> netfilter userspace tools, it will add the complexity I think, we
> wanted to implement a simple solution in QEMU.

I didn't get the point that why netfilter is needed? Do you mean the
packet comparing needs to be stateful?

> Another reason is
> that using other userspace tools will affect the performance, the
> context switch between kernel and userspace may be an overhead.

We can use 100% time of this process but looks like your RFC of filter
just did it in iothread?

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/27/2015 03:53 PM, Jason Wang wrote:
>
>
> On 07/27/2015 01:51 PM, Yang Hongyang wrote:
>> On 07/27/2015 12:49 PM, Jason Wang wrote:
>>>
>>>
>>> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
>>>>
>>>>
>>>> On 07/27/2015 11:24 AM, Jason Wang wrote:
>>>>>
>>>>>
>>>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
>>>>>> Hi Jason,
>>>>>>
>>>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>>>>>> Hi Stefan:
>>>>>>>>        Thanks for your comments!
>>>>>>>>
>>>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
>>>>>>>>>> compare
>>>>>>>>> packets.
>>>>>>>>>
>>>>>>>>> I thought there is a kernel module to do that?
>>>>>>>>        Yes, that is the previous solution the COLO sub-community
>>>>>>>> choose
>>>>>>>> to go, but we realized it might be not the best choices, and
>>>>>>>> thus we
>>>>>>>> want to bring discussion back here :)  More comments are welcome.
>>>>>>>>
>>>>>>>
>>>>>>> Hi:
>>>>>>>
>>>>>>> Could you pls describe more details on this decision? What's the
>>>>>>> reason
>>>>>>> that you realize it was not the best choice?
>>>>>>
>>>>>> Below is my opinion:
>>>>>>
>>>>>> We realized that there're disadvantages do it in kernel spaces:
>>>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>>>>>       implemented as a nf conntrack extension. Adding a extension
>>>>>> need to
>>>>>>       modify the extension struct in-kernel, so recompile kernel is
>>>>>> needed.
>>>>>
>>>>> There's no need to do all in kernel, you can use a separate process to
>>>>> do the comparing and trigger the state sync through monitor.
>>>>
>>>> I don't get it, colo-proxy kernel module using a kthread do the
>>>> comparing and
>>>> trigger the state sync. We implemented it as a nf conntrack extension
>>>> module,
>>>> so we need to extend the extension struct in-kernel, although it just
>>>> needs
>>>> few lines changes to kernel, but a recompile of kernel is needed.
>>>> Are you
>>>> talking about not implement it as a nf conntrack extension?
>>>
>>> Yes, I mean implement the comparing in userspace but not in qemu.
>>
>> Yes, it is an alternative, that requires other components such as
>> netfilter userspace tools, it will add the complexity I think, we
>> wanted to implement a simple solution in QEMU.
>
> I didn't get the point that why netfilter is needed? Do you mean the
> packet comparing needs to be stateful?

Yes.

>
>> Another reason is
>> that using other userspace tools will affect the performance, the
>> context switch between kernel and userspace may be an overhead.
>
> We can use 100% time of this process but looks like your RFC of filter
> just did it in iothread?

That's not the colo-proxy case, colo-proxy will require a separate
thread to do the comparing work.

>
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dr. David Alan Gilbert]

* Jason Wang (jasowang@redhat.com) wrote:
> 
> 
> On 07/27/2015 01:51 PM, Yang Hongyang wrote:
> > On 07/27/2015 12:49 PM, Jason Wang wrote:
> >>
> >>
> >> On 07/27/2015 11:54 AM, Yang Hongyang wrote:
> >>>
> >>>
> >>> On 07/27/2015 11:24 AM, Jason Wang wrote:
> >>>>
> >>>>
> >>>> On 07/24/2015 04:04 PM, Yang Hongyang wrote:
> >>>>> Hi Jason,
> >>>>>
> >>>>> On 07/24/2015 10:12 AM, Jason Wang wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
> >>>>>>> Hi Stefan:
> >>>>>>>       Thanks for your comments!
> >>>>>>>
> >>>>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
> >>>>>>>>> We are planning to implement colo-proxy in qemu to cache and
> >>>>>>>>> compare
> >>>>>>>> packets.
> >>>>>>>>
> >>>>>>>> I thought there is a kernel module to do that?
> >>>>>>>       Yes, that is the previous solution the COLO sub-community
> >>>>>>> choose
> >>>>>>> to go, but we realized it might be not the best choices, and
> >>>>>>> thus we
> >>>>>>> want to bring discussion back here :)  More comments are welcome.
> >>>>>>>
> >>>>>>
> >>>>>> Hi:
> >>>>>>
> >>>>>> Could you pls describe more details on this decision? What's the
> >>>>>> reason
> >>>>>> that you realize it was not the best choice?
> >>>>>
> >>>>> Below is my opinion:
> >>>>>
> >>>>> We realized that there're disadvantages do it in kernel spaces:
> >>>>> 1. We need to recompile kernel: the colo-proxy kernel module is
> >>>>>      implemented as a nf conntrack extension. Adding a extension
> >>>>> need to
> >>>>>      modify the extension struct in-kernel, so recompile kernel is
> >>>>> needed.
> >>>>
> >>>> There's no need to do all in kernel, you can use a separate process to
> >>>> do the comparing and trigger the state sync through monitor.
> >>>
> >>> I don't get it, colo-proxy kernel module using a kthread do the
> >>> comparing and
> >>> trigger the state sync. We implemented it as a nf conntrack extension
> >>> module,
> >>> so we need to extend the extension struct in-kernel, although it just
> >>> needs
> >>> few lines changes to kernel, but a recompile of kernel is needed.
> >>> Are you
> >>> talking about not implement it as a nf conntrack extension?
> >>
> >> Yes, I mean implement the comparing in userspace but not in qemu.
> >
> > Yes, it is an alternative, that requires other components such as
> > netfilter userspace tools, it will add the complexity I think, we
> > wanted to implement a simple solution in QEMU.
> 
> I didn't get the point that why netfilter is needed? Do you mean the
> packet comparing needs to be stateful?

The current kernel world does a few things that take advantage
of the netfilter code:
   1) It's stateful hanging state off conntrack
   2) It modifies sequence numbers off the secondary to match what the
      primary did when it created the stream.
   3) Comparison is on a per-stream basis so that the order of unrelated
      packets doesn't cause a miscompare.

Dave

> > Another reason is
> > that using other userspace tools will affect the performance, the
> > context switch between kernel and userspace may be an overhead.
> 
> We can use 100% time of this process but looks like your RFC of filter
> just did it in iothread?
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dr. David Alan Gilbert]

* Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
> Hi Jason,
> 
> On 07/24/2015 10:12 AM, Jason Wang wrote:
> >
> >
> >On 07/24/2015 10:04 AM, Dong, Eddie wrote:
> >>Hi Stefan:
> >>	Thanks for your comments!
> >>
> >>>On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
> >>>>We are planning to implement colo-proxy in qemu to cache and compare
> >>>packets.
> >>>
> >>>I thought there is a kernel module to do that?
> >>	Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :)  More comments are welcome.
> >>
> >
> >Hi:
> >
> >Could you pls describe more details on this decision? What's the reason
> >that you realize it was not the best choice?
> 
> Below is my opinion:
> 
> We realized that there're disadvantages do it in kernel spaces:
> 1. We need to recompile kernel: the colo-proxy kernel module is
>    implemented as a nf conntrack extension. Adding a extension need to
>    modify the extension struct in-kernel, so recompile kernel is needed.

That change is tiny though, so I don't think the change to the kernel
is a big issue (but I'm not a netfilter guy).

(For those following, the patch is:
https://github.com/coloft/colo-proxy/blob/master/patch4kernel/0001-colo-patch-for-kernel.patch
)
The comparison modules are bigger though, but still not massive.

> 2. We need to recompile iptables/nftables to use together with the colo-proxy
>    kernel module.

Again, the changes to iptables are small; so I don't think this should
influence it too much.

The bigger problem shown by 1&2 is that these changes are single-use - just for
COLO, which does make it a little harder to justify.

> 3. Need to configure primary host to forward input packets to secondary as
>    well as configure secondary to forward output packets to primary host, the
>    network topology and configuration is too complex for a regular user.

Yes, and that bit is HARD - it took me quite a while to get it right; however,
we'll still need to forward packets between primary and secondary, and all that
hard setup should get rolled into something like libvirt, so perhaps it's not really
that bad for the user in the end.

> You can refer to http://wiki.qemu.org/Features/COLO
> to see the network topology and the steps to setup an env.
> 
> Setup a test env is too complex. The usability is so important to a feature
> like COLO which provide VM FT solution, if fewer people can/willing to
> setup the env, the feature is useless. So we decide to develop user space
> colo-proxy.
> 
> The advantage is obvious,
> 1. we do not need to recompile kernel.
> 2. No need to recompile iptables/nftables.
> 3. we do not need to deal with the network configuration, we just using a
>    socket connection between 2 QEMUs to forward packets.
> 4. A complete VM FT solution in one go, we have already developed the block
>    replication in QEMU, so with the network replication in QEMU, all
>    components we needed are within QEMU, this is very important, it greatly
>    improves the usability of COLO feature! We hope it will gain more testers,
>    users and developers.
> 5. QEMU will gain a complete VM FT solution and the most advantage FT solution
>    so far!
> 
> Overall, usability is the most important factor that impact our choice.

My biggest worry is your reliance on SLIRP for the TCP/IP stack; it
doesn't get much work done on it and I worry about it's reliability for
using it for the level of complexity you need.

Your current kernel implementation gets all the nf_conntrack stuff for free
which is very powerful.

However, I can see some advantages from doing it in user space; it would
be easier to debug, and possibly easier to configure, and might also be easier
to handle continuous FT (i.e. transferring the state of the proxy to a new COLO
connection).

I think at the moment I'd still prefer kernel space (especially since your kernel
code now works pretty reliably!)

Another thought; if you're main worry is to do with the complexity of kernel
changes, had you considered looking at the bpf-jit - I'm not sure if it can
do what you need, but perhaps it's worth a look?

Dave
P.S. I think 'proxy' is still the right word to describe it rather than 'agency'.

> 
> 
> >
> >Thanks
> >.
> >
> 
> -- 
> Thanks,
> Yang.
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

Hi Dave,

   Thanks for the comments!

On 07/27/2015 06:40 PM, Dr. David Alan Gilbert wrote:
> * Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
>> Hi Jason,
>>
>> On 07/24/2015 10:12 AM, Jason Wang wrote:
>>>
>>>
>>> On 07/24/2015 10:04 AM, Dong, Eddie wrote:
>>>> Hi Stefan:
>>>> 	Thanks for your comments!
>>>>
>>>>> On Mon, Jul 20, 2015 at 02:42:33PM +0800, Li Zhijian wrote:
>>>>>> We are planning to implement colo-proxy in qemu to cache and compare
>>>>> packets.
>>>>>
>>>>> I thought there is a kernel module to do that?
>>>> 	Yes, that is the previous solution the COLO sub-community choose to go, but we realized it might be not the best choices, and thus we want to bring discussion back here :)  More comments are welcome.
>>>>
>>>
>>> Hi:
>>>
>>> Could you pls describe more details on this decision? What's the reason
>>> that you realize it was not the best choice?
>>
>> Below is my opinion:
>>
>> We realized that there're disadvantages do it in kernel spaces:
>> 1. We need to recompile kernel: the colo-proxy kernel module is
>>     implemented as a nf conntrack extension. Adding a extension need to
>>     modify the extension struct in-kernel, so recompile kernel is needed.
>
> That change is tiny though, so I don't think the change to the kernel
> is a big issue (but I'm not a netfilter guy).
>
> (For those following, the patch is:
> https://github.com/coloft/colo-proxy/blob/master/patch4kernel/0001-colo-patch-for-kernel.patch
> )
> The comparison modules are bigger though, but still not massive.
>
>> 2. We need to recompile iptables/nftables to use together with the colo-proxy
>>     kernel module.
>
> Again, the changes to iptables are small; so I don't think this should
> influence it too much.

Yes, these changes are small, but even a small change needs to recompile
the component and reinstall it, for user, it is not friendly...

>
> The bigger problem shown by 1&2 is that these changes are single-use - just for
> COLO, which does make it a little harder to justify.

That's true.

>
>> 3. Need to configure primary host to forward input packets to secondary as
>>     well as configure secondary to forward output packets to primary host, the
>>     network topology and configuration is too complex for a regular user.
>
> Yes, and that bit is HARD - it took me quite a while to get it right; however,
> we'll still need to forward packets between primary and secondary,

If we forward in qemu using a socket connection, a separate forward nic will not
be needed, and all tc stuff will not needed, will make configuration easier I
think.

> and all that
> hard setup should get rolled into something like libvirt, so perhaps it's not really
> that bad for the user in the end.
>
>> You can refer to http://wiki.qemu.org/Features/COLO
>> to see the network topology and the steps to setup an env.
>>
>> Setup a test env is too complex. The usability is so important to a feature
>> like COLO which provide VM FT solution, if fewer people can/willing to
>> setup the env, the feature is useless. So we decide to develop user space
>> colo-proxy.
>>
>> The advantage is obvious,
>> 1. we do not need to recompile kernel.
>> 2. No need to recompile iptables/nftables.
>> 3. we do not need to deal with the network configuration, we just using a
>>     socket connection between 2 QEMUs to forward packets.
>> 4. A complete VM FT solution in one go, we have already developed the block
>>     replication in QEMU, so with the network replication in QEMU, all
>>     components we needed are within QEMU, this is very important, it greatly
>>     improves the usability of COLO feature! We hope it will gain more testers,
>>     users and developers.
>> 5. QEMU will gain a complete VM FT solution and the most advantage FT solution
>>     so far!
>>
>> Overall, usability is the most important factor that impact our choice.
>
> My biggest worry is your reliance on SLIRP for the TCP/IP stack; it
> doesn't get much work done on it and I worry about it's reliability for
> using it for the level of complexity you need.
>
> Your current kernel implementation gets all the nf_conntrack stuff for free
> which is very powerful.
>
> However, I can see some advantages from doing it in user space; it would
> be easier to debug, and possibly easier to configure, and might also be easier
> to handle continuous FT (i.e. transferring the state of the proxy to a new COLO
> connection).
>
> I think at the moment I'd still prefer kernel space (especially since your kernel
> code now works pretty reliably!)
>
> Another thought; if you're main worry is to do with the complexity of kernel
> changes, had you considered looking at the bpf-jit - I'm not sure if it can
> do what you need, but perhaps it's worth a look?

Will have a look, thank you!

>
> Dave
> P.S. I think 'proxy' is still the right word to describe it rather than 'agency'.
>
>>
>>
>>>
>>> Thanks
>>> .
>>>
>>
>> --
>> Thanks,
>> Yang.
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dong, Eddie]

BTW,  I felt it is better to be called as an agency, rather than a proxy. Any comments from native speaker?

Thx Eddie



> Hi, all
> 
> We are planning to implement colo-proxy in qemu to cache and compare
> packets.
> This module is one of the important component of COLO project and now it is
> still in early stage, so any comments and feedback are warmly welcomed, thanks
> in advance.
> 
> ## Background
> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
> project is a high availability solution. Both Primary VM (PVM) and Secondary
> VM
> (SVM) run in parallel. They receive the same request from client, and generate
> responses in parallel too. If the response packets from PVM and SVM are
> identical, they are released immediately. Otherwise, a VM checkpoint (on
> demand) is conducted.
> Paper:
> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
> COLO on Xen:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> COLO on Qemu/KVM:
> http://wiki.qemu.org/Features/COLO
> 
> By the needs of capturing response packets from PVM and SVM and finding out
> whether they are identical, we introduce a new module to qemu networking
> called colo-proxy.
> 
> This document describes the design of the colo-proxy module
> 
> ## Glossary
>    PVM - Primary VM, which provides services to clients.
>    SVM - Secondary VM, a hot standby and replication of PVM.
>    PN - Primary Node, the host which PVM runs on
>    SN - Secondary Node, the host which SVM runs on
> 
> ## Workflow ##
> The following image show the qemu networking packet datapath between
> guest's NIC and qemu's backend in colo-proxy.
> 
> +---+                                        +---+
> |PN |                                        |SN |
> +---+--------------------------+
> +---+--------------------------+ +------------------------------+
> |               +-------+      |             |   +-------+                  |
> +--------+      |chkpoint<--------[socket]------->chkpoint         +--------+
> |PVM     |      +---^---+      |             |   +---+---+         |SVM     |
> |        |  +proxy--v--------+ |             |       |             |        |
> |        |  |                | |             |       |             |        |
> | +---+  |  | +TCP/IP stack+ | |             | +-----v-------proxy | +---+  |
> +-|NIC|--+  | |            | | |             | |                 | +-|NIC|--+
> | +^-++     | | +--------+ | | |             | | +TCP/IP stack-+ |   +^--+  |
> |  | +------> | | compare| | <-[socket]-forward- | +--------+  | |    |     |
> |  |        | | +---+----+ | | |             | | | |seq&ack |  | <----+     |
> |  |        | +-----|------+ | |             | | | |adjust  |  | |          |
> |  |        |       |        | |             | | | +--------+  | |          |
> |  +-----------<+>-----copy&forward-[socket]---> +-------------+ |          |
> |           +---|---|--------+ |             | +------------^----+          |
> |               |   |          |             |              |               |
> |               |   |          |             |              x               |
> |            +--+---v----+     |             |            +-v---------+     |
> | QEMU       |  backend  |     |             | QEMU       |  backend  |     |
> +------------+  (tap)    +-----+             +------------+  (tap)    +-----+
>               +-----------+                                +-----------+
> 
> ## Our Idea ##
> 
> ### Net filter
> In current QEMU, a packet is transported between networking backend(tap)
> and qemu network adapter(NIC) directly. Backend and adapter is linked by
> NetClientState->peer in qemu as following
>         +----------------------------------------+
>         v                                        |
> +NetClientState+   +------->+NetClientState+    |
> |info->type=TAP|   |        |info->type=NIC|    |
> +--------------+   |        +--------------+    |
> |   *peer      +---+        |   *peer      +----+
> +--------------+            +--------------+
> |name="tap0"   |            |name="e1000"  |
> +--------------+            +--------------+
> | ...          |            | ...          |
> +--------------+            +--------------+
> 
> In COLO QEMU, we insert a net filter named colo-proxy between backend and
> adapter like below:
> typedef struct COLOState {
>      NetClientState nc;
>      NetClientState *peer;
> } COLOState;
>     +------->+NetClientState+            +NetClientState+<--------+
>     |        |info->type=TAP|            |info->type=NIC|         |
>     |        +--------------+            +--------------+         |
> +-----------+   *peer      |            |   *peer      +------------+
> |  |        +--------------+            +--------------+         |  |
> |  |        |name="tap0"   |            |name="e1000"  |         |  |
> |  |        +--------------+            +--------------+         |  |
> |  |        | ...          |            | ...          |         |  |
> |  |        +--------------+            +--------------+         |  |
> |  |                                                             |  |
> |  |   +-COLOState------------+       +-COLOState------------+   |  |
> +--------->+NetClientState+<- - +   +---->+NetClientState+<---------+
>     |   |   |info->type=COLO   | |   | |   |info->type=COLO   |   |
>     |   |   +--------------+   | |   | |   +--------------+   |   |
>     +-------+   *peer      |   | |   | |   |   *peer      +-------+
>         |   +--------------+   | |   | |   +--------------+   |
>         |   |name="colo1"  |   | |   | |   |name="colo2"  |   |
>         |   +--------------+   | |   | |   +--------------+   |
>         |                      | |   | |                      |
>         |   +--------------+---------+ |   +--------------+   |
>         |   |   *peer      |   | |     |   |   *peer      |   |
>         |   +--------------+   | +---------+--------------+   |
>         +----------------------+       +----------------------+
> 
> After we insert colo-proxy filter, all packets will pass by this filter and more
> important thing is that we can analysis packet by ourselves.
> 
> ### QEMU space TCP/IP stack(re-use SLIRP) ### We need a QEMU space TCP/IP
> stack to help us to analysis packet. After looking into QEMU, we found that
> SLIRP
> http://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.
> 29
> is a good choice for us. SLIRP proivdes a full TCP/IP stack within QEMU, it can
> help use to handle the packet written to/read from backend(tap) device which is
> just like a link layer(L2) packet.
> 
> ### packet enqueue and compare ###
> Together with QEMU space TCP/IP stack, we enqueue all packets sent by PVM
> and SVM on Primary QEMU, and then compare the packet payload for each
> connection.
> 
> ### Net filter Usage ###
> On both Primary/Secondary host, invoke QEMU with the following parameters
> to insert a net filter(colo-proxy):
>    "-netdev tap,id=hn0 -device e1000,netdev=hn0 \
>     -netdev colo,id=colo,backend=hn0"
> 

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/20/2015 02:42 PM, Li Zhijian wrote:
> Hi, all
>
> We are planning to implement colo-proxy in qemu to cache and compare
> packets.
> This module is one of the important component of COLO project and now
> it is
> still in early stage, so any comments and feedback are warmly welcomed,
> thanks in advance.
>
> ## Background
> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
> Service)
> project is a high availability solution. Both Primary VM (PVM) and
> Secondary VM
> (SVM) run in parallel. They receive the same request from client, and
> generate
> responses in parallel too. If the response packets from PVM and SVM are
> identical, they are released immediately. Otherwise, a VM checkpoint
> (on demand)
> is conducted.
> Paper:
> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
> COLO on Xen:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> COLO on Qemu/KVM:
> http://wiki.qemu.org/Features/COLO
>
> By the needs of capturing response packets from PVM and SVM and
> finding out
> whether they are identical, we introduce a new module to qemu
> networking called
> colo-proxy.

A question here, the packet comparing may be very tricky. For example,
some protocol use random data to generate unpredictable id or something
else. One example is ipv6_select_ident() in Linux. So COLO needs a
mechanism to make sure PVM and SVM can generate same random data?

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Gonglei]

On 2015/7/30 12:23, Jason Wang wrote:
> 
> 
> On 07/20/2015 02:42 PM, Li Zhijian wrote:
>> Hi, all
>>
>> We are planning to implement colo-proxy in qemu to cache and compare
>> packets.
>> This module is one of the important component of COLO project and now
>> it is
>> still in early stage, so any comments and feedback are warmly welcomed,
>> thanks in advance.
>>
>> ## Background
>> COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop
>> Service)
>> project is a high availability solution. Both Primary VM (PVM) and
>> Secondary VM
>> (SVM) run in parallel. They receive the same request from client, and
>> generate
>> responses in parallel too. If the response packets from PVM and SVM are
>> identical, they are released immediately. Otherwise, a VM checkpoint
>> (on demand)
>> is conducted.
>> Paper:
>> http://www.socc2013.org/home/program/a3-dong.pdf?attredirects=0
>> COLO on Xen:
>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>> COLO on Qemu/KVM:
>> http://wiki.qemu.org/Features/COLO
>>
>> By the needs of capturing response packets from PVM and SVM and
>> finding out
>> whether they are identical, we introduce a new module to qemu
>> networking called
>> colo-proxy.
> 
> A question here, the packet comparing may be very tricky. For example,
> some protocol use random data to generate unpredictable id or something
> else. One example is ipv6_select_ident() in Linux. So COLO needs a
> mechanism to make sure PVM and SVM can generate same random data?
> 
Good question, the random data connection is a big problem for COLO. At
present, it will trigger checkpoint processing because of the different random data.
I don't think any mechanisms can assure two different machines generate
the same random data. If you have any ideas, pls tell us :)

Frequent checkpoint can handle this scenario, but maybe will cause the
performance poor. :(

Regards,
-Gonglei

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dong, Eddie]

> >
> > A question here, the packet comparing may be very tricky. For example,
> > some protocol use random data to generate unpredictable id or
> > something else. One example is ipv6_select_ident() in Linux. So COLO
> > needs a mechanism to make sure PVM and SVM can generate same random
> data?
> >
> Good question, the random data connection is a big problem for COLO. At
> present, it will trigger checkpoint processing because of the different random
> data.
> I don't think any mechanisms can assure two different machines generate the
> same random data. If you have any ideas, pls tell us :)
> 
> Frequent checkpoint can handle this scenario, but maybe will cause the
> performance poor. :(
> 
The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...

Thx Eddie

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dr. David Alan Gilbert]

* Dong, Eddie (eddie.dong@intel.com) wrote:
> > >
> > > A question here, the packet comparing may be very tricky. For example,
> > > some protocol use random data to generate unpredictable id or
> > > something else. One example is ipv6_select_ident() in Linux. So COLO
> > > needs a mechanism to make sure PVM and SVM can generate same random
> > data?
> > >
> > Good question, the random data connection is a big problem for COLO. At
> > present, it will trigger checkpoint processing because of the different random
> > data.
> > I don't think any mechanisms can assure two different machines generate the
> > same random data. If you have any ideas, pls tell us :)
> > 
> > Frequent checkpoint can handle this scenario, but maybe will cause the
> > performance poor. :(
> > 
> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...

They do diverge pretty quickly though; I have simple examples which
reliably cause a checkpoint because of simple randomness in applications.

Dave

> Thx Eddie
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Jason Wang]

On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>> A question here, the packet comparing may be very tricky. For example,
>>>> some protocol use random data to generate unpredictable id or
>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>> data?
>>> Good question, the random data connection is a big problem for COLO. At
>>> present, it will trigger checkpoint processing because of the different random
>>> data.
>>> I don't think any mechanisms can assure two different machines generate the
>>> same random data. If you have any ideas, pls tell us :)
>>>
>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>> performance poor. :(
>>>
>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
> They do diverge pretty quickly though; I have simple examples which
> reliably cause a checkpoint because of simple randomness in applications.
>
> Dave
>

And it will become even worse if hwrng is used in guest.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dr. David Alan Gilbert]

* Jason Wang (jasowang@redhat.com) wrote:
> 
> 
> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
> > * Dong, Eddie (eddie.dong@intel.com) wrote:
> >>>> A question here, the packet comparing may be very tricky. For example,
> >>>> some protocol use random data to generate unpredictable id or
> >>>> something else. One example is ipv6_select_ident() in Linux. So COLO
> >>>> needs a mechanism to make sure PVM and SVM can generate same random
> >>> data?
> >>> Good question, the random data connection is a big problem for COLO. At
> >>> present, it will trigger checkpoint processing because of the different random
> >>> data.
> >>> I don't think any mechanisms can assure two different machines generate the
> >>> same random data. If you have any ideas, pls tell us :)
> >>>
> >>> Frequent checkpoint can handle this scenario, but maybe will cause the
> >>> performance poor. :(
> >>>
> >> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
> > They do diverge pretty quickly though; I have simple examples which
> > reliably cause a checkpoint because of simple randomness in applications.
> >
> > Dave
> >
> 
> And it will become even worse if hwrng is used in guest.

Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
once established, tends to work well without triggering checkpoints;
and static web pages also work well.  Examples of things that do cause
more checkpoints are, displaying guest statistics (e.g. running top
in that ssh) which is timing dependent, and dynamically generated
web pages that include a unique ID (bugzilla's password reset link in
it's front page was a fun one), I think also establishing
new encrypted connections cause the same randomness.

However, it's worth remembering that COLO is trying to reduce the
number of checkpoints compared to a simple checkpointing world
which would be aiming to do a checkpoint ~100 times a second,
and for compute bound workloads, or ones that don't expose
the randomness that much, it can get checkpoints of a few seconds
in length which greatly reduces the overhead.

Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Gonglei]

On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
> * Jason Wang (jasowang@redhat.com) wrote:
>>
>>
>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
>>> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>>>> A question here, the packet comparing may be very tricky. For example,
>>>>>> some protocol use random data to generate unpredictable id or
>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>>>> data?
>>>>> Good question, the random data connection is a big problem for COLO. At
>>>>> present, it will trigger checkpoint processing because of the different random
>>>>> data.
>>>>> I don't think any mechanisms can assure two different machines generate the
>>>>> same random data. If you have any ideas, pls tell us :)
>>>>>
>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>>>> performance poor. :(
>>>>>
>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
>>> They do diverge pretty quickly though; I have simple examples which
>>> reliably cause a checkpoint because of simple randomness in applications.
>>>
>>> Dave
>>>
>>
>> And it will become even worse if hwrng is used in guest.
> 
> Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
> once established, tends to work well without triggering checkpoints;
> and static web pages also work well.  Examples of things that do cause
> more checkpoints are, displaying guest statistics (e.g. running top
> in that ssh) which is timing dependent, and dynamically generated
> web pages that include a unique ID (bugzilla's password reset link in
> it's front page was a fun one), I think also establishing
> new encrypted connections cause the same randomness.
> 
> However, it's worth remembering that COLO is trying to reduce the
> number of checkpoints compared to a simple checkpointing world
> which would be aiming to do a checkpoint ~100 times a second,
> and for compute bound workloads, or ones that don't expose
> the randomness that much, it can get checkpoints of a few seconds
> in length which greatly reduces the overhead.
> 

Yes. That's the truth.
We can set two different modes for different scenarios. Maybe Named
1) frequent checkpoint mode for multi-connections and randomness scenarios
and 2) non-frequent checkpoint mode for other scenarios.

But that's the next plan, we are thinking about that.

Regards,
-Gonglei

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dr. David Alan Gilbert]

* Gonglei (arei.gonglei@huawei.com) wrote:
> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
> > * Jason Wang (jasowang@redhat.com) wrote:
> >>
> >>
> >> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
> >>> * Dong, Eddie (eddie.dong@intel.com) wrote:
> >>>>>> A question here, the packet comparing may be very tricky. For example,
> >>>>>> some protocol use random data to generate unpredictable id or
> >>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
> >>>>>> needs a mechanism to make sure PVM and SVM can generate same random
> >>>>> data?
> >>>>> Good question, the random data connection is a big problem for COLO. At
> >>>>> present, it will trigger checkpoint processing because of the different random
> >>>>> data.
> >>>>> I don't think any mechanisms can assure two different machines generate the
> >>>>> same random data. If you have any ideas, pls tell us :)
> >>>>>
> >>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
> >>>>> performance poor. :(
> >>>>>
> >>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
> >>> They do diverge pretty quickly though; I have simple examples which
> >>> reliably cause a checkpoint because of simple randomness in applications.
> >>>
> >>> Dave
> >>>
> >>
> >> And it will become even worse if hwrng is used in guest.
> > 
> > Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
> > once established, tends to work well without triggering checkpoints;
> > and static web pages also work well.  Examples of things that do cause
> > more checkpoints are, displaying guest statistics (e.g. running top
> > in that ssh) which is timing dependent, and dynamically generated
> > web pages that include a unique ID (bugzilla's password reset link in
> > it's front page was a fun one), I think also establishing
> > new encrypted connections cause the same randomness.
> > 
> > However, it's worth remembering that COLO is trying to reduce the
> > number of checkpoints compared to a simple checkpointing world
> > which would be aiming to do a checkpoint ~100 times a second,
> > and for compute bound workloads, or ones that don't expose
> > the randomness that much, it can get checkpoints of a few seconds
> > in length which greatly reduces the overhead.
> > 
> 
> Yes. That's the truth.
> We can set two different modes for different scenarios. Maybe Named
> 1) frequent checkpoint mode for multi-connections and randomness scenarios
> and 2) non-frequent checkpoint mode for other scenarios.
> 
> But that's the next plan, we are thinking about that.

I have some code that tries to automatically switch between those;
it measures the checkpoint lengths, and if they're consistently short
it sends a different message byte to the secondary at the start of the
checkpoint, so that it doesn't bother running.   Every so often it
then flips back to a COLO checkpoint to see if the checkpoints
are still really fast.

Dave

> 
> Regards,
> -Gonglei
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [POC] colo-proxy in qemu

[zhanghailiang]

On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
> * Gonglei (arei.gonglei@huawei.com) wrote:
>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
>>> * Jason Wang (jasowang@redhat.com) wrote:
>>>>
>>>>
>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>>>>>> A question here, the packet comparing may be very tricky. For example,
>>>>>>>> some protocol use random data to generate unpredictable id or
>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>>>>>> data?
>>>>>>> Good question, the random data connection is a big problem for COLO. At
>>>>>>> present, it will trigger checkpoint processing because of the different random
>>>>>>> data.
>>>>>>> I don't think any mechanisms can assure two different machines generate the
>>>>>>> same random data. If you have any ideas, pls tell us :)
>>>>>>>
>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>>>>>> performance poor. :(
>>>>>>>
>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
>>>>> They do diverge pretty quickly though; I have simple examples which
>>>>> reliably cause a checkpoint because of simple randomness in applications.
>>>>>
>>>>> Dave
>>>>>
>>>>
>>>> And it will become even worse if hwrng is used in guest.
>>>
>>> Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
>>> once established, tends to work well without triggering checkpoints;
>>> and static web pages also work well.  Examples of things that do cause
>>> more checkpoints are, displaying guest statistics (e.g. running top
>>> in that ssh) which is timing dependent, and dynamically generated
>>> web pages that include a unique ID (bugzilla's password reset link in
>>> it's front page was a fun one), I think also establishing
>>> new encrypted connections cause the same randomness.
>>>
>>> However, it's worth remembering that COLO is trying to reduce the
>>> number of checkpoints compared to a simple checkpointing world
>>> which would be aiming to do a checkpoint ~100 times a second,
>>> and for compute bound workloads, or ones that don't expose
>>> the randomness that much, it can get checkpoints of a few seconds
>>> in length which greatly reduces the overhead.
>>>
>>
>> Yes. That's the truth.
>> We can set two different modes for different scenarios. Maybe Named
>> 1) frequent checkpoint mode for multi-connections and randomness scenarios
>> and 2) non-frequent checkpoint mode for other scenarios.
>>
>> But that's the next plan, we are thinking about that.
>
> I have some code that tries to automatically switch between those;
> it measures the checkpoint lengths, and if they're consistently short
> it sends a different message byte to the secondary at the start of the
> checkpoint, so that it doesn't bother running.   Every so often it
> then flips back to a COLO checkpoint to see if the checkpoints
> are still really fast.
>

Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM?
Resume to common COLO mode until the checkpoint lengths is so not short ?

Thanks.

> Dave
>
>>
>> Regards,
>> -Gonglei
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dr. David Alan Gilbert]

* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
> >* Gonglei (arei.gonglei@huawei.com) wrote:
> >>On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
> >>>* Jason Wang (jasowang@redhat.com) wrote:
> >>>>
> >>>>
> >>>>On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
> >>>>>* Dong, Eddie (eddie.dong@intel.com) wrote:
> >>>>>>>>A question here, the packet comparing may be very tricky. For example,
> >>>>>>>>some protocol use random data to generate unpredictable id or
> >>>>>>>>something else. One example is ipv6_select_ident() in Linux. So COLO
> >>>>>>>>needs a mechanism to make sure PVM and SVM can generate same random
> >>>>>>>data?
> >>>>>>>Good question, the random data connection is a big problem for COLO. At
> >>>>>>>present, it will trigger checkpoint processing because of the different random
> >>>>>>>data.
> >>>>>>>I don't think any mechanisms can assure two different machines generate the
> >>>>>>>same random data. If you have any ideas, pls tell us :)
> >>>>>>>
> >>>>>>>Frequent checkpoint can handle this scenario, but maybe will cause the
> >>>>>>>performance poor. :(
> >>>>>>>
> >>>>>>The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
> >>>>>They do diverge pretty quickly though; I have simple examples which
> >>>>>reliably cause a checkpoint because of simple randomness in applications.
> >>>>>
> >>>>>Dave
> >>>>>
> >>>>
> >>>>And it will become even worse if hwrng is used in guest.
> >>>
> >>>Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
> >>>once established, tends to work well without triggering checkpoints;
> >>>and static web pages also work well.  Examples of things that do cause
> >>>more checkpoints are, displaying guest statistics (e.g. running top
> >>>in that ssh) which is timing dependent, and dynamically generated
> >>>web pages that include a unique ID (bugzilla's password reset link in
> >>>it's front page was a fun one), I think also establishing
> >>>new encrypted connections cause the same randomness.
> >>>
> >>>However, it's worth remembering that COLO is trying to reduce the
> >>>number of checkpoints compared to a simple checkpointing world
> >>>which would be aiming to do a checkpoint ~100 times a second,
> >>>and for compute bound workloads, or ones that don't expose
> >>>the randomness that much, it can get checkpoints of a few seconds
> >>>in length which greatly reduces the overhead.
> >>>
> >>
> >>Yes. That's the truth.
> >>We can set two different modes for different scenarios. Maybe Named
> >>1) frequent checkpoint mode for multi-connections and randomness scenarios
> >>and 2) non-frequent checkpoint mode for other scenarios.
> >>
> >>But that's the next plan, we are thinking about that.
> >
> >I have some code that tries to automatically switch between those;
> >it measures the checkpoint lengths, and if they're consistently short
> >it sends a different message byte to the secondary at the start of the
> >checkpoint, so that it doesn't bother running.   Every so often it
> >then flips back to a COLO checkpoint to see if the checkpoints
> >are still really fast.
> >
> 
> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM?
> Resume to common COLO mode until the checkpoint lengths is so not short ?

  We still have to do checkpoints, but we send a special message to the SVM so that
the SVM just takes the checkpoint but does not run.

  I'll send the code after I've updated it to your current version; but it's
quite rough/experimental.

It works something like

 -----------run PVM     run SVM
     COLO     <long gap>
     mode       miscompare
                checkpoint
 -----------run PVM     run SVM
     COLO     <short gap>
     mode       miscompare
                checkpoint
 -----------run PVM     run SVM
     COLO     <short gap>
     mode       miscompare         < After a few short runs
                checkpoint
 -----------run PVM     SVM idle   \
   Passive    <fixed delay>        |  - repeat 'n' times
     mode       checkpoint         /
 -----------run PVM     run SVM
     COLO     <short gap>          < Still a short gap
     mode       miscompare
 -----------run PVM     SVM idle   \
   Passive    <fixed delay>        |  - repeat 'n' times
     mode       checkpoint         /
 -----------run PVM     run SVM
     COLO     <long gap>          < long gap now, stay in COLO
     mode       miscompare
                checkpoint
 -----------run PVM     run SVM
     COLO     <long gap>
     mode       miscompare
                checkpoint
     
So it saves the CPU time on the SVM, and the comparison traffic, and is
automatic at switching into the passive mode.

It used to be more useful, but your minimum COLO run time that you
added a few versions ago helps a lot in the cases where there are miscompares,
and the delay after the miscompare before you take the checkpoint also helps
in the case where the data is very random.

Dave

> 
> Thanks.
> 
> >Dave
> >
> >>
> >>Regards,
> >>-Gonglei
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >.
> >
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
>>> * Gonglei (arei.gonglei@huawei.com) wrote:
>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
>>>>> * Jason Wang (jasowang@redhat.com) wrote:
>>>>>>
>>>>>>
>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>>>>>>>> A question here, the packet comparing may be very tricky. For example,
>>>>>>>>>> some protocol use random data to generate unpredictable id or
>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>>>>>>>> data?
>>>>>>>>> Good question, the random data connection is a big problem for COLO. At
>>>>>>>>> present, it will trigger checkpoint processing because of the different random
>>>>>>>>> data.
>>>>>>>>> I don't think any mechanisms can assure two different machines generate the
>>>>>>>>> same random data. If you have any ideas, pls tell us :)
>>>>>>>>>
>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>>>>>>>> performance poor. :(
>>>>>>>>>
>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
>>>>>>> They do diverge pretty quickly though; I have simple examples which
>>>>>>> reliably cause a checkpoint because of simple randomness in applications.
>>>>>>>
>>>>>>> Dave
>>>>>>>
>>>>>>
>>>>>> And it will become even worse if hwrng is used in guest.
>>>>>
>>>>> Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
>>>>> once established, tends to work well without triggering checkpoints;
>>>>> and static web pages also work well.  Examples of things that do cause
>>>>> more checkpoints are, displaying guest statistics (e.g. running top
>>>>> in that ssh) which is timing dependent, and dynamically generated
>>>>> web pages that include a unique ID (bugzilla's password reset link in
>>>>> it's front page was a fun one), I think also establishing
>>>>> new encrypted connections cause the same randomness.
>>>>>
>>>>> However, it's worth remembering that COLO is trying to reduce the
>>>>> number of checkpoints compared to a simple checkpointing world
>>>>> which would be aiming to do a checkpoint ~100 times a second,
>>>>> and for compute bound workloads, or ones that don't expose
>>>>> the randomness that much, it can get checkpoints of a few seconds
>>>>> in length which greatly reduces the overhead.
>>>>>
>>>>
>>>> Yes. That's the truth.
>>>> We can set two different modes for different scenarios. Maybe Named
>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios
>>>> and 2) non-frequent checkpoint mode for other scenarios.
>>>>
>>>> But that's the next plan, we are thinking about that.
>>>
>>> I have some code that tries to automatically switch between those;
>>> it measures the checkpoint lengths, and if they're consistently short
>>> it sends a different message byte to the secondary at the start of the
>>> checkpoint, so that it doesn't bother running.   Every so often it
>>> then flips back to a COLO checkpoint to see if the checkpoints
>>> are still really fast.
>>>
>>
>> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM?
>> Resume to common COLO mode until the checkpoint lengths is so not short ?
>
>    We still have to do checkpoints, but we send a special message to the SVM so that
> the SVM just takes the checkpoint but does not run.
>
>    I'll send the code after I've updated it to your current version; but it's
> quite rough/experimental.
>
> It works something like
>
>   -----------run PVM     run SVM
>       COLO     <long gap>
>       mode       miscompare
>                  checkpoint
>   -----------run PVM     run SVM
>       COLO     <short gap>
>       mode       miscompare
>                  checkpoint
>   -----------run PVM     run SVM
>       COLO     <short gap>
>       mode       miscompare         < After a few short runs
>                  checkpoint
>   -----------run PVM     SVM idle   \
>     Passive    <fixed delay>        |  - repeat 'n' times
>       mode       checkpoint         /
>   -----------run PVM     run SVM
>       COLO     <short gap>          < Still a short gap
>       mode       miscompare
>   -----------run PVM     SVM idle   \
>     Passive    <fixed delay>        |  - repeat 'n' times
>       mode       checkpoint         /
>   -----------run PVM     run SVM
>       COLO     <long gap>          < long gap now, stay in COLO
>       mode       miscompare
>                  checkpoint
>   -----------run PVM     run SVM
>       COLO     <long gap>
>       mode       miscompare
>                  checkpoint
>
> So it saves the CPU time on the SVM, and the comparison traffic, and is
> automatic at switching into the passive mode.
>
> It used to be more useful, but your minimum COLO run time that you
> added a few versions ago helps a lot in the cases where there are miscompares,
> and the delay after the miscompare before you take the checkpoint also helps
> in the case where the data is very random.

This is great! This is exactly what we were thinking about, when random
scenario will fallback to MC/Remus like FT. Thank you very much!
I have a question, do you also modify colo-proxy kernel module? because
in the fixed checkpoint mode, I think we need to buffer the network
packets, and release them at checkpoint.

>
> Dave
>
>>
>> Thanks.
>>
>>> Dave
>>>
>>>>
>>>> Regards,
>>>> -Gonglei
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>> .
>>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Dr. David Alan Gilbert]

* Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
> 
> 
> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote:
> >* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> >>On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
> >>>* Gonglei (arei.gonglei@huawei.com) wrote:
> >>>>On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
> >>>>>* Jason Wang (jasowang@redhat.com) wrote:
> >>>>>>
> >>>>>>
> >>>>>>On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
> >>>>>>>* Dong, Eddie (eddie.dong@intel.com) wrote:
> >>>>>>>>>>A question here, the packet comparing may be very tricky. For example,
> >>>>>>>>>>some protocol use random data to generate unpredictable id or
> >>>>>>>>>>something else. One example is ipv6_select_ident() in Linux. So COLO
> >>>>>>>>>>needs a mechanism to make sure PVM and SVM can generate same random
> >>>>>>>>>data?
> >>>>>>>>>Good question, the random data connection is a big problem for COLO. At
> >>>>>>>>>present, it will trigger checkpoint processing because of the different random
> >>>>>>>>>data.
> >>>>>>>>>I don't think any mechanisms can assure two different machines generate the
> >>>>>>>>>same random data. If you have any ideas, pls tell us :)
> >>>>>>>>>
> >>>>>>>>>Frequent checkpoint can handle this scenario, but maybe will cause the
> >>>>>>>>>performance poor. :(
> >>>>>>>>>
> >>>>>>>>The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
> >>>>>>>They do diverge pretty quickly though; I have simple examples which
> >>>>>>>reliably cause a checkpoint because of simple randomness in applications.
> >>>>>>>
> >>>>>>>Dave
> >>>>>>>
> >>>>>>
> >>>>>>And it will become even worse if hwrng is used in guest.
> >>>>>
> >>>>>Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
> >>>>>once established, tends to work well without triggering checkpoints;
> >>>>>and static web pages also work well.  Examples of things that do cause
> >>>>>more checkpoints are, displaying guest statistics (e.g. running top
> >>>>>in that ssh) which is timing dependent, and dynamically generated
> >>>>>web pages that include a unique ID (bugzilla's password reset link in
> >>>>>it's front page was a fun one), I think also establishing
> >>>>>new encrypted connections cause the same randomness.
> >>>>>
> >>>>>However, it's worth remembering that COLO is trying to reduce the
> >>>>>number of checkpoints compared to a simple checkpointing world
> >>>>>which would be aiming to do a checkpoint ~100 times a second,
> >>>>>and for compute bound workloads, or ones that don't expose
> >>>>>the randomness that much, it can get checkpoints of a few seconds
> >>>>>in length which greatly reduces the overhead.
> >>>>>
> >>>>
> >>>>Yes. That's the truth.
> >>>>We can set two different modes for different scenarios. Maybe Named
> >>>>1) frequent checkpoint mode for multi-connections and randomness scenarios
> >>>>and 2) non-frequent checkpoint mode for other scenarios.
> >>>>
> >>>>But that's the next plan, we are thinking about that.
> >>>
> >>>I have some code that tries to automatically switch between those;
> >>>it measures the checkpoint lengths, and if they're consistently short
> >>>it sends a different message byte to the secondary at the start of the
> >>>checkpoint, so that it doesn't bother running.   Every so often it
> >>>then flips back to a COLO checkpoint to see if the checkpoints
> >>>are still really fast.
> >>>
> >>
> >>Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM?
> >>Resume to common COLO mode until the checkpoint lengths is so not short ?
> >
> >   We still have to do checkpoints, but we send a special message to the SVM so that
> >the SVM just takes the checkpoint but does not run.
> >
> >   I'll send the code after I've updated it to your current version; but it's
> >quite rough/experimental.
> >
> >It works something like
> >
> >  -----------run PVM     run SVM
> >      COLO     <long gap>
> >      mode       miscompare
> >                 checkpoint
> >  -----------run PVM     run SVM
> >      COLO     <short gap>
> >      mode       miscompare
> >                 checkpoint
> >  -----------run PVM     run SVM
> >      COLO     <short gap>
> >      mode       miscompare         < After a few short runs
> >                 checkpoint
> >  -----------run PVM     SVM idle   \
> >    Passive    <fixed delay>        |  - repeat 'n' times
> >      mode       checkpoint         /
> >  -----------run PVM     run SVM
> >      COLO     <short gap>          < Still a short gap
> >      mode       miscompare
> >  -----------run PVM     SVM idle   \
> >    Passive    <fixed delay>        |  - repeat 'n' times
> >      mode       checkpoint         /
> >  -----------run PVM     run SVM
> >      COLO     <long gap>          < long gap now, stay in COLO
> >      mode       miscompare
> >                 checkpoint
> >  -----------run PVM     run SVM
> >      COLO     <long gap>
> >      mode       miscompare
> >                 checkpoint
> >
> >So it saves the CPU time on the SVM, and the comparison traffic, and is
> >automatic at switching into the passive mode.
> >
> >It used to be more useful, but your minimum COLO run time that you
> >added a few versions ago helps a lot in the cases where there are miscompares,
> >and the delay after the miscompare before you take the checkpoint also helps
> >in the case where the data is very random.
> 
> This is great! This is exactly what we were thinking about, when random
> scenario will fallback to MC/Remus like FT. Thank you very much!
> I have a question, do you also modify colo-proxy kernel module? because
> in the fixed checkpoint mode, I think we need to buffer the network
> packets, and release them at checkpoint.

Yes, we do need to buffer and release them at the end, but I've not modified
colo-proxy so far.  Doesn't the current code on PMY already need to buffer packets
that are generated after the first miscompare and before the checkpoint and
then release them at the checkpoint?

Dave

> 
> >
> >Dave
> >
> >>
> >>Thanks.
> >>
> >>>Dave
> >>>
> >>>>
> >>>>Regards,
> >>>>-Gonglei
> >>>>
> >>>--
> >>>Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>>
> >>>.
> >>>
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >.
> >
> 
> -- 
> Thanks,
> Yang.
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/31/2015 01:53 AM, Dr. David Alan Gilbert wrote:
> * Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
>>
>>
>> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote:
>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
>>>>> * Gonglei (arei.gonglei@huawei.com) wrote:
>>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
>>>>>>> * Jason Wang (jasowang@redhat.com) wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
>>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>>>>>>>>>> A question here, the packet comparing may be very tricky. For example,
>>>>>>>>>>>> some protocol use random data to generate unpredictable id or
>>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>>>>>>>>>> data?
>>>>>>>>>>> Good question, the random data connection is a big problem for COLO. At
>>>>>>>>>>> present, it will trigger checkpoint processing because of the different random
>>>>>>>>>>> data.
>>>>>>>>>>> I don't think any mechanisms can assure two different machines generate the
>>>>>>>>>>> same random data. If you have any ideas, pls tell us :)
>>>>>>>>>>>
>>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>>>>>>>>>> performance poor. :(
>>>>>>>>>>>
>>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
>>>>>>>>> They do diverge pretty quickly though; I have simple examples which
>>>>>>>>> reliably cause a checkpoint because of simple randomness in applications.
>>>>>>>>>
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>>>
>>>>>>>> And it will become even worse if hwrng is used in guest.
>>>>>>>
>>>>>>> Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
>>>>>>> once established, tends to work well without triggering checkpoints;
>>>>>>> and static web pages also work well.  Examples of things that do cause
>>>>>>> more checkpoints are, displaying guest statistics (e.g. running top
>>>>>>> in that ssh) which is timing dependent, and dynamically generated
>>>>>>> web pages that include a unique ID (bugzilla's password reset link in
>>>>>>> it's front page was a fun one), I think also establishing
>>>>>>> new encrypted connections cause the same randomness.
>>>>>>>
>>>>>>> However, it's worth remembering that COLO is trying to reduce the
>>>>>>> number of checkpoints compared to a simple checkpointing world
>>>>>>> which would be aiming to do a checkpoint ~100 times a second,
>>>>>>> and for compute bound workloads, or ones that don't expose
>>>>>>> the randomness that much, it can get checkpoints of a few seconds
>>>>>>> in length which greatly reduces the overhead.
>>>>>>>
>>>>>>
>>>>>> Yes. That's the truth.
>>>>>> We can set two different modes for different scenarios. Maybe Named
>>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios
>>>>>> and 2) non-frequent checkpoint mode for other scenarios.
>>>>>>
>>>>>> But that's the next plan, we are thinking about that.
>>>>>
>>>>> I have some code that tries to automatically switch between those;
>>>>> it measures the checkpoint lengths, and if they're consistently short
>>>>> it sends a different message byte to the secondary at the start of the
>>>>> checkpoint, so that it doesn't bother running.   Every so often it
>>>>> then flips back to a COLO checkpoint to see if the checkpoints
>>>>> are still really fast.
>>>>>
>>>>
>>>> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM?
>>>> Resume to common COLO mode until the checkpoint lengths is so not short ?
>>>
>>>    We still have to do checkpoints, but we send a special message to the SVM so that
>>> the SVM just takes the checkpoint but does not run.
>>>
>>>    I'll send the code after I've updated it to your current version; but it's
>>> quite rough/experimental.
>>>
>>> It works something like
>>>
>>>   -----------run PVM     run SVM
>>>       COLO     <long gap>
>>>       mode       miscompare
>>>                  checkpoint
>>>   -----------run PVM     run SVM
>>>       COLO     <short gap>
>>>       mode       miscompare
>>>                  checkpoint
>>>   -----------run PVM     run SVM
>>>       COLO     <short gap>
>>>       mode       miscompare         < After a few short runs
>>>                  checkpoint
>>>   -----------run PVM     SVM idle   \
>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>       mode       checkpoint         /
>>>   -----------run PVM     run SVM
>>>       COLO     <short gap>          < Still a short gap
>>>       mode       miscompare
>>>   -----------run PVM     SVM idle   \
>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>       mode       checkpoint         /
>>>   -----------run PVM     run SVM
>>>       COLO     <long gap>          < long gap now, stay in COLO
>>>       mode       miscompare
>>>                  checkpoint
>>>   -----------run PVM     run SVM
>>>       COLO     <long gap>
>>>       mode       miscompare
>>>                  checkpoint
>>>
>>> So it saves the CPU time on the SVM, and the comparison traffic, and is
>>> automatic at switching into the passive mode.
>>>
>>> It used to be more useful, but your minimum COLO run time that you
>>> added a few versions ago helps a lot in the cases where there are miscompares,
>>> and the delay after the miscompare before you take the checkpoint also helps
>>> in the case where the data is very random.
>>
>> This is great! This is exactly what we were thinking about, when random
>> scenario will fallback to MC/Remus like FT. Thank you very much!
>> I have a question, do you also modify colo-proxy kernel module? because
>> in the fixed checkpoint mode, I think we need to buffer the network
>> packets, and release them at checkpoint.
>
> Yes, we do need to buffer and release them at the end, but I've not modified
> colo-proxy so far.  Doesn't the current code on PMY already need to buffer packets
> that are generated after the first miscompare

Yes, they are buffered,

> and before the checkpoint and
> then release them at the checkpoint?

but will be release only if the packets compare returns identical. so in order
to support this fallback mode, we need to modify it to release the packets at
the checkpoint, there won't be too much code though.

>
> Dave
>
>>
>>>
>>> Dave
>>>
>>>>
>>>> Thanks.
>>>>
>>>>> Dave
>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> -Gonglei
>>>>>>
>>>>> --
>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>
>>>>> .
>>>>>
>>>>
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>> .
>>>
>>
>> --
>> Thanks,
>> Yang.
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[zhanghailiang]

On 2015/7/31 9:08, Yang Hongyang wrote:
>
>
> On 07/31/2015 01:53 AM, Dr. David Alan Gilbert wrote:
>> * Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
>>>
>>>
>>> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote:
>>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
>>>>>> * Gonglei (arei.gonglei@huawei.com) wrote:
>>>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
>>>>>>>> * Jason Wang (jasowang@redhat.com) wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
>>>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>>>>>>>>>>> A question here, the packet comparing may be very tricky. For example,
>>>>>>>>>>>>> some protocol use random data to generate unpredictable id or
>>>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>>>>>>>>>>> data?
>>>>>>>>>>>> Good question, the random data connection is a big problem for COLO. At
>>>>>>>>>>>> present, it will trigger checkpoint processing because of the different random
>>>>>>>>>>>> data.
>>>>>>>>>>>> I don't think any mechanisms can assure two different machines generate the
>>>>>>>>>>>> same random data. If you have any ideas, pls tell us :)
>>>>>>>>>>>>
>>>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>>>>>>>>>>> performance poor. :(
>>>>>>>>>>>>
>>>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
>>>>>>>>>> They do diverge pretty quickly though; I have simple examples which
>>>>>>>>>> reliably cause a checkpoint because of simple randomness in applications.
>>>>>>>>>>
>>>>>>>>>> Dave
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> And it will become even worse if hwrng is used in guest.
>>>>>>>>
>>>>>>>> Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
>>>>>>>> once established, tends to work well without triggering checkpoints;
>>>>>>>> and static web pages also work well.  Examples of things that do cause
>>>>>>>> more checkpoints are, displaying guest statistics (e.g. running top
>>>>>>>> in that ssh) which is timing dependent, and dynamically generated
>>>>>>>> web pages that include a unique ID (bugzilla's password reset link in
>>>>>>>> it's front page was a fun one), I think also establishing
>>>>>>>> new encrypted connections cause the same randomness.
>>>>>>>>
>>>>>>>> However, it's worth remembering that COLO is trying to reduce the
>>>>>>>> number of checkpoints compared to a simple checkpointing world
>>>>>>>> which would be aiming to do a checkpoint ~100 times a second,
>>>>>>>> and for compute bound workloads, or ones that don't expose
>>>>>>>> the randomness that much, it can get checkpoints of a few seconds
>>>>>>>> in length which greatly reduces the overhead.
>>>>>>>>
>>>>>>>
>>>>>>> Yes. That's the truth.
>>>>>>> We can set two different modes for different scenarios. Maybe Named
>>>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios
>>>>>>> and 2) non-frequent checkpoint mode for other scenarios.
>>>>>>>
>>>>>>> But that's the next plan, we are thinking about that.
>>>>>>
>>>>>> I have some code that tries to automatically switch between those;
>>>>>> it measures the checkpoint lengths, and if they're consistently short
>>>>>> it sends a different message byte to the secondary at the start of the
>>>>>> checkpoint, so that it doesn't bother running.   Every so often it
>>>>>> then flips back to a COLO checkpoint to see if the checkpoints
>>>>>> are still really fast.
>>>>>>
>>>>>
>>>>> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM?
>>>>> Resume to common COLO mode until the checkpoint lengths is so not short ?
>>>>
>>>>    We still have to do checkpoints, but we send a special message to the SVM so that
>>>> the SVM just takes the checkpoint but does not run.
>>>>
>>>>    I'll send the code after I've updated it to your current version; but it's
>>>> quite rough/experimental.
>>>>
>>>> It works something like
>>>>
>>>>   -----------run PVM     run SVM
>>>>       COLO     <long gap>
>>>>       mode       miscompare
>>>>                  checkpoint
>>>>   -----------run PVM     run SVM
>>>>       COLO     <short gap>
>>>>       mode       miscompare
>>>>                  checkpoint
>>>>   -----------run PVM     run SVM
>>>>       COLO     <short gap>
>>>>       mode       miscompare         < After a few short runs
>>>>                  checkpoint
>>>>   -----------run PVM     SVM idle   \
>>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>>       mode       checkpoint         /
>>>>   -----------run PVM     run SVM
>>>>       COLO     <short gap>          < Still a short gap
>>>>       mode       miscompare
>>>>   -----------run PVM     SVM idle   \
>>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>>       mode       checkpoint         /
>>>>   -----------run PVM     run SVM
>>>>       COLO     <long gap>          < long gap now, stay in COLO
>>>>       mode       miscompare
>>>>                  checkpoint
>>>>   -----------run PVM     run SVM
>>>>       COLO     <long gap>
>>>>       mode       miscompare
>>>>                  checkpoint
>>>>
>>>> So it saves the CPU time on the SVM, and the comparison traffic, and is
>>>> automatic at switching into the passive mode.
>>>>
>>>> It used to be more useful, but your minimum COLO run time that you
>>>> added a few versions ago helps a lot in the cases where there are miscompares,
>>>> and the delay after the miscompare before you take the checkpoint also helps
>>>> in the case where the data is very random.
>>>
>>> This is great! This is exactly what we were thinking about, when random
>>> scenario will fallback to MC/Remus like FT. Thank you very much!
>>> I have a question, do you also modify colo-proxy kernel module? because
>>> in the fixed checkpoint mode, I think we need to buffer the network
>>> packets, and release them at checkpoint.
>>
>> Yes, we do need to buffer and release them at the end, but I've not modified
>> colo-proxy so far.  Doesn't the current code on PMY already need to buffer packets
>> that are generated after the first miscompare
>
> Yes, they are buffered,
>
>> and before the checkpoint and
>> then release them at the checkpoint?
>
> but will be release only if the packets compare returns identical. so in order
> to support this fallback mode, we need to modify it to release the packets at
> the checkpoint, there won't be too much code though.
>

No, when do checkpoint, we will send all the residual queued packets.
So it is already supported.

>>
>> Dave
>>
>>>
>>>>
>>>> Dave
>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>>> Dave
>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> -Gonglei
>>>>>>>
>>>>>> --
>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>>
>>>>>> .
>>>>>>
>>>>>
>>>>>
>>>> --
>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>> .
>>>>
>>>
>>> --
>>> Thanks,
>>> Yang.
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>> .
>>
>

Re: [Qemu-devel] [POC] colo-proxy in qemu

[Yang Hongyang]

On 07/31/2015 09:28 AM, zhanghailiang wrote:
> On 2015/7/31 9:08, Yang Hongyang wrote:
>>
>>
>> On 07/31/2015 01:53 AM, Dr. David Alan Gilbert wrote:
>>> * Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
>>>>
>>>>
>>>> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote:
>>>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>>>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
>>>>>>> * Gonglei (arei.gonglei@huawei.com) wrote:
>>>>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
>>>>>>>>> * Jason Wang (jasowang@redhat.com) wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
>>>>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>>>>>>>>>>>> A question here, the packet comparing may be very tricky. For
>>>>>>>>>>>>>> example,
>>>>>>>>>>>>>> some protocol use random data to generate unpredictable id or
>>>>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>>>>>>>>>>>> data?
>>>>>>>>>>>>> Good question, the random data connection is a big problem for
>>>>>>>>>>>>> COLO. At
>>>>>>>>>>>>> present, it will trigger checkpoint processing because of the
>>>>>>>>>>>>> different random
>>>>>>>>>>>>> data.
>>>>>>>>>>>>> I don't think any mechanisms can assure two different machines
>>>>>>>>>>>>> generate the
>>>>>>>>>>>>> same random data. If you have any ideas, pls tell us :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>>>>>>>>>>>> performance poor. :(
>>>>>>>>>>>>>
>>>>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have
>>>>>>>>>>>> identical internal state, so the pattern used to generate random
>>>>>>>>>>>> data has high possibility to generate identical data at short time,
>>>>>>>>>>>> at least...
>>>>>>>>>>> They do diverge pretty quickly though; I have simple examples which
>>>>>>>>>>> reliably cause a checkpoint because of simple randomness in
>>>>>>>>>>> applications.
>>>>>>>>>>>
>>>>>>>>>>> Dave
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And it will become even worse if hwrng is used in guest.
>>>>>>>>>
>>>>>>>>> Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
>>>>>>>>> once established, tends to work well without triggering checkpoints;
>>>>>>>>> and static web pages also work well.  Examples of things that do cause
>>>>>>>>> more checkpoints are, displaying guest statistics (e.g. running top
>>>>>>>>> in that ssh) which is timing dependent, and dynamically generated
>>>>>>>>> web pages that include a unique ID (bugzilla's password reset link in
>>>>>>>>> it's front page was a fun one), I think also establishing
>>>>>>>>> new encrypted connections cause the same randomness.
>>>>>>>>>
>>>>>>>>> However, it's worth remembering that COLO is trying to reduce the
>>>>>>>>> number of checkpoints compared to a simple checkpointing world
>>>>>>>>> which would be aiming to do a checkpoint ~100 times a second,
>>>>>>>>> and for compute bound workloads, or ones that don't expose
>>>>>>>>> the randomness that much, it can get checkpoints of a few seconds
>>>>>>>>> in length which greatly reduces the overhead.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Yes. That's the truth.
>>>>>>>> We can set two different modes for different scenarios. Maybe Named
>>>>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios
>>>>>>>> and 2) non-frequent checkpoint mode for other scenarios.
>>>>>>>>
>>>>>>>> But that's the next plan, we are thinking about that.
>>>>>>>
>>>>>>> I have some code that tries to automatically switch between those;
>>>>>>> it measures the checkpoint lengths, and if they're consistently short
>>>>>>> it sends a different message byte to the secondary at the start of the
>>>>>>> checkpoint, so that it doesn't bother running.   Every so often it
>>>>>>> then flips back to a COLO checkpoint to see if the checkpoints
>>>>>>> are still really fast.
>>>>>>>
>>>>>>
>>>>>> Do you mean if there are consistent checkpoint requests, not do checkpoint
>>>>>> but just send a special message to SVM?
>>>>>> Resume to common COLO mode until the checkpoint lengths is so not short ?
>>>>>
>>>>>    We still have to do checkpoints, but we send a special message to the
>>>>> SVM so that
>>>>> the SVM just takes the checkpoint but does not run.
>>>>>
>>>>>    I'll send the code after I've updated it to your current version; but it's
>>>>> quite rough/experimental.
>>>>>
>>>>> It works something like
>>>>>
>>>>>   -----------run PVM     run SVM
>>>>>       COLO     <long gap>
>>>>>       mode       miscompare
>>>>>                  checkpoint
>>>>>   -----------run PVM     run SVM
>>>>>       COLO     <short gap>
>>>>>       mode       miscompare
>>>>>                  checkpoint
>>>>>   -----------run PVM     run SVM
>>>>>       COLO     <short gap>
>>>>>       mode       miscompare         < After a few short runs
>>>>>                  checkpoint
>>>>>   -----------run PVM     SVM idle   \
>>>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>>>       mode       checkpoint         /
>>>>>   -----------run PVM     run SVM
>>>>>       COLO     <short gap>          < Still a short gap
>>>>>       mode       miscompare
>>>>>   -----------run PVM     SVM idle   \
>>>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>>>       mode       checkpoint         /
>>>>>   -----------run PVM     run SVM
>>>>>       COLO     <long gap>          < long gap now, stay in COLO
>>>>>       mode       miscompare
>>>>>                  checkpoint
>>>>>   -----------run PVM     run SVM
>>>>>       COLO     <long gap>
>>>>>       mode       miscompare
>>>>>                  checkpoint
>>>>>
>>>>> So it saves the CPU time on the SVM, and the comparison traffic, and is
>>>>> automatic at switching into the passive mode.
>>>>>
>>>>> It used to be more useful, but your minimum COLO run time that you
>>>>> added a few versions ago helps a lot in the cases where there are miscompares,
>>>>> and the delay after the miscompare before you take the checkpoint also helps
>>>>> in the case where the data is very random.
>>>>
>>>> This is great! This is exactly what we were thinking about, when random
>>>> scenario will fallback to MC/Remus like FT. Thank you very much!
>>>> I have a question, do you also modify colo-proxy kernel module? because
>>>> in the fixed checkpoint mode, I think we need to buffer the network
>>>> packets, and release them at checkpoint.
>>>
>>> Yes, we do need to buffer and release them at the end, but I've not modified
>>> colo-proxy so far.  Doesn't the current code on PMY already need to buffer
>>> packets
>>> that are generated after the first miscompare
>>
>> Yes, they are buffered,
>>
>>> and before the checkpoint and
>>> then release them at the checkpoint?
>>
>> but will be release only if the packets compare returns identical. so in order
>> to support this fallback mode, we need to modify it to release the packets at
>> the checkpoint, there won't be too much code though.
>>
>
> No, when do checkpoint, we will send all the residual queued packets.
> So it is already supported.

Great, my memory is wrong, sorry...

>
>>>
>>> Dave
>>>
>>>>
>>>>>
>>>>> Dave
>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>> Dave
>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> -Gonglei
>>>>>>>>
>>>>>>> --
>>>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>>>
>>>>>>> .
>>>>>>>
>>>>>>
>>>>>>
>>>>> --
>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>> .
>>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Yang.
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>> .
>>>
>>
>
>
> .
>

-- 
Thanks,
Yang.

Re: [Qemu-devel] [POC] colo-proxy in qemu

[zhanghailiang]

On 2015/7/31 1:53, Dr. David Alan Gilbert wrote:
> * Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
>>
>>
>> On 07/30/2015 09:59 PM, Dr. David Alan Gilbert wrote:
>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>>> On 2015/7/30 20:30, Dr. David Alan Gilbert wrote:
>>>>> * Gonglei (arei.gonglei@huawei.com) wrote:
>>>>>> On 2015/7/30 19:56, Dr. David Alan Gilbert wrote:
>>>>>>> * Jason Wang (jasowang@redhat.com) wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 07/30/2015 04:03 PM, Dr. David Alan Gilbert wrote:
>>>>>>>>> * Dong, Eddie (eddie.dong@intel.com) wrote:
>>>>>>>>>>>> A question here, the packet comparing may be very tricky. For example,
>>>>>>>>>>>> some protocol use random data to generate unpredictable id or
>>>>>>>>>>>> something else. One example is ipv6_select_ident() in Linux. So COLO
>>>>>>>>>>>> needs a mechanism to make sure PVM and SVM can generate same random
>>>>>>>>>>> data?
>>>>>>>>>>> Good question, the random data connection is a big problem for COLO. At
>>>>>>>>>>> present, it will trigger checkpoint processing because of the different random
>>>>>>>>>>> data.
>>>>>>>>>>> I don't think any mechanisms can assure two different machines generate the
>>>>>>>>>>> same random data. If you have any ideas, pls tell us :)
>>>>>>>>>>>
>>>>>>>>>>> Frequent checkpoint can handle this scenario, but maybe will cause the
>>>>>>>>>>> performance poor. :(
>>>>>>>>>>>
>>>>>>>>>> The assumption is that, after VM checkpoint, SVM and PVM have identical internal state, so the pattern used to generate random data has high possibility to generate identical data at short time, at least...
>>>>>>>>> They do diverge pretty quickly though; I have simple examples which
>>>>>>>>> reliably cause a checkpoint because of simple randomness in applications.
>>>>>>>>>
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>>>
>>>>>>>> And it will become even worse if hwrng is used in guest.
>>>>>>>
>>>>>>> Yes; it seems quite application dependent;  (on IPv4) an ssh connection,
>>>>>>> once established, tends to work well without triggering checkpoints;
>>>>>>> and static web pages also work well.  Examples of things that do cause
>>>>>>> more checkpoints are, displaying guest statistics (e.g. running top
>>>>>>> in that ssh) which is timing dependent, and dynamically generated
>>>>>>> web pages that include a unique ID (bugzilla's password reset link in
>>>>>>> it's front page was a fun one), I think also establishing
>>>>>>> new encrypted connections cause the same randomness.
>>>>>>>
>>>>>>> However, it's worth remembering that COLO is trying to reduce the
>>>>>>> number of checkpoints compared to a simple checkpointing world
>>>>>>> which would be aiming to do a checkpoint ~100 times a second,
>>>>>>> and for compute bound workloads, or ones that don't expose
>>>>>>> the randomness that much, it can get checkpoints of a few seconds
>>>>>>> in length which greatly reduces the overhead.
>>>>>>>
>>>>>>
>>>>>> Yes. That's the truth.
>>>>>> We can set two different modes for different scenarios. Maybe Named
>>>>>> 1) frequent checkpoint mode for multi-connections and randomness scenarios
>>>>>> and 2) non-frequent checkpoint mode for other scenarios.
>>>>>>
>>>>>> But that's the next plan, we are thinking about that.
>>>>>
>>>>> I have some code that tries to automatically switch between those;
>>>>> it measures the checkpoint lengths, and if they're consistently short
>>>>> it sends a different message byte to the secondary at the start of the
>>>>> checkpoint, so that it doesn't bother running.   Every so often it
>>>>> then flips back to a COLO checkpoint to see if the checkpoints
>>>>> are still really fast.
>>>>>
>>>>
>>>> Do you mean if there are consistent checkpoint requests, not do checkpoint but just send a special message to SVM?
>>>> Resume to common COLO mode until the checkpoint lengths is so not short ?
>>>
>>>    We still have to do checkpoints, but we send a special message to the SVM so that
>>> the SVM just takes the checkpoint but does not run.
>>>
>>>    I'll send the code after I've updated it to your current version; but it's
>>> quite rough/experimental.
>>>

Yes, please, we can merge them into our branch. ;)

>>> It works something like
>>>
>>>   -----------run PVM     run SVM
>>>       COLO     <long gap>
>>>       mode       miscompare
>>>                  checkpoint
>>>   -----------run PVM     run SVM
>>>       COLO     <short gap>
>>>       mode       miscompare
>>>                  checkpoint
>>>   -----------run PVM     run SVM
>>>       COLO     <short gap>
>>>       mode       miscompare         < After a few short runs
>>>                  checkpoint
>>>   -----------run PVM     SVM idle   \
>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>       mode       checkpoint         /
>>>   -----------run PVM     run SVM
>>>       COLO     <short gap>          < Still a short gap
>>>       mode       miscompare
>>>   -----------run PVM     SVM idle   \
>>>     Passive    <fixed delay>        |  - repeat 'n' times
>>>       mode       checkpoint         /
>>>   -----------run PVM     run SVM
>>>       COLO     <long gap>          < long gap now, stay in COLO
>>>       mode       miscompare
>>>                  checkpoint
>>>   -----------run PVM     run SVM
>>>       COLO     <long gap>
>>>       mode       miscompare
>>>                  checkpoint
>>>
>>> So it saves the CPU time on the SVM, and the comparison traffic, and is
>>> automatic at switching into the passive mode.
>>>

That's a good solution, actually, we have a plan to realize the checkpoint strategy,
which can automatically adapt to different situation, including period checkpoint (MC/Remus mode),
COLO mode, mix mode (just like your above method), also, it it a good idea to apply command for users to
choose the mode~

>>> It used to be more useful, but your minimum COLO run time that you
>>> added a few versions ago helps a lot in the cases where there are miscompares,
>>> and the delay after the miscompare before you take the checkpoint also helps
>>> in the case where the data is very random.
>>
>> This is great! This is exactly what we were thinking about, when random
>> scenario will fallback to MC/Remus like FT. Thank you very much!
>> I have a question, do you also modify colo-proxy kernel module? because
>> in the fixed checkpoint mode, I think we need to buffer the network
>> packets, and release them at checkpoint.
>
> Yes, we do need to buffer and release them at the end, but I've not modified
> colo-proxy so far.  Doesn't the current code on PMY already need to buffer packets
> that are generated after the first miscompare and before the checkpoint and
> then release them at the checkpoint?
>

Yes, we have support this already, for PMY, it will queue all packets sent from PVM,
and waits for corresponding net packets sent from SVM, after comparing, if it is consistent
it will be send out, if not, there will be a mis-compare checkpoint request. After a checkpoint,
we will release all packets that queued. So if the SVM is not running, we will queued all packets,
without any comparison, and just release them after a new checkpoint.

Thanks,
zhanghailiang

>>
>>>
>>> Dave
>>>
>>>>
>>>> Thanks.
>>>>
>>>>> Dave
>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> -Gonglei
>>>>>>
>>>>> --
>>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>>
>>>>> .
>>>>>
>>>>
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>> .
>>>
>>
>> --
>> Thanks,
>> Yang.
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>

[Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Samuel Thibault]

Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>

---
Jan Kiszka, le Wed 29 Jul 2015 09:36:15 +0200, a écrit :
> On 2015-07-29 00:12, Samuel Thibault wrote:
> > Hello,
> > 
> > Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
> >> Of course, I'm fine with handing this over to someone who'd like to
> >> pick up. Do we have volunteers?
> >>
> >> Samuel, would you like to do this? As a subsystem maintainer, you are
> >> already familiar with QEMU processes.
> > 
> > I can help with maintenance, yes.
> 
> "Help with" will easily mean "be the one and only". ;)

Right, I understand that :) That's still preferable to have two contact
people.

diff --git a/MAINTAINERS b/MAINTAINERS
index 978b717..b506df5 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1018,6 +1018,7 @@ F: scripts/qmp/
 T: git git://repo.or.cz/qemu/armbru.git qapi-next
 
 SLIRP
+M: Samuel Thibault <samuel.thibault@ens-lyon.org>
 M: Jan Kiszka <jan.kiszka@siemens.com>
 S: Maintained
 F: slirp/

Re: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

On Thu, Aug 06, 2015 at 01:10:14PM +0200, Samuel Thibault wrote:
> Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
> 
> ---
> Jan Kiszka, le Wed 29 Jul 2015 09:36:15 +0200, a écrit :
> > On 2015-07-29 00:12, Samuel Thibault wrote:
> > > Hello,
> > > 
> > > Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
> > >> Of course, I'm fine with handing this over to someone who'd like to
> > >> pick up. Do we have volunteers?
> > >>
> > >> Samuel, would you like to do this? As a subsystem maintainer, you are
> > >> already familiar with QEMU processes.
> > > 
> > > I can help with maintenance, yes.
> > 
> > "Help with" will easily mean "be the one and only". ;)
> 
> Right, I understand that :) That's still preferable to have two contact
> people.

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH] MAINTAINERS: Add Samuel Thibault as slirp maintainer

[Jan Kiszka]

On 2015-08-06 13:10, Samuel Thibault wrote:
> Signed-off-by: Samuel Thibault <samuel.thibault@ens-lyon.org>
> 
> ---
> Jan Kiszka, le Wed 29 Jul 2015 09:36:15 +0200, a écrit :
>> On 2015-07-29 00:12, Samuel Thibault wrote:
>>> Hello,
>>>
>>> Jan Kiszka, le Mon 27 Jul 2015 15:33:27 +0200, a écrit :
>>>> Of course, I'm fine with handing this over to someone who'd like to
>>>> pick up. Do we have volunteers?
>>>>
>>>> Samuel, would you like to do this? As a subsystem maintainer, you are
>>>> already familiar with QEMU processes.
>>>
>>> I can help with maintenance, yes.
>>
>> "Help with" will easily mean "be the one and only". ;)
> 
> Right, I understand that :) That's still preferable to have two contact
> people.
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 978b717..b506df5 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1018,6 +1018,7 @@ F: scripts/qmp/
>  T: git git://repo.or.cz/qemu/armbru.git qapi-next
>  
>  SLIRP
> +M: Samuel Thibault <samuel.thibault@ens-lyon.org>
>  M: Jan Kiszka <jan.kiszka@siemens.com>
>  S: Maintained
>  F: slirp/
> 

Acked-by: Jan Kiszka <jan.kiszka@siemens.com>

Thanks a lot, Samuel, for offering your help! I know how full your
agenda already is, so it's even more appreciated.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux