[Qemu-devel] [PATCH V3 0/3] COLO-compare: Make COLO-compare support remote COLO-frame

[Qemu-devel] [PATCH V3 0/3] COLO-compare: Make COLO-compare support remote COLO-frame

[Zhang Chen]

This series focus on COLO-proxy remote colo-frame support.
Xen COLO-frame is the first user. We add a new chardev socket
in colo-compare as the way of communicate with remote COLO-frame.
And remote COLO-frame notify colo-proxy part depend on this serise:
https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg03904.html

I will send another part of this series after depend patchset have
been merged.

V3:
 - Fix codestyle.

V2:
 - Rename this series.
 - Change communication way to remote colo-frame.
 - Some bugfix.
 - Split the main function, anther part wait depend patchset.


Zhang Chen (3):
  COLO-compare: Add new parameter for communicate with remote colo-frame
  COLO-compare: Add remote checkpoint notify chardev socket handler
    frame
  COLO-compare: Add remote initialization and checkpoint notification

 net/colo-compare.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
 qemu-options.hx    | 41 ++++++++++++++++++++----
 2 files changed, 124 insertions(+), 8 deletions(-)

-- 
2.7.4

[Qemu-devel] [PATCH V3 1/3] COLO-compare: Add new parameter for communicate with remote colo-frame

[Zhang Chen]

We add the "notify_dev=chardevID" parameter. colo-compare can connect with
remote colo-frame through chardev socket, Xen colo-frame is the first user.
We can notify colo-frame do checkpoint events.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
 net/colo-compare.c | 20 ++++++++++++++++++++
 qemu-options.hx    | 41 +++++++++++++++++++++++++++++++++++------
 2 files changed, 55 insertions(+), 6 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 2639c7f..6f8c439 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -68,6 +68,7 @@ typedef struct CompareState {
     char *pri_indev;
     char *sec_indev;
     char *outdev;
+    char *notify_dev;
     CharBackend chr_pri_in;
     CharBackend chr_sec_in;
     CharBackend chr_out;
@@ -653,6 +654,21 @@ static void compare_set_outdev(Object *obj, const char *value, Error **errp)
     s->outdev = g_strdup(value);
 }
 
+static char *compare_get_notify_dev(Object *obj, Error **errp)
+{
+    CompareState *s = COLO_COMPARE(obj);
+
+    return g_strdup(s->notify_dev);
+}
+
+static void compare_set_notify_dev(Object *obj, const char *value, Error **errp)
+{
+    CompareState *s = COLO_COMPARE(obj);
+
+    g_free(s->notify_dev);
+    s->notify_dev = g_strdup(value);
+}
+
 static void compare_pri_rs_finalize(SocketReadState *pri_rs)
 {
     CompareState *s = container_of(pri_rs, CompareState, pri_rs);
@@ -795,6 +811,9 @@ static void colo_compare_init(Object *obj)
     object_property_add_str(obj, "outdev",
                             compare_get_outdev, compare_set_outdev,
                             NULL);
+    object_property_add_str(obj, "notify_dev",
+                            compare_get_notify_dev, compare_set_notify_dev,
+                            NULL);
 }
 
 static void colo_compare_finalize(Object *obj)
@@ -819,6 +838,7 @@ static void colo_compare_finalize(Object *obj)
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
+    g_free(s->notify_dev);
 }
 
 static const TypeInfo colo_compare_info = {
diff --git a/qemu-options.hx b/qemu-options.hx
index f63f7dc..1f3b3d2 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -4092,23 +4092,26 @@ Dump the network traffic on netdev @var{dev} to the file specified by
 The file format is libpcap, so it can be analyzed with tools such as tcpdump
 or Wireshark.
 
-@item -object colo-compare,id=@var{id},primary_in=@var{chardevid},secondary_in=@var{chardevid},
-outdev=@var{chardevid}
+@item -object colo-compare,id=@var{id},primary_in=@var{chardevid},secondary_in=@var{chardevid},outdev=@var{chardevid},notify_dev=@var{chardevid}
 
-Colo-compare gets packet from primary_in@var{chardevid} and secondary_in@var{chardevid}, than compare primary packet with
-secondary packet. If the packets are same, we will output primary
-packet to outdev@var{chardevid}, else we will notify colo-frame
+Colo-compare gets packet from primary_in@var{chardevid} and secondary_in@var{chardevid},
+than compare primary packet with secondary packet. If the packets are same,
+we will output primary packet to outdev@var{chardevid},else we will notify colo-frame
 do checkpoint and send primary packet to outdev@var{chardevid}.
+If we use Xen COLO, will use notify_dev to notify remote colo-frame do checkpoint(like Xen),
+at the same time send primary packet to outdev@var{chardevid}.
 
 we must use it with the help of filter-mirror and filter-redirector.
 
 @example
 
+KVM COLO
+
 primary:
 -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
 -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
 -chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
--chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
+-chardev socket,id=compare1,host=3.3.3.3,port=9004,server,wait
 -chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
 -chardev socket,id=compare0-0,host=3.3.3.3,port=9001
 -chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
@@ -4126,6 +4129,32 @@ secondary:
 -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
 -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
 
+
+Xen COLO
+
+primary:
+-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
+-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
+-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
+-chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
+-chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
+-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
+-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
+-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
+-chardev socket,id=notify_way,host=3.3.3.3,port=9009,server,nowait
+-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
+-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
+-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
+-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,notify_dev=nofity_way
+
+secondary:
+-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown
+-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
+-chardev socket,id=red0,host=3.3.3.3,port=9003
+-chardev socket,id=red1,host=3.3.3.3,port=9004
+-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
+-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
+
 @end example
 
 If you want to know the detail of above command line, you can read
-- 
2.7.4

[Qemu-devel] [PATCH V3 2/3] COLO-compare: Add remote checkpoint notify chardev socket handler frame

[Zhang Chen]

Add chardev handler to get remote colo-frame's notify, Xen colo is the first user.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
 net/colo-compare.c | 39 +++++++++++++++++++++++++++++++++++++--
 1 file changed, 37 insertions(+), 2 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 6f8c439..10a328e 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -72,8 +72,10 @@ typedef struct CompareState {
     CharBackend chr_pri_in;
     CharBackend chr_sec_in;
     CharBackend chr_out;
+    CharBackend chr_notify_dev;
     SocketReadState pri_rs;
     SocketReadState sec_rs;
+    SocketReadState notify_rs;
 
     /* connection list: the connections belonged to this NIC could be found
      * in this list.
@@ -567,6 +569,19 @@ static void compare_sec_chr_in(void *opaque, const uint8_t *buf, int size)
     }
 }
 
+static void compare_notify_chr(void *opaque, const uint8_t *buf, int size)
+{
+    CompareState *s = COLO_COMPARE(opaque);
+    int ret;
+
+    ret = net_fill_rstate(&s->notify_rs, buf, size);
+    if (ret == -1) {
+        qemu_chr_fe_set_handlers(&s->chr_notify_dev, NULL, NULL, NULL,
+                                 NULL, NULL, true);
+        error_report("colo-compare notify_dev error");
+    }
+}
+
 /*
  * Check old packet regularly so it can watch for any packets
  * that the secondary hasn't produced equivalents of.
@@ -589,9 +604,14 @@ static void *colo_compare_thread(void *opaque)
     s->worker_context = g_main_context_new();
 
     qemu_chr_fe_set_handlers(&s->chr_pri_in, compare_chr_can_read,
-                          compare_pri_chr_in, NULL, s, s->worker_context, true);
+                             compare_pri_chr_in, NULL, s,
+                             s->worker_context, true);
     qemu_chr_fe_set_handlers(&s->chr_sec_in, compare_chr_can_read,
-                          compare_sec_chr_in, NULL, s, s->worker_context, true);
+                             compare_sec_chr_in, NULL, s,
+                             s->worker_context, true);
+    qemu_chr_fe_set_handlers(&s->chr_notify_dev, compare_chr_can_read,
+                             compare_notify_chr, NULL, s,
+                             s->worker_context, true);
 
     s->compare_loop = g_main_loop_new(s->worker_context, FALSE);
 
@@ -694,6 +714,10 @@ static void compare_sec_rs_finalize(SocketReadState *sec_rs)
     }
 }
 
+static void compare_notify_rs_finalize(SocketReadState *notify_rs)
+{
+    /* Get remote colo-frame's notify and handle the message */
+}
 
 /*
  * Return 0 is success.
@@ -757,6 +781,15 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
         return;
     }
 
+    /* This parameter is Optional, Remote checkpoint notify need it */
+    if (s->notify_dev) {
+        if (find_and_check_chardev(&chr, s->notify_dev, errp) ||
+            !qemu_chr_fe_init(&s->chr_notify_dev, chr, errp)) {
+            return;
+        }
+        net_socket_rs_init(&s->notify_rs, compare_notify_rs_finalize);
+    }
+
     net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize);
     net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize);
 
@@ -825,6 +858,7 @@ static void colo_compare_finalize(Object *obj)
     qemu_chr_fe_set_handlers(&s->chr_sec_in, NULL, NULL, NULL, NULL,
                              s->worker_context, true);
     qemu_chr_fe_deinit(&s->chr_out);
+    qemu_chr_fe_deinit(&s->chr_notify_dev);
 
     g_main_loop_quit(s->compare_loop);
     qemu_thread_join(&s->thread);
@@ -839,6 +873,7 @@ static void colo_compare_finalize(Object *obj)
     g_free(s->sec_indev);
     g_free(s->outdev);
     g_free(s->notify_dev);
+
 }
 
 static const TypeInfo colo_compare_info = {
-- 
2.7.4

[Qemu-devel] [PATCH V3 3/3] COLO-compare: Add remote initialization and checkpoint notification

[Zhang Chen]

Initialize remote communication socket and make a handshack with remote colo-frame.
Then use this way to notify remote colo-frame do checkpoint.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
 net/colo-compare.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 10a328e..8886640 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -100,6 +100,12 @@ enum {
     SECONDARY_IN,
 };
 
+typedef enum ProxyRemoteRequest {
+    USERSPACE_PROXY_INIT = 0,
+    USERSPACE_PROXY_GET_REMOTE_INIT = 1,
+    USERSPACE_PROXY_NOTIFY_CHECKPOINT = 2
+} ProxyRemoteRequest;
+
 static int compare_chr_send(CharBackend *out,
                             const uint8_t *buf,
                             uint32_t size);
@@ -498,6 +504,18 @@ static void colo_compare_connection(void *opaque, void *user_data)
             trace_colo_compare_main("packet different");
             g_queue_push_tail(&conn->primary_list, pkt);
             /* TODO: colo_notify_checkpoint();*/
+            /*
+             * If we have notify_dev that means COLO didn't running on KVM,
+             * So we use remote checkpoint notify, Xen is the first user.
+             */
+            if (s->notify_dev) {
+                ProxyRemoteRequest msg = USERSPACE_PROXY_NOTIFY_CHECKPOINT;
+                ret = compare_chr_send(&s->chr_notify_dev, (uint8_t *)msg,
+                                       sizeof(msg));
+                if (ret < 0) {
+                    error_report("Notify remote COLO-frame failed");
+                }
+            }
             break;
         }
     }
@@ -716,7 +734,21 @@ static void compare_sec_rs_finalize(SocketReadState *sec_rs)
 
 static void compare_notify_rs_finalize(SocketReadState *notify_rs)
 {
+    CompareState *s = container_of(notify_rs, CompareState, notify_rs);
+
     /* Get remote colo-frame's notify and handle the message */
+    char *data = g_memdup(notify_rs->buf, notify_rs->packet_len);
+    ProxyRemoteRequest msg = USERSPACE_PROXY_GET_REMOTE_INIT;
+    ProxyRemoteRequest msg_recv = *data;
+    int ret;
+
+    if (msg_recv == USERSPACE_PROXY_INIT) {
+        ret = compare_chr_send(&s->chr_notify_dev, (uint8_t *)msg,
+                               sizeof(msg));
+        if (ret < 0) {
+            error_report("Notify remote COLO-frame INIT failed");
+        }
+    }
 }
 
 /*
-- 
2.7.4

Re: [Qemu-devel] [PATCH V3 0/3] COLO-compare: Make COLO-compare support remote COLO-frame

[Jason Wang]

On 2017年06月06日 16:12, Zhang Chen wrote:
> This series focus on COLO-proxy remote colo-frame support.
> Xen COLO-frame is the first user. We add a new chardev socket
> in colo-compare as the way of communicate with remote COLO-frame.
> And remote COLO-frame notify colo-proxy part depend on this serise:
> https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg03904.html
>
> I will send another part of this series after depend patchset have
> been merged.
>
> V3:
>   - Fix codestyle.
>
> V2:
>   - Rename this series.
>   - Change communication way to remote colo-frame.
>   - Some bugfix.
>   - Split the main function, anther part wait depend patchset.
>
>
> Zhang Chen (3):
>    COLO-compare: Add new parameter for communicate with remote colo-frame
>    COLO-compare: Add remote checkpoint notify chardev socket handler
>      frame
>    COLO-compare: Add remote initialization and checkpoint notification
>
>   net/colo-compare.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
>   qemu-options.hx    | 41 ++++++++++++++++++++----
>   2 files changed, 124 insertions(+), 8 deletions(-)
>

Thanks for the series.

To speed up things, I would like to see IOThread conversion first. Then 
we can have this on top.

Re: [Qemu-devel] [PATCH V3 0/3] COLO-compare: Make COLO-compare support remote COLO-frame

[Dr. David Alan Gilbert]

* Zhang Chen (zhangchen.fnst@cn.fujitsu.com) wrote:
> This series focus on COLO-proxy remote colo-frame support.
> Xen COLO-frame is the first user. We add a new chardev socket
> in colo-compare as the way of communicate with remote COLO-frame.
> And remote COLO-frame notify colo-proxy part depend on this serise:
> https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg03904.html

Can you explain a bit more about how the 'remote colo-frame' works?
Is this the comparison that's separate or what?

Dave

> I will send another part of this series after depend patchset have
> been merged.
> 
> V3:
>  - Fix codestyle.
> 
> V2:
>  - Rename this series.
>  - Change communication way to remote colo-frame.
>  - Some bugfix.
>  - Split the main function, anther part wait depend patchset.
> 
> 
> Zhang Chen (3):
>   COLO-compare: Add new parameter for communicate with remote colo-frame
>   COLO-compare: Add remote checkpoint notify chardev socket handler
>     frame
>   COLO-compare: Add remote initialization and checkpoint notification
> 
>  net/colo-compare.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
>  qemu-options.hx    | 41 ++++++++++++++++++++----
>  2 files changed, 124 insertions(+), 8 deletions(-)
> 
> -- 
> 2.7.4
> 
> 
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH V3 0/3] COLO-compare: Make COLO-compare support remote COLO-frame

[Zhang Chen]

On 06/29/2017 06:36 PM, Dr. David Alan Gilbert wrote:
> * Zhang Chen (zhangchen.fnst@cn.fujitsu.com) wrote:
>> This series focus on COLO-proxy remote colo-frame support.
>> Xen COLO-frame is the first user. We add a new chardev socket
>> in colo-compare as the way of communicate with remote COLO-frame.
>> And remote COLO-frame notify colo-proxy part depend on this serise:
>> https://lists.nongnu.org/archive/html/qemu-devel/2017-04/msg03904.html
> Can you explain a bit more about how the 'remote colo-frame' works?
> Is this the comparison that's separate or what?

COLO consists of three modules, colo-frame, colo-proxy, block-replication.
If we use qemu/kvm COLO, the three parts all in qemu side.
If we use Xen COLO, the colo-frame in Xen side(the remote COLO-frame),
and colo-proxy,block-replication in qemu side. So, in this case,
remote colo-frame need use qmp command and socket connect to 
block-replication
and colo-proxy finish COLO job.
KVM COLO and Xen COLO use the same colo-proxy to compare net packet.

Thanks
Zhang Chen

>
> Dave
>
>> I will send another part of this series after depend patchset have
>> been merged.
>>
>> V3:
>>   - Fix codestyle.
>>
>> V2:
>>   - Rename this series.
>>   - Change communication way to remote colo-frame.
>>   - Some bugfix.
>>   - Split the main function, anther part wait depend patchset.
>>
>>
>> Zhang Chen (3):
>>    COLO-compare: Add new parameter for communicate with remote colo-frame
>>    COLO-compare: Add remote checkpoint notify chardev socket handler
>>      frame
>>    COLO-compare: Add remote initialization and checkpoint notification
>>
>>   net/colo-compare.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
>>   qemu-options.hx    | 41 ++++++++++++++++++++----
>>   2 files changed, 124 insertions(+), 8 deletions(-)
>>
>> -- 
>> 2.7.4
>>
>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
>

-- 
Thanks
Zhang Chen