[Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Zhang Chen]

This series focus on COLO Xen support.
We add a new chardev socket in colo-compare as the way
of communicate with Xen COLO-frame.
Xen part:
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg02067.html

This series is a frame, have some TODO job depend on this patch:
https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg05055.html


Zhang Chen (3):
  COLO-compare: Add new parameter for communicate with Xen colo-frame
  COLO-compare: Add Xen notify chardev socket handler frame
  COLO-compare: Add colo-compare Xen notify

 net/colo-compare.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 qemu-options.hx    | 38 ++++++++++++++++++++++++---
 2 files changed, 111 insertions(+), 4 deletions(-)

-- 
2.7.4

[Qemu-devel] [PATCH 1/3] COLO-compare: Add new parameter for communicate with Xen colo-frame

[Zhang Chen]

We add the "notify_dev=chardevID" parameter. colo-compare can connect with
Xen colo-frame through chardev socket.like notify colo-frame do checkpoint events.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
 net/colo-compare.c | 20 ++++++++++++++++++++
 qemu-options.hx    | 38 ++++++++++++++++++++++++++++++++++----
 2 files changed, 54 insertions(+), 4 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 6b38723..57d336b 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -68,6 +68,7 @@ typedef struct CompareState {
     char *pri_indev;
     char *sec_indev;
     char *outdev;
+    char *notify_dev;
     CharBackend chr_pri_in;
     CharBackend chr_sec_in;
     CharBackend chr_out;
@@ -585,6 +586,21 @@ static void compare_set_outdev(Object *obj, const char *value, Error **errp)
     s->outdev = g_strdup(value);
 }
 
+static char *compare_get_notify_dev(Object *obj, Error **errp)
+{
+    CompareState *s = COLO_COMPARE(obj);
+
+    return g_strdup(s->notify_dev);
+}
+
+static void compare_set_notify_dev(Object *obj, const char *value, Error **errp)
+{
+    CompareState *s = COLO_COMPARE(obj);
+
+    g_free(s->notify_dev);
+    s->notify_dev = g_strdup(value);
+}
+
 static void compare_pri_rs_finalize(SocketReadState *pri_rs)
 {
     CompareState *s = container_of(pri_rs, CompareState, pri_rs);
@@ -727,6 +743,9 @@ static void colo_compare_init(Object *obj)
     object_property_add_str(obj, "outdev",
                             compare_get_outdev, compare_set_outdev,
                             NULL);
+    object_property_add_str(obj, "notify_dev",
+                            compare_get_notify_dev, compare_set_notify_dev,
+                            NULL);
 }
 
 static void colo_compare_finalize(Object *obj)
@@ -751,6 +770,7 @@ static void colo_compare_finalize(Object *obj)
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
+    g_free(s->notify_dev);
 }
 
 static const TypeInfo colo_compare_info = {
diff --git a/qemu-options.hx b/qemu-options.hx
index 5633d39..9a3fdcf 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3963,17 +3963,21 @@ The file format is libpcap, so it can be analyzed with tools such as tcpdump
 or Wireshark.
 
 @item -object colo-compare,id=@var{id},primary_in=@var{chardevid},secondary_in=@var{chardevid},
-outdev=@var{chardevid}
+outdev=@var{chardevid},notify_dev=@var{chardevid}
 
-Colo-compare gets packet from primary_in@var{chardevid} and secondary_in@var{chardevid}, than compare primary packet with
-secondary packet. If the packets are same, we will output primary
-packet to outdev@var{chardevid}, else we will notify colo-frame
+Colo-compare gets packet from primary_in@var{chardevid} and secondary_in@var{chardevid},
+than compare primary packet with secondary packet. If the packets are same,
+we will output primary packet to outdev@var{chardevid},else we will notify colo-frame
 do checkpoint and send primary packet to outdev@var{chardevid}.
+If we use Xen COLO, will use notify_dev to notify Xen colo-frame do checkpoint,
+at the same time send primary packet to outdev@var{chardevid}.
 
 we must use it with the help of filter-mirror and filter-redirector.
 
 @example
 
+KVM COLO
+
 primary:
 -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
 -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
@@ -3996,6 +4000,32 @@ secondary:
 -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
 -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
 
+
+Xen COLO
+
+primary:
+-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
+-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
+-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
+-chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
+-chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
+-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
+-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
+-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
+-chardev socket,id=notify_way,host=3.3.3.3,port=9009,server,nowait
+-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
+-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
+-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
+-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,notify_dev=nofity_way
+
+secondary:
+-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown
+-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
+-chardev socket,id=red0,host=3.3.3.3,port=9003
+-chardev socket,id=red1,host=3.3.3.3,port=9004
+-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
+-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
+
 @end example
 
 If you want to know the detail of above command line, you can read
-- 
2.7.4

[Qemu-devel] [PATCH 2/3] COLO-compare: Add Xen notify chardev socket handler frame

[Zhang Chen]

Add chardev handler to get Xen colo-frame's notify.

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
 net/colo-compare.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 57d336b..947a9e2 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -72,8 +72,10 @@ typedef struct CompareState {
     CharBackend chr_pri_in;
     CharBackend chr_sec_in;
     CharBackend chr_out;
+    CharBackend chr_notify_dev;
     SocketReadState pri_rs;
     SocketReadState sec_rs;
+    SocketReadState notify_rs;
 
     /* connection list: the connections belonged to this NIC could be found
      * in this list.
@@ -513,6 +515,19 @@ static gboolean check_old_packet_regular(void *opaque)
     return TRUE;
 }
 
+static void compare_notify_chr(void *opaque, const uint8_t *buf, int size)
+{
+    CompareState *s = COLO_COMPARE(opaque);
+    int ret;
+
+    ret = net_fill_rstate(&s->notify_rs, buf, size);
+    if (ret == -1) {
+        qemu_chr_fe_set_handlers(&s->chr_notify_dev, NULL, NULL, NULL,
+                                 NULL, NULL, true);
+        error_report("colo-compare notify_dev error");
+    }
+}
+
 static void *colo_compare_thread(void *opaque)
 {
     CompareState *s = opaque;
@@ -524,6 +539,8 @@ static void *colo_compare_thread(void *opaque)
                           compare_pri_chr_in, NULL, s, s->worker_context, true);
     qemu_chr_fe_set_handlers(&s->chr_sec_in, compare_chr_can_read,
                           compare_sec_chr_in, NULL, s, s->worker_context, true);
+    qemu_chr_fe_set_handlers(&s->chr_notify_dev, compare_chr_can_read,
+                             compare_notify_chr, NULL, s, s->worker_context, true);
 
     s->compare_loop = g_main_loop_new(s->worker_context, FALSE);
 
@@ -626,6 +643,10 @@ static void compare_sec_rs_finalize(SocketReadState *sec_rs)
     }
 }
 
+static void compare_notify_rs_finalize(SocketReadState *notify_rs)
+{
+    /* Get Xen colo-frame's notify and handle the message */
+}
 
 /*
  * Return 0 is success.
@@ -689,8 +710,14 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
         return;
     }
 
+    if (find_and_check_chardev(&chr, s->notify_dev, errp) ||
+        !qemu_chr_fe_init(&s->chr_notify_dev, chr, errp)) {
+        return;
+    }
+
     net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize);
     net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize);
+    net_socket_rs_init(&s->notify_rs, compare_notify_rs_finalize);
 
     g_queue_init(&s->conn_list);
 
@@ -757,6 +784,7 @@ static void colo_compare_finalize(Object *obj)
     qemu_chr_fe_set_handlers(&s->chr_sec_in, NULL, NULL, NULL, NULL,
                              s->worker_context, true);
     qemu_chr_fe_deinit(&s->chr_out);
+    qemu_chr_fe_deinit(&s->chr_notify_dev);
 
     g_main_loop_quit(s->compare_loop);
     qemu_thread_join(&s->thread);
-- 
2.7.4

[Qemu-devel] [PATCH 3/3] COLO-compare: Add colo-compare Xen notify

[Zhang Chen]

This patch have some TODO job.
Depend on patch:
https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg05055.html

Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
 net/colo-compare.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 947a9e2..073d1f8 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -103,6 +103,7 @@ enum {
 static int compare_chr_send(CharBackend *out,
                             const uint8_t *buf,
                             uint32_t size);
+static void colo_flush_packets(void *opaque, void *user_data);
 
 static gint seq_sorter(Packet *a, Packet *b, gpointer data)
 {
@@ -430,6 +431,15 @@ static void colo_compare_connection(void *opaque, void *user_data)
             trace_colo_compare_main("packet different");
             g_queue_push_tail(&conn->primary_list, pkt);
             /* TODO: colo_notify_checkpoint();*/
+            /* If we have notify_dev that means COLO run on Xen */
+            if (s->notify_dev) {
+                char msg[] = "DO_CHECKPOINT";
+                ret = compare_chr_send(&s->chr_notify_dev, (uint8_t *)msg,
+                                       strlen(msg));
+                if (ret < 0) {
+                    error_report("Notify Xen COLO-frame failed");
+                }
+            }
             break;
         }
     }
@@ -645,7 +655,26 @@ static void compare_sec_rs_finalize(SocketReadState *sec_rs)
 
 static void compare_notify_rs_finalize(SocketReadState *notify_rs)
 {
+    CompareState *s = container_of(notify_rs, CompareState, notify_rs);
+
     /* Get Xen colo-frame's notify and handle the message */
+    char *data = g_memdup(notify_rs->buf, notify_rs->packet_len);
+    char msg[] = "COLO_COMPARE_GET_XEN_INIT";
+    int ret;
+
+    if (!strcmp(data, "COLO_USERSPACE_PROXY_INIT")) {
+        ret = compare_chr_send(&s->chr_notify_dev, (uint8_t *)msg,
+                               strlen(msg));
+        if (ret < 0) {
+            error_report("Notify Xen COLO-frame INIT failed");
+        }
+    }
+
+    if (!strcmp(data, "COLO_CHECKPOINT")) {
+        /* colo-compare do checkpoint, flush pri packet and remove sec packet */
+        g_queue_foreach(&s->conn_list, colo_flush_packets, s);
+    }
+
 }
 
 /*
-- 
2.7.4

Re: [Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Zhang Chen]

Hi~~~ All~

No news for a long time, anyone can give me some comments?


Thanks

Zhang Chen


On 02/28/2017 06:48 PM, Zhang Chen wrote:
> This series focus on COLO Xen support.
> We add a new chardev socket in colo-compare as the way
> of communicate with Xen COLO-frame.
> Xen part:
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg02067.html
>
> This series is a frame, have some TODO job depend on this patch:
> https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg05055.html
>
>
> Zhang Chen (3):
>    COLO-compare: Add new parameter for communicate with Xen colo-frame
>    COLO-compare: Add Xen notify chardev socket handler frame
>    COLO-compare: Add colo-compare Xen notify
>
>   net/colo-compare.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   qemu-options.hx    | 38 ++++++++++++++++++++++++---
>   2 files changed, 111 insertions(+), 4 deletions(-)
>

-- 
Thanks
Zhang Chen

Re: [Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Jason Wang]

On 2017年03月13日 14:18, Zhang Chen wrote:
> Hi~~~ All~
>
> No news for a long time, anyone can give me some comments?

Hi,

A question is why use two kinds of colo-frames? This seems not good as 
lots of the code were duplicated.

Thanks

>
>
> Thanks
>
> Zhang Chen
>
>
> On 02/28/2017 06:48 PM, Zhang Chen wrote:
>> This series focus on COLO Xen support.
>> We add a new chardev socket in colo-compare as the way
>> of communicate with Xen COLO-frame.
>> Xen part:
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg02067.html 
>>
>>
>> This series is a frame, have some TODO job depend on this patch:
>> https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg05055.html
>>
>>
>> Zhang Chen (3):
>>    COLO-compare: Add new parameter for communicate with Xen colo-frame
>>    COLO-compare: Add Xen notify chardev socket handler frame
>>    COLO-compare: Add colo-compare Xen notify
>>
>>   net/colo-compare.c | 77 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   qemu-options.hx    | 38 ++++++++++++++++++++++++---
>>   2 files changed, 111 insertions(+), 4 deletions(-)
>>
>

Re: [Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Zhang Chen]

On 03/13/2017 02:28 PM, Jason Wang wrote:
>
>
> On 2017年03月13日 14:18, Zhang Chen wrote:
>> Hi~~~ All~
>>
>> No news for a long time, anyone can give me some comments?
>
> Hi,
>
> A question is why use two kinds of colo-frames? This seems not good as 
> lots of the code were duplicated.

Because Xen colo-frame based on Xen Remus.
Remus do some job like the migration job in qemu/kvm, and in Xen HVM we 
can not do migration in qemu,
So colo-frame code must running in Xen side.
In addition, Xen HVM use qemu to simulation device that Colo-proxy and 
Replication can use same one codes in qemu.

Thanks
Zhang Chen

>
> Thanks
>
>>
>>
>> Thanks
>>
>> Zhang Chen
>>
>>
>> On 02/28/2017 06:48 PM, Zhang Chen wrote:
>>> This series focus on COLO Xen support.
>>> We add a new chardev socket in colo-compare as the way
>>> of communicate with Xen COLO-frame.
>>> Xen part:
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg02067.html 
>>>
>>>
>>> This series is a frame, have some TODO job depend on this patch:
>>> https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg05055.html
>>>
>>>
>>> Zhang Chen (3):
>>>    COLO-compare: Add new parameter for communicate with Xen colo-frame
>>>    COLO-compare: Add Xen notify chardev socket handler frame
>>>    COLO-compare: Add colo-compare Xen notify
>>>
>>>   net/colo-compare.c | 77 
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>   qemu-options.hx    | 38 ++++++++++++++++++++++++---
>>>   2 files changed, 111 insertions(+), 4 deletions(-)
>>>
>>
>
>
>
> .
>

-- 
Thanks
Zhang Chen

Re: [Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Zhang Chen]

On 03/13/2017 03:10 PM, Zhang Chen wrote:
>
>
> On 03/13/2017 02:28 PM, Jason Wang wrote:
>>
>>
>> On 2017年03月13日 14:18, Zhang Chen wrote:
>>> Hi~~~ All~
>>>
>>> No news for a long time, anyone can give me some comments?
>>
>> Hi,
>>
>> A question is why use two kinds of colo-frames? This seems not good 
>> as lots of the code were duplicated.
>
> Because Xen colo-frame based on Xen Remus.
> Remus do some job like the migration job in qemu/kvm, and in Xen HVM 
> we can not do migration in qemu,
> So colo-frame code must running in Xen side.
> In addition, Xen HVM use qemu to simulation device that Colo-proxy and 
> Replication can use same one codes in qemu.

Detail:
https://wiki.xenproject.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

Thanks
Zhang Chen

>
> Thanks
> Zhang Chen
>
>>
>> Thanks
>>
>>>
>>>
>>> Thanks
>>>
>>> Zhang Chen
>>>
>>>
>>> On 02/28/2017 06:48 PM, Zhang Chen wrote:
>>>> This series focus on COLO Xen support.
>>>> We add a new chardev socket in colo-compare as the way
>>>> of communicate with Xen COLO-frame.
>>>> Xen part:
>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg02067.html 
>>>>
>>>>
>>>> This series is a frame, have some TODO job depend on this patch:
>>>> https://lists.nongnu.org/archive/html/qemu-devel/2017-02/msg05055.html
>>>>
>>>>
>>>> Zhang Chen (3):
>>>>    COLO-compare: Add new parameter for communicate with Xen colo-frame
>>>>    COLO-compare: Add Xen notify chardev socket handler frame
>>>>    COLO-compare: Add colo-compare Xen notify
>>>>
>>>>   net/colo-compare.c | 77 
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>   qemu-options.hx    | 38 ++++++++++++++++++++++++---
>>>>   2 files changed, 111 insertions(+), 4 deletions(-)
>>>>
>>>
>>
>>
>>
>> .
>>
>

-- 
Thanks
Zhang Chen

Re: [Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Jason Wang]

On 2017年03月13日 15:21, Zhang Chen wrote:
>
>
> On 03/13/2017 03:10 PM, Zhang Chen wrote:
>>
>>
>> On 03/13/2017 02:28 PM, Jason Wang wrote:
>>>
>>>
>>> On 2017年03月13日 14:18, Zhang Chen wrote:
>>>> Hi~~~ All~
>>>>
>>>> No news for a long time, anyone can give me some comments?
>>>
>>> Hi,
>>>
>>> A question is why use two kinds of colo-frames? This seems not good 
>>> as lots of the code were duplicated.
>>
>> Because Xen colo-frame based on Xen Remus.
>> Remus do some job like the migration job in qemu/kvm, and in Xen HVM 
>> we can not do migration in qemu,
>> So colo-frame code must running in Xen side.
>> In addition, Xen HVM use qemu to simulation device that Colo-proxy 
>> and Replication can use same one codes in qemu.
>
> Detail:
> https://wiki.xenproject.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>
> Thanks
> Zhang Chen 

Ok, the idea looks ok but let's don't make the code specific to any 
hypersior (e.g Xen). Instead, let's define a more generic API like:

1) rename the series to something like "support remote checkpoint", and 
mention Xen is the first user
2) do not use something like "COLO_USERSPACE_PROXY_INIT", you can just 
use a more generic name like "USERSPACE_PROXY_INIT"
3) instead of using raw strings, you can use soemthing like TLV (or 
refer the vhost-user protocol).

Btw, is colo-compare the only user that need to be co-operated with Xen? 
If not, better generalize the API.

Thanks

Re: [Qemu-devel] [PATCH 0/3] COLO-compare: Make COLO-compare support Xen

[Zhang Chen]

On 03/14/2017 03:37 PM, Jason Wang wrote:
>
>
> On 2017年03月13日 15:21, Zhang Chen wrote:
>>
>>
>> On 03/13/2017 03:10 PM, Zhang Chen wrote:
>>>
>>>
>>> On 03/13/2017 02:28 PM, Jason Wang wrote:
>>>>
>>>>
>>>> On 2017年03月13日 14:18, Zhang Chen wrote:
>>>>> Hi~~~ All~
>>>>>
>>>>> No news for a long time, anyone can give me some comments?
>>>>
>>>> Hi,
>>>>
>>>> A question is why use two kinds of colo-frames? This seems not good 
>>>> as lots of the code were duplicated.
>>>
>>> Because Xen colo-frame based on Xen Remus.
>>> Remus do some job like the migration job in qemu/kvm, and in Xen HVM 
>>> we can not do migration in qemu,
>>> So colo-frame code must running in Xen side.
>>> In addition, Xen HVM use qemu to simulation device that Colo-proxy 
>>> and Replication can use same one codes in qemu.
>>
>> Detail:
>> https://wiki.xenproject.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>>
>> Thanks
>> Zhang Chen 
>
> Ok, the idea looks ok but let's don't make the code specific to any 
> hypersior (e.g Xen). Instead, let's define a more generic API like:
>
> 1) rename the series to something like "support remote checkpoint", 
> and mention Xen is the first user
> 2) do not use something like "COLO_USERSPACE_PROXY_INIT", you can just 
> use a more generic name like "USERSPACE_PROXY_INIT"
> 3) instead of using raw strings, you can use soemthing like TLV (or 
> refer the vhost-user protocol).
>
> Btw, is colo-compare the only user that need to be co-operated with 
> Xen? If not, better generalize the API.

Yes, only colo-compare.
I got your point, will fix this series in next version.

Thanks
Zhang Chen

>
> Thanks
>
>
>
> .
>

-- 
Thanks
Zhang Chen