* [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy
@ 2017-02-15 9:54 Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
` (7 more replies)
0 siblings, 8 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
Because of some reason, We no longer support COLO kernel proxy.
So we send this patch set to make Xen use userspace colo-proxy in qemu.
Below is a COLO userspace proxy ascii figure:
Primary qemu Secondary qemu
+--------------------------------------------------------------+ +----------------------------------------------------------------+
| +----------------------------------------------------------+ | | +-----------------------------------------------------------+ |
| | | | | | | |
| | guest | | | | guest | |
| | | | | | | |
| +-------^--------------------------+-----------------------+ | | +---------------------+--------+----------------------------+ |
| | | | | ^ | |
| | | | | | | |
| | +------------------------------------------------------+ | | | |
|netfilter| | | | | | netfilter | | |
| +----------+ +----------------------------+ | | | +-----------------------------------------------------------+ |
| | | | | | out | | | | | | filter excute order | |
| | | | +-----------------------------+ | | | | | | +-------------------> | |
| | | | | | | | | | | | | | TCP | |
| | +-----+--+-+ +-----v----+ +-----v----+ |pri +----+----+sec| | | | +------------+ +---+----+---v+rewriter++ +------------+ | |
| | | | | | | | |in | |in | | | | | | | | | | | | |
| | | filter | | filter | | filter +------> colo <------+ +--------> filter +--> adjust | adjust +--> filter | | |
| | | mirror | |redirector| |redirector| | | compare | | | | | | redirector | | ack | seq | | redirector | | |
| | | | | | | | | | | | | | | | | | | | | | | |
| | +----^-----+ +----+-----+ +----------+ | +---------+ | | | | +------------+ +--------+--------------+ +---+--------+ | |
| | | tx | rx rx | | | | | tx all | rx | |
| | | | | | | | +-----------------------------------------------------------+ |
| | | +--------------+ | | | | | |
| | | filter excute order | | | | | | |
| | | +----------------> | | | +--------------------------------------------------------+ |
| +-----------------------------------------+ | | |
| | | | | |
+--------------------------------------------------------------+ +----------------------------------------------------------------+
|guest receive | guest send
| |
+--------+----------------------------v------------------------+
| | NOTE: filter direction is rx/tx/all
| tap | rx:receive packets sent to the netdev
| | tx:receive packets sent by the netdev
+--------------------------------------------------------------+
You can know the detail from here:
http://wiki.qemu.org/Features/COLO
https://github.com/qemu/qemu/blob/master/docs/colo-proxy.txt
V2:
- Address wei's comments, use macro to reuse codes.
- Do some work on the last patch.
- Fix some typo.
- Add LIBXL_HAVE_COLO_USERSPACE_PROXY macro in libxl.h.
- Some bug fix.
V1:
- Initial patch
Zhang Chen (7):
COLO-Proxy: Add remus command to open userspace proxy
COLO-Proxy: Setup userspace colo-proxy on primary side
tools/libxl: refactor do_domain_create()
COLO-Proxy: Setup userspace colo-proxy on secondary side
COLO-Proxy: Add primary userspace colo proxy start args
COLO-Proxy: Add secondary userspace colo-proxy start args
COLO-Proxy: Use socket to get checkpoint event.
docs/man/xl.pod.1.in | 5 ++
tools/libxl/libxl.h | 6 ++
tools/libxl/libxl_colo.h | 7 +++
tools/libxl/libxl_colo_proxy.c | 99 +++++++++++++++++++++++++++++
tools/libxl/libxl_colo_restore.c | 19 ++++--
tools/libxl/libxl_colo_save.c | 19 +++++-
tools/libxl/libxl_create.c | 21 ++++---
tools/libxl/libxl_dm.c | 132 +++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_nic.c | 109 ++++++++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 64 ++++++++++++++++---
tools/libxl/xl_cmdimpl.c | 120 +++++++++++++++++++++++++++++++++--
tools/libxl/xl_cmdtable.c | 3 +-
12 files changed, 575 insertions(+), 29 deletions(-)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH RFC V2 1/7] COLO-Proxy: Add remus command to open userspace proxy
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
@ 2017-02-15 9:54 ` Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side Zhang Chen
` (6 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
Add remus '-p' to enable userspace colo proxy(in qemu).
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
docs/man/xl.pod.1.in | 5 +++++
tools/libxl/libxl.h | 6 ++++++
tools/libxl/libxl_colo.h | 5 +++++
tools/libxl/libxl_colo_save.c | 2 ++
tools/libxl/libxl_types.idl | 17 +++++++++--------
tools/libxl/xl_cmdimpl.c | 13 ++++++++++++-
tools/libxl/xl_cmdtable.c | 3 ++-
7 files changed, 41 insertions(+), 10 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 09c1faa..4260777 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -553,6 +553,11 @@ Disable disk replication. Requires enabling unsafe mode.
Enable COLO HA. This conflicts with B<-i> and B<-b>, and memory
checkpoint compression must be disabled.
+=item B<-p>
+
+Use userspace COLO Proxy. This option must be used in conjunction
+with B<-c>.
+
=back
=item B<pause> I<domain-id>
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 3924464..fce7fab 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -870,6 +870,12 @@ typedef struct libxl__ctx libxl_ctx;
*/
#define LIBXL_HAVE_REMUS 1
+/*
+ * LIBXL_HAVE_COLO_USERSPACE_PROXY
+ * If this is defined, then libxl supports COLO userspace proxy.
+ */
+#define LIBXL_HAVE_COLO_USERSPACE_PROXY 1
+
typedef uint8_t libxl_mac[6];
#define LIBXL_MAC_FMT "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx"
#define LIBXL_MAC_FMTLEN ((2*6)+5) /* 6 hex bytes plus 5 colons */
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 682275c..4746d8c 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -64,6 +64,11 @@ struct libxl__colo_proxy_state {
int sock_fd;
int index;
+ /*
+ * Private, True means use userspace colo proxy
+ * False means use kernel colo proxy.
+ */
+ bool is_userspace_proxy;
};
struct libxl__colo_save_state {
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 6277770..eb8336c 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -101,6 +101,8 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->qdisk_setuped = false;
css->qdisk_used = false;
libxl__ev_child_init(&css->child);
+ css->cps.is_userspace_proxy =
+ libxl_defbool_val(dss->remus->userspace_colo_proxy);
if (dss->remus->netbufscript)
css->colo_proxy_script = libxl__strdup(gc, dss->remus->netbufscript);
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a612d1f..1bd2057 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -844,14 +844,15 @@ libxl_sched_credit2_params = Struct("sched_credit2_params", [
], dispose_fn=None)
libxl_domain_remus_info = Struct("domain_remus_info",[
- ("interval", integer),
- ("allow_unsafe", libxl_defbool),
- ("blackhole", libxl_defbool),
- ("compression", libxl_defbool),
- ("netbuf", libxl_defbool),
- ("netbufscript", string),
- ("diskbuf", libxl_defbool),
- ("colo", libxl_defbool)
+ ("interval", integer),
+ ("allow_unsafe", libxl_defbool),
+ ("blackhole", libxl_defbool),
+ ("compression", libxl_defbool),
+ ("netbuf", libxl_defbool),
+ ("netbufscript", string),
+ ("diskbuf", libxl_defbool),
+ ("colo", libxl_defbool),
+ ("userspace_colo_proxy", libxl_defbool)
])
libxl_event_type = Enumeration("event_type", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 7e8a8ae..99baeef 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -8893,7 +8893,7 @@ int main_remus(int argc, char **argv)
memset(&r_info, 0, sizeof(libxl_domain_remus_info));
- SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ec", NULL, "remus", 2) {
+ SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ecp", NULL, "remus", 2) {
case 'i':
r_info.interval = atoi(optarg);
break;
@@ -8923,6 +8923,9 @@ int main_remus(int argc, char **argv)
break;
case 'c':
libxl_defbool_set(&r_info.colo, true);
+ break;
+ case 'p':
+ libxl_defbool_set(&r_info.userspace_colo_proxy, true);
}
domid = find_domain(argv[optind]);
@@ -8931,9 +8934,17 @@ int main_remus(int argc, char **argv)
/* Defaults */
libxl_defbool_setdefault(&r_info.blackhole, false);
libxl_defbool_setdefault(&r_info.colo, false);
+ libxl_defbool_setdefault(&r_info.userspace_colo_proxy, false);
+
if (!libxl_defbool_val(r_info.colo) && !r_info.interval)
r_info.interval = 200;
+ if (libxl_defbool_val(r_info.userspace_colo_proxy) &&
+ !libxl_defbool_val(r_info.colo)) {
+ fprintf(stderr, "Option -p must be used in conjunction with -c");
+ exit(-1);
+ }
+
if (libxl_defbool_val(r_info.colo)) {
if (r_info.interval || libxl_defbool_val(r_info.blackhole) ||
!libxl_defbool_is_default(r_info.netbuf) ||
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 588d5d9..409eceb 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -506,7 +506,8 @@ struct cmd_spec cmd_table[] = {
"-n Disable network output buffering. Works only in unsafe mode.\n"
"-d Disable disk replication. Works only in unsafe mode.\n"
"-c Enable COLO HA. It is conflict with -i and -b, and memory\n"
- " checkpoint must be disabled"
+ " checkpoint must be disabled.\n"
+ "-p Use COLO userspace proxy."
},
#endif
{ "devd",
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC V2 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
@ 2017-02-15 9:54 ` Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 3/7] tools/libxl: refactor do_domain_create() Zhang Chen
` (5 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
In this patch we close kernel COLO-Proxy on primary side.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
tools/libxl/libxl_colo_save.c | 9 +++++++--
2 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
index 0983f42..dd902fc 100644
--- a/tools/libxl/libxl_colo_proxy.c
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
STATE_AO_GC(cps->ao);
+ /* If enable userspace proxy mode, we don't need setup kernel proxy */
+ if (cps->is_userspace_proxy)
+ return 0;
+
skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
if (skfd < 0) {
LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
@@ -222,6 +226,13 @@ out:
void colo_proxy_teardown(libxl__colo_proxy_state *cps)
{
+ /*
+ * If enable userspace proxy mode,
+ * we don't need teardown kernel proxy
+ */
+ if (cps->is_userspace_proxy)
+ return;
+
if (cps->sock_fd >= 0) {
close(cps->sock_fd);
cps->sock_fd = -1;
@@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
void colo_proxy_preresume(libxl__colo_proxy_state *cps)
{
+ /*
+ * If enable userspace proxy mode,
+ * we don't need preresume kernel proxy
+ */
+ if (cps->is_userspace_proxy)
+ return;
+
colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
/* TODO: need to handle if the call fails... */
}
@@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
STATE_AO_GC(cps->ao);
+ /*
+ * enable userspace proxy mode, tmp sleep.
+ * then we will add qemu API support this func.
+ */
+ if (cps->is_userspace_proxy) {
+ sleep(timeout_us / 1000000);
+ return 0;
+ }
+
size = colo_proxy_recv(cps, &buff, timeout_us);
/* timeout, return no checkpoint message. */
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index eb8336c..91e3fce 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -110,8 +110,13 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
libxl__xen_script_dir_path());
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
- (1 << LIBXL__DEVICE_KIND_VBD);
+ /* If enable userspace proxy mode, we don't need VIF */
+ if (css->cps.is_userspace_proxy)
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ else
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
+
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC V2 3/7] tools/libxl: refactor do_domain_create()
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side Zhang Chen
@ 2017-02-15 9:54 ` Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side Zhang Chen
` (4 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
We use params->colo_proxy_script to make do_domain_create()
doesn't take "colo_proxy_script" anymore.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_create.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index e3bc257..e741b9a 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1608,7 +1608,6 @@ static void domain_create_cb(libxl__egc *egc,
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd, int send_back_fd,
const libxl_domain_restore_params *params,
- const char *colo_proxy_script,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
@@ -1632,7 +1631,14 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
}
cdcs->dcs.callback = domain_create_cb;
cdcs->dcs.domid_soft_reset = INVALID_DOMID;
- cdcs->dcs.colo_proxy_script = colo_proxy_script;
+
+ if (cdcs->dcs.restore_params.checkpointed_stream ==
+ LIBXL_CHECKPOINTED_STREAM_COLO)
+ cdcs->dcs.colo_proxy_script =
+ cdcs->dcs.restore_params.colo_proxy_script;
+ else
+ cdcs->dcs.colo_proxy_script = NULL;
+
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
@@ -1820,7 +1826,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncprogress_how *aop_console_how)
{
unset_disk_colo_restore(d_config);
- return do_domain_create(ctx, d_config, domid, -1, -1, NULL, NULL,
+ return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
ao_how, aop_console_how);
}
@@ -1831,17 +1837,14 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- char *colo_proxy_script = NULL;
-
if (params->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
- colo_proxy_script = params->colo_proxy_script;
set_disk_colo_restore(d_config);
} else {
unset_disk_colo_restore(d_config);
}
return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
- params, colo_proxy_script, ao_how, aop_console_how);
+ params, ao_how, aop_console_how);
}
int libxl_domain_soft_reset(libxl_ctx *ctx,
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC V2 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (2 preceding siblings ...)
2017-02-15 9:54 ` [PATCH RFC V2 3/7] tools/libxl: refactor do_domain_create() Zhang Chen
@ 2017-02-15 9:54 ` Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 5/7] COLO-Proxy: Add primary userspace colo proxy start args Zhang Chen
` (3 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
In this patch we add a function to close COLO kernel Proxy on secondary side.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo_restore.c | 8 ++++++--
tools/libxl/libxl_create.c | 8 ++++++--
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 18 +++++++++++++++---
4 files changed, 28 insertions(+), 7 deletions(-)
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index 6a96328..c6d239a 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -774,8 +774,12 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
STATE_AO_GC(crs->ao);
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
- (1 << LIBXL__DEVICE_KIND_VBD);
+ if (crs->cps.is_userspace_proxy)
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ else
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
+
cds->callback = colo_restore_setup_cds_done;
cds->ao = ao;
cds->domid = crs->domid;
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index e741b9a..409945a 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1633,11 +1633,15 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
cdcs->dcs.domid_soft_reset = INVALID_DOMID;
if (cdcs->dcs.restore_params.checkpointed_stream ==
- LIBXL_CHECKPOINTED_STREAM_COLO)
+ LIBXL_CHECKPOINTED_STREAM_COLO) {
cdcs->dcs.colo_proxy_script =
cdcs->dcs.restore_params.colo_proxy_script;
- else
+ cdcs->dcs.crs.cps.is_userspace_proxy =
+ libxl_defbool_val(cdcs->dcs.restore_params.userspace_colo_proxy);
+ } else {
cdcs->dcs.colo_proxy_script = NULL;
+ cdcs->dcs.crs.cps.is_userspace_proxy = false;
+ }
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 1bd2057..89c2c9d 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -390,6 +390,7 @@ libxl_domain_restore_params = Struct("domain_restore_params", [
("checkpointed_stream", integer),
("stream_version", uint32, {'init_val': '1'}),
("colo_proxy_script", string),
+ ("userspace_colo_proxy", libxl_defbool),
])
libxl_sched_params = Struct("sched_params",[
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 99baeef..b286d47 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -162,6 +162,7 @@ struct domain_create {
char *extra_config; /* extra config string */
const char *restore_file;
char *colo_proxy_script;
+ bool userspace_colo_proxy;
int migrate_fd; /* -1 means none */
int send_back_fd; /* -1 means none */
char **migration_domname_r; /* from malloc */
@@ -3024,6 +3025,8 @@ start:
params.stream_version =
(hdr.mandatory_flags & XL_MANDATORY_FLAG_STREAMv2) ? 2 : 1;
params.colo_proxy_script = dom_info->colo_proxy_script;
+ libxl_defbool_set(¶ms.userspace_colo_proxy,
+ dom_info->userspace_colo_proxy);
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
@@ -4824,7 +4827,8 @@ static void migrate_receive(int debug, int daemonize, int monitor,
int pause_after_migration,
int send_fd, int recv_fd,
libxl_checkpointed_stream checkpointed,
- char *colo_proxy_script)
+ char *colo_proxy_script,
+ bool userspace_colo_proxy)
{
uint32_t domid;
int rc, rc2;
@@ -4852,6 +4856,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
dom_info.migration_domname_r = &migration_domname;
dom_info.checkpointed_stream = checkpointed;
dom_info.colo_proxy_script = colo_proxy_script;
+ dom_info.userspace_colo_proxy = userspace_colo_proxy;
rc = create_domain(&dom_info);
if (rc < 0) {
@@ -5051,11 +5056,13 @@ int main_migrate_receive(int argc, char **argv)
int debug = 0, daemonize = 1, monitor = 1, pause_after_migration = 0;
libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
int opt;
+ bool userspace_colo_proxy = false;
char *script = NULL;
static struct option opts[] = {
{"colo", 0, 0, 0x100},
/* It is a shame that the management code for disk is not here. */
{"coloft-script", 1, 0, 0x200},
+ {"userspace-colo-proxy", 0, 0, 0x300},
COMMON_LONG_OPTS
};
@@ -5079,6 +5086,9 @@ int main_migrate_receive(int argc, char **argv)
case 0x200:
script = optarg;
break;
+ case 0x300:
+ userspace_colo_proxy = true;
+ break;
case 'p':
pause_after_migration = 1;
break;
@@ -5090,7 +5100,7 @@ int main_migrate_receive(int argc, char **argv)
}
migrate_receive(debug, daemonize, monitor, pause_after_migration,
STDOUT_FILENO, STDIN_FILENO,
- checkpointed, script);
+ checkpointed, script, userspace_colo_proxy);
return EXIT_SUCCESS;
}
@@ -8984,11 +8994,13 @@ int main_remus(int argc, char **argv)
"-r",
daemonize ? "" : " -e");
} else {
- xasprintf(&rune, "exec %s %s xl migrate-receive %s %s %s %s",
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s %s %s %s",
ssh_command, host,
"--colo",
r_info.netbufscript ? "--coloft-script" : "",
r_info.netbufscript ? r_info.netbufscript : "",
+ libxl_defbool_val(r_info.userspace_colo_proxy) ?
+ "--userspace-colo-proxy" : "",
daemonize ? "" : " -e");
}
}
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC V2 5/7] COLO-Proxy: Add primary userspace colo proxy start args
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (3 preceding siblings ...)
2017-02-15 9:54 ` [PATCH RFC V2 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side Zhang Chen
@ 2017-02-15 9:54 ` Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 6/7] COLO-Proxy: Add secondary userspace colo-proxy " Zhang Chen
` (2 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
Qemu need this args to start userspace colo-proxy.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_dm.c | 98 +++++++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_nic.c | 78 ++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 31 +++++++++++++-
tools/libxl/xl_cmdimpl.c | 58 +++++++++++++++++++++++++++
4 files changed, 264 insertions(+), 1 deletion(-)
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 281058d..abd4edd 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -1244,7 +1244,105 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
nics[i].devid, ifname,
libxl_tapif_script(gc),
libxl_tapif_script(gc)));
+
+ /* Userspace COLO Proxy need this */
+#define APPEND_COLO_SOCK_SERVER(sock_id, sock_ip, sock_port) ({ \
+ if (nics[i].colo_##sock_id && \
+ nics[i].colo_##sock_ip && \
+ nics[i].colo_##sock_port) { \
+ flexarray_append(dm_args, "-chardev"); \
+ flexarray_append(dm_args, \
+ GCSPRINTF("socket,id=%s,host=%s,port=%s,server,nowait", \
+ nics[i].colo_##sock_id, \
+ nics[i].colo_##sock_ip, \
+ nics[i].colo_##sock_port)); \
+ } \
+})
+
+#define APPEND_COLO_SOCK_CLIENT(sock_id, sock_ip, sock_port) ({ \
+ if (nics[i].colo_##sock_id && \
+ nics[i].colo_##sock_ip && \
+ nics[i].colo_##sock_port) { \
+ flexarray_append(dm_args, "-chardev"); \
+ flexarray_append(dm_args, \
+ GCSPRINTF("socket,id=%s,host=%s,port=%s", \
+ nics[i].colo_##sock_id, \
+ nics[i].colo_##sock_ip, \
+ nics[i].colo_##sock_port)); \
+ } \
+})
+
+ if (state->saved_state) {
+ /* secondary colo run */
+ } else {
+ /* primary colo run */
+
+ APPEND_COLO_SOCK_SERVER(sock_mirror_id,
+ sock_mirror_ip,
+ sock_mirror_port);
+
+ APPEND_COLO_SOCK_SERVER(sock_compare_pri_in_id,
+ sock_compare_pri_in_ip,
+ sock_compare_pri_in_port);
+
+ APPEND_COLO_SOCK_SERVER(sock_compare_sec_in_id,
+ sock_compare_sec_in_ip,
+ sock_compare_sec_in_port);
+
+ APPEND_COLO_SOCK_SERVER(sock_redirector0_id,
+ sock_redirector0_ip,
+ sock_redirector0_port);
+
+ APPEND_COLO_SOCK_CLIENT(sock_redirector1_id,
+ sock_redirector1_ip,
+ sock_redirector1_port);
+
+ APPEND_COLO_SOCK_CLIENT(sock_redirector2_id,
+ sock_redirector2_ip,
+ sock_redirector2_port);
+
+ if (nics[i].colo_filter_mirror_queue &&
+ nics[i].colo_filter_mirror_outdev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-mirror,id=m1,netdev=net%d,queue=%s,outdev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_mirror_queue,
+ nics[i].colo_filter_mirror_outdev));
+ }
+ if (nics[i].colo_filter_redirector0_queue &&
+ nics[i].colo_filter_redirector0_indev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=r1,netdev=net%d,queue=%s,indev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_redirector0_queue,
+ nics[i].colo_filter_redirector0_indev));
+ }
+ if (nics[i].colo_filter_redirector1_queue &&
+ nics[i].colo_filter_redirector1_outdev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=r2,netdev=net%d,queue=%s,outdev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_redirector1_queue,
+ nics[i].colo_filter_redirector1_outdev));
+ }
+ if (nics[i].colo_compare_pri_in &&
+ nics[i].colo_compare_sec_in &&
+ nics[i].colo_compare_out) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("colo-compare,id=c1,primary_in=%s,secondary_in=%s,outdev=%s",
+ nics[i].colo_compare_pri_in,
+ nics[i].colo_compare_sec_in,
+ nics[i].colo_compare_out));
+ }
+ }
ioemu_nics++;
+
+#undef APPEND_COLO_SOCK_SERVER
+#undef APPEND_COLO_SOCK_CLIENT
}
}
/* If we have no emulated nics, tell qemu not to create any */
diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
index 61b55ca..7c57bcf 100644
--- a/tools/libxl/libxl_nic.c
+++ b/tools/libxl/libxl_nic.c
@@ -196,6 +196,45 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
flexarray_append(back, nic->coloft_forwarddev);
}
+#define MAYBE_ADD_COLO_ARGS(arg) ({ \
+ if (nic->colo_##arg) { \
+ flexarray_append(back, "colo_"#arg); \
+ flexarray_append(back, nic->colo_##arg); \
+ } \
+})
+
+ MAYBE_ADD_COLO_ARGS(sock_mirror_id);
+ MAYBE_ADD_COLO_ARGS(sock_mirror_ip);
+ MAYBE_ADD_COLO_ARGS(sock_mirror_port);
+ MAYBE_ADD_COLO_ARGS(sock_compare_pri_in_id);
+ MAYBE_ADD_COLO_ARGS(sock_compare_pri_in_ip);
+ MAYBE_ADD_COLO_ARGS(sock_compare_pri_in_port);
+ MAYBE_ADD_COLO_ARGS(sock_compare_sec_in_id);
+ MAYBE_ADD_COLO_ARGS(sock_compare_sec_in_ip);
+ MAYBE_ADD_COLO_ARGS(sock_compare_sec_in_port);
+ MAYBE_ADD_COLO_ARGS(sock_redirector0_id);
+ MAYBE_ADD_COLO_ARGS(sock_redirector0_ip);
+ MAYBE_ADD_COLO_ARGS(sock_redirector0_port);
+ MAYBE_ADD_COLO_ARGS(sock_redirector1_id);
+ MAYBE_ADD_COLO_ARGS(sock_redirector1_ip);
+ MAYBE_ADD_COLO_ARGS(sock_redirector1_port);
+ MAYBE_ADD_COLO_ARGS(sock_redirector2_id);
+ MAYBE_ADD_COLO_ARGS(sock_redirector2_ip);
+ MAYBE_ADD_COLO_ARGS(sock_redirector2_port);
+ MAYBE_ADD_COLO_ARGS(filter_mirror_queue);
+ MAYBE_ADD_COLO_ARGS(filter_mirror_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector0_queue);
+ MAYBE_ADD_COLO_ARGS(filter_redirector0_indev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector0_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector1_queue);
+ MAYBE_ADD_COLO_ARGS(filter_redirector1_indev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector1_outdev);
+ MAYBE_ADD_COLO_ARGS(compare_pri_in);
+ MAYBE_ADD_COLO_ARGS(compare_sec_in);
+ MAYBE_ADD_COLO_ARGS(compare_out);
+
+#undef MAYBE_ADD_COLO_ARGS
+
flexarray_append(back, "mac");
flexarray_append(back,GCSPRINTF(LIBXL_MAC_FMT, LIBXL_MAC_BYTES(nic->mac)));
if (nic->ip) {
@@ -349,6 +388,45 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
(const char **)(&nic->coloft_forwarddev));
if (rc) goto out;
+#define CHECK_COLO_ARGS(arg) ({ \
+ rc = libxl__xs_read_checked(NOGC, XBT_NULL, \
+ GCSPRINTF("%s/colo_"#arg, libxl_path), \
+ (const char **)(&nic->colo_##arg)); \
+ if (rc) goto out; \
+})
+
+ CHECK_COLO_ARGS(sock_mirror_id);
+ CHECK_COLO_ARGS(sock_mirror_ip);
+ CHECK_COLO_ARGS(sock_mirror_port);
+ CHECK_COLO_ARGS(sock_compare_pri_in_id);
+ CHECK_COLO_ARGS(sock_compare_pri_in_ip);
+ CHECK_COLO_ARGS(sock_compare_pri_in_port);
+ CHECK_COLO_ARGS(sock_compare_sec_in_id);
+ CHECK_COLO_ARGS(sock_compare_sec_in_ip);
+ CHECK_COLO_ARGS(sock_compare_sec_in_port);
+ CHECK_COLO_ARGS(sock_redirector0_id);
+ CHECK_COLO_ARGS(sock_redirector0_ip);
+ CHECK_COLO_ARGS(sock_redirector0_port);
+ CHECK_COLO_ARGS(sock_redirector1_id);
+ CHECK_COLO_ARGS(sock_redirector1_ip);
+ CHECK_COLO_ARGS(sock_redirector1_port);
+ CHECK_COLO_ARGS(sock_redirector2_id);
+ CHECK_COLO_ARGS(sock_redirector2_ip);
+ CHECK_COLO_ARGS(sock_redirector2_port);
+ CHECK_COLO_ARGS(filter_mirror_queue);
+ CHECK_COLO_ARGS(filter_mirror_outdev);
+ CHECK_COLO_ARGS(filter_redirector0_queue);
+ CHECK_COLO_ARGS(filter_redirector0_indev);
+ CHECK_COLO_ARGS(filter_redirector0_outdev);
+ CHECK_COLO_ARGS(filter_redirector1_queue);
+ CHECK_COLO_ARGS(filter_redirector1_indev);
+ CHECK_COLO_ARGS(filter_redirector1_outdev);
+ CHECK_COLO_ARGS(compare_pri_in);
+ CHECK_COLO_ARGS(compare_sec_in);
+ CHECK_COLO_ARGS(compare_out);
+
+#undef CHECK_COLO_ARGS
+
/* vif_ioemu nics use the same xenstore entries as vif interfaces */
rc = libxl__xs_read_checked(gc, XBT_NULL,
GCSPRINTF("%s/type", libxl_path), &tmp);
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 89c2c9d..07ce345 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -629,7 +629,36 @@ libxl_device_nic = Struct("device_nic", [
("gatewaydev", string),
# Note that the COLO configuration settings should be considered unstable.
# They may change incompatibly in future versions of Xen.
- ("coloft_forwarddev", string)
+ ("coloft_forwarddev", string),
+ ("colo_sock_mirror_id", string),
+ ("colo_sock_mirror_ip", string),
+ ("colo_sock_mirror_port", string),
+ ("colo_sock_compare_pri_in_id", string),
+ ("colo_sock_compare_pri_in_ip", string),
+ ("colo_sock_compare_pri_in_port", string),
+ ("colo_sock_compare_sec_in_id", string),
+ ("colo_sock_compare_sec_in_ip", string),
+ ("colo_sock_compare_sec_in_port", string),
+ ("colo_sock_redirector0_id", string),
+ ("colo_sock_redirector0_ip", string),
+ ("colo_sock_redirector0_port", string),
+ ("colo_sock_redirector1_id", string),
+ ("colo_sock_redirector1_ip", string),
+ ("colo_sock_redirector1_port", string),
+ ("colo_sock_redirector2_id", string),
+ ("colo_sock_redirector2_ip", string),
+ ("colo_sock_redirector2_port", string),
+ ("colo_filter_mirror_queue", string),
+ ("colo_filter_mirror_outdev", string),
+ ("colo_filter_redirector0_queue", string),
+ ("colo_filter_redirector0_indev", string),
+ ("colo_filter_redirector0_outdev", string),
+ ("colo_filter_redirector1_queue", string),
+ ("colo_filter_redirector1_indev", string),
+ ("colo_filter_redirector1_outdev", string),
+ ("colo_compare_pri_in", string),
+ ("colo_compare_sec_in", string),
+ ("colo_compare_out", string)
])
libxl_device_pci = Struct("device_pci", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index b286d47..e6cb015 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1060,6 +1060,64 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
parse_vif_rate(config, oparg, nic);
} else if (MATCH_OPTION("forwarddev", token, oparg)) {
replace_string(&nic->coloft_forwarddev, oparg);
+ } else if (MATCH_OPTION("colo_sock_mirror_id", token, oparg)) {
+ replace_string(&nic->colo_sock_mirror_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_mirror_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_mirror_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_mirror_port", token, oparg)) {
+ replace_string(&nic->colo_sock_mirror_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_sec_in_id", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_sec_in_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_sec_in_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_sec_in_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_sec_in_port", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_sec_in_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector0_id", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector0_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector0_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector0_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector0_port", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector0_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector1_id", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector1_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector1_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector1_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector1_port", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector1_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector2_id", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector2_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector2_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector2_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector2_port", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector2_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_pri_in_id", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_pri_in_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_pri_in_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_pri_in_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_pri_in_port", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_pri_in_port, oparg);
+ } else if (MATCH_OPTION("colo_filter_mirror_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_mirror_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_mirror_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_mirror_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector0_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector0_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector0_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector0_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector0_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector0_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector1_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector1_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector1_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector1_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector1_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector1_outdev, oparg);
+ } else if (MATCH_OPTION("colo_compare_pri_in", token, oparg)) {
+ replace_string(&nic->colo_compare_pri_in, oparg);
+ } else if (MATCH_OPTION("colo_compare_sec_in", token, oparg)) {
+ replace_string(&nic->colo_compare_sec_in, oparg);
+ } else if (MATCH_OPTION("colo_compare_out", token, oparg)) {
+ replace_string(&nic->colo_compare_out, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC V2 6/7] COLO-Proxy: Add secondary userspace colo-proxy start args
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (4 preceding siblings ...)
2017-02-15 9:54 ` [PATCH RFC V2 5/7] COLO-Proxy: Add primary userspace colo proxy start args Zhang Chen
@ 2017-02-15 9:54 ` Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
2017-02-15 16:18 ` [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Konrad Rzeszutek Wilk
7 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
Qemu need this args to start userspace colo-proxy.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_dm.c | 34 ++++++++++++++++++++++++++++++++++
tools/libxl/libxl_nic.c | 27 +++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 15 ++++++++++++++-
tools/libxl/xl_cmdimpl.c | 27 +++++++++++++++++++++++++++
4 files changed, 102 insertions(+), 1 deletion(-)
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index abd4edd..0fabd64 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -1274,6 +1274,40 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
if (state->saved_state) {
/* secondary colo run */
+
+ APPEND_COLO_SOCK_CLIENT(sock_sec_redirector0_id,
+ sock_sec_redirector0_ip,
+ sock_sec_redirector0_port);
+
+ APPEND_COLO_SOCK_CLIENT(sock_sec_redirector1_id,
+ sock_sec_redirector1_ip,
+ sock_sec_redirector1_port);
+
+ if (nics[i].colo_filter_sec_redirector0_queue &&
+ nics[i].colo_filter_sec_redirector0_indev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=rs1,netdev=net%d,queue=%s,indev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_sec_redirector0_queue,
+ nics[i].colo_filter_sec_redirector0_indev));
+ }
+ if (nics[i].colo_filter_sec_redirector1_queue &&
+ nics[i].colo_filter_sec_redirector1_indev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=rs2,netdev=net%d,queue=%s,outdev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_sec_redirector1_queue,
+ nics[i].colo_filter_sec_redirector1_outdev));
+ }
+ if (nics[i].colo_filter_sec_rewriter0_queue) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-rewriter,id=rs3,netdev=net%d,queue=%s",
+ nics[i].devid,
+ nics[i].colo_filter_sec_rewriter0_queue));
+ }
} else {
/* primary colo run */
diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
index 7c57bcf..5e1fecd 100644
--- a/tools/libxl/libxl_nic.c
+++ b/tools/libxl/libxl_nic.c
@@ -233,6 +233,20 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
MAYBE_ADD_COLO_ARGS(compare_sec_in);
MAYBE_ADD_COLO_ARGS(compare_out);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector0_id);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector0_ip);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector0_port);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector1_id);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector1_ip);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector1_port);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector0_queue);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector0_indev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector0_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_queue);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_indev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_rewriter0_queue);
+
#undef MAYBE_ADD_COLO_ARGS
flexarray_append(back, "mac");
@@ -424,6 +438,19 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
CHECK_COLO_ARGS(compare_pri_in);
CHECK_COLO_ARGS(compare_sec_in);
CHECK_COLO_ARGS(compare_out);
+ CHECK_COLO_ARGS(sock_sec_redirector0_id);
+ CHECK_COLO_ARGS(sock_sec_redirector0_ip);
+ CHECK_COLO_ARGS(sock_sec_redirector0_port);
+ CHECK_COLO_ARGS(sock_sec_redirector1_id);
+ CHECK_COLO_ARGS(sock_sec_redirector1_ip);
+ CHECK_COLO_ARGS(sock_sec_redirector1_port);
+ CHECK_COLO_ARGS(filter_sec_redirector0_queue);
+ CHECK_COLO_ARGS(filter_sec_redirector0_indev);
+ CHECK_COLO_ARGS(filter_sec_redirector0_outdev);
+ CHECK_COLO_ARGS(filter_sec_redirector1_queue);
+ CHECK_COLO_ARGS(filter_sec_redirector1_indev);
+ CHECK_COLO_ARGS(filter_sec_redirector1_outdev);
+ CHECK_COLO_ARGS(filter_sec_rewriter0_queue);
#undef CHECK_COLO_ARGS
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 07ce345..47e96b1 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -658,7 +658,20 @@ libxl_device_nic = Struct("device_nic", [
("colo_filter_redirector1_outdev", string),
("colo_compare_pri_in", string),
("colo_compare_sec_in", string),
- ("colo_compare_out", string)
+ ("colo_compare_out", string),
+ ("colo_sock_sec_redirector0_id", string),
+ ("colo_sock_sec_redirector0_ip", string),
+ ("colo_sock_sec_redirector0_port", string),
+ ("colo_sock_sec_redirector1_id", string),
+ ("colo_sock_sec_redirector1_ip", string),
+ ("colo_sock_sec_redirector1_port", string),
+ ("colo_filter_sec_redirector0_queue", string),
+ ("colo_filter_sec_redirector0_indev", string),
+ ("colo_filter_sec_redirector0_outdev", string),
+ ("colo_filter_sec_redirector1_queue", string),
+ ("colo_filter_sec_redirector1_indev", string),
+ ("colo_filter_sec_redirector1_outdev", string),
+ ("colo_filter_sec_rewriter0_queue", string)
])
libxl_device_pci = Struct("device_pci", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index e6cb015..32a47f6 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1118,6 +1118,33 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
replace_string(&nic->colo_compare_sec_in, oparg);
} else if (MATCH_OPTION("colo_compare_out", token, oparg)) {
replace_string(&nic->colo_compare_out, oparg);
+
+ } else if (MATCH_OPTION("colo_sock_sec_redirector0_id", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector0_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector0_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector0_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector0_port", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector0_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector1_id", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector1_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector1_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector1_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector1_port", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector1_port, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector0_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector0_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector0_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector0_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector0_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector0_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector1_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector1_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector1_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector1_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector1_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector1_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_rewriter0_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_rewriter0_queue, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH RFC V2 7/7] COLO-Proxy: Use socket to get checkpoint event.
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (5 preceding siblings ...)
2017-02-15 9:54 ` [PATCH RFC V2 6/7] COLO-Proxy: Add secondary userspace colo-proxy " Zhang Chen
@ 2017-02-15 9:54 ` Zhang Chen
2017-02-15 16:24 ` Konrad Rzeszutek Wilk
2017-02-15 16:18 ` [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Konrad Rzeszutek Wilk
7 siblings, 1 reply; 12+ messages in thread
From: Zhang Chen @ 2017-02-15 9:54 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Yang Hongyang, Bian Naimeng, eddie, Li Zhijian, Zhang Chen
We use kernel colo proxy's way to get the checkpoint event
from qemu colo-compare.
Qemu colo-compare need add a API to support this(I will add this in qemu).
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo.h | 2 +
tools/libxl/libxl_colo_proxy.c | 84 +++++++++++++++++++++++++++++++++++++---
tools/libxl/libxl_colo_restore.c | 11 ++++--
tools/libxl/libxl_colo_save.c | 22 +++++++----
tools/libxl/libxl_nic.c | 4 ++
tools/libxl/libxl_types.idl | 4 +-
tools/libxl/xl_cmdimpl.c | 4 ++
7 files changed, 113 insertions(+), 18 deletions(-)
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 4746d8c..6c01b55 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -69,6 +69,8 @@ struct libxl__colo_proxy_state {
* False means use kernel colo proxy.
*/
bool is_userspace_proxy;
+ const char *checkpoint_host;
+ const char *checkpoint_port;
};
struct libxl__colo_save_state {
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
index dd902fc..9d21cf1 100644
--- a/tools/libxl/libxl_colo_proxy.c
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -18,6 +18,9 @@
#include "libxl_internal.h"
#include <netlink/netlink.h>
+#include <arpa/inet.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
/* Consistent with the new COLO netlink channel in kernel side */
#define NETLINK_COLO 28
@@ -76,6 +79,26 @@ static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
return ret;
}
+static int colo_userspace_proxy_recv(libxl__colo_proxy_state *cps,
+ char *buff,
+ unsigned int timeout_us)
+{
+ struct timeval tv;
+ int ret;
+
+ STATE_AO_GC(cps->ao);
+
+ if (timeout_us) {
+ tv.tv_sec = timeout_us / 1000000;
+ tv.tv_usec = timeout_us % 1000000;
+ setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+
+ ret = recv(cps->sock_fd, buff, sizeof(buff),0);
+
+ return ret;
+}
+
/* error: return -1, otherwise return 0 */
static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
unsigned int timeout_us)
@@ -153,8 +176,45 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
STATE_AO_GC(cps->ao);
/* If enable userspace proxy mode, we don't need setup kernel proxy */
- if (cps->is_userspace_proxy)
+ if (cps->is_userspace_proxy) {
+ struct sockaddr_in addr;
+ int port;
+ char recvbuff[1024];
+
+ memset(&addr, 0, sizeof(addr));
+ port = atoi(cps->checkpoint_port);
+ addr.sin_family = AF_INET;
+ addr.sin_port = htons(port);
+ addr.sin_addr.s_addr = inet_addr(cps->checkpoint_host);
+
+ skfd = socket(AF_INET, SOCK_STREAM, 0);
+ if (skfd < 0) {
+ LOGD(ERROR, ao->domid, "can not create a TCP socket: %s",
+ strerror(errno));
+ goto out;
+ }
+
+ cps->sock_fd = skfd;
+
+ if (connect(skfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
+ LOGD(ERROR, ao->domid, "connect error");
+ goto out;
+ }
+
+ char sendbuf[] = "COLO_USERSPACE_PROXY_INIT";
+ ret = send(skfd, sendbuf, strlen(sendbuf),0);
+ if (ret < 0)
+ goto out;
+
+ ret = colo_userspace_proxy_recv(cps, recvbuff, 500000);
+ if (size < 0) {
+ LOGD(ERROR, ao->domid, "Can't recv msg from qemu colo-compare: %s",
+ strerror(errno));
+ goto out;
+ }
+
return 0;
+ }
skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
if (skfd < 0) {
@@ -247,8 +307,11 @@ void colo_proxy_preresume(libxl__colo_proxy_state *cps)
* If enable userspace proxy mode,
* we don't need preresume kernel proxy
*/
- if (cps->is_userspace_proxy)
+ if (cps->is_userspace_proxy) {
+ char sendbuf[] = "COLO_CHECKPOINT";
+ send(cps->sock_fd, sendbuf, strlen(sendbuf),0);
return;
+ }
colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
/* TODO: need to handle if the call fails... */
@@ -277,16 +340,25 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
struct nlmsghdr *h;
struct colo_msg *m;
int ret = -1;
+ char recvbuff[1024];
STATE_AO_GC(cps->ao);
/*
- * enable userspace proxy mode, tmp sleep.
- * then we will add qemu API support this func.
+ * enable userspace proxy mode.
+ * Then we will add qemu API support for this func.
*/
if (cps->is_userspace_proxy) {
- sleep(timeout_us / 1000000);
- return 0;
+ ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
+ if (ret <= 0)
+ return 0;
+
+ if (!strcmp(recvbuff, "DO_CHECKPOINT")) {
+ return 1;
+ } else {
+ LOGD(ERROR, ao->domid, "receive qemu colo-compare checkpoint error");
+ return -1;
+ }
}
size = colo_proxy_recv(cps, &buff, timeout_us);
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index c6d239a..065ea00 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -613,7 +613,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
}
}
- colo_proxy_preresume(&crs->cps);
+ if (!crs->cps.is_userspace_proxy)
+ colo_proxy_preresume(&crs->cps);
colo_restore_resume_vm(egc, crcs);
@@ -786,9 +787,11 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
cds->ops = colo_restore_ops;
crs->cps.ao = ao;
- if (colo_proxy_setup(&crs->cps)) {
- LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
- goto out;
+ if (!crs->cps.is_userspace_proxy) {
+ if (colo_proxy_setup(&crs->cps)) {
+ LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
+ goto out;
+ }
}
if (init_device_subkind(cds))
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 91e3fce..a0cfc5a 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -86,6 +86,7 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
libxl__checkpoint_devices_state *const cds = &dss->cds;
libxl__srm_save_autogen_callbacks *const callbacks =
&dss->sws.shs.callbacks.save.a;
+ libxl_device_nic *nics;
STATE_AO_GC(dss->ao);
@@ -110,24 +111,31 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
libxl__xen_script_dir_path());
- /* If enable userspace proxy mode, we don't need VIF */
- if (css->cps.is_userspace_proxy)
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
- else
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
- (1 << LIBXL__DEVICE_KIND_VBD);
-
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
cds->domid = dss->domid;
cds->concrete_data = css;
+ /* If enable userspace proxy mode, we don't need VIF */
+ if (css->cps.is_userspace_proxy) {
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+
+ /* Use this args we can connect to qemu colo-compare */
+ nics = libxl_device_nic_list(CTX, cds->domid, &cds->num_nics);
+ css->cps.checkpoint_host = nics->colo_checkpoint_host;
+ css->cps.checkpoint_port = nics->colo_checkpoint_port;
+ } else {
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
+ }
+
css->srs.ao = ao;
css->srs.fd = css->recv_fd;
css->srs.back_channel = true;
libxl__stream_read_start(egc, &css->srs);
css->cps.ao = ao;
+
if (colo_proxy_setup(&css->cps)) {
LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
goto out;
diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
index 5e1fecd..6bc6146 100644
--- a/tools/libxl/libxl_nic.c
+++ b/tools/libxl/libxl_nic.c
@@ -246,6 +246,8 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_indev);
MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_outdev);
MAYBE_ADD_COLO_ARGS(filter_sec_rewriter0_queue);
+ MAYBE_ADD_COLO_ARGS(checkpoint_host);
+ MAYBE_ADD_COLO_ARGS(checkpoint_port);
#undef MAYBE_ADD_COLO_ARGS
@@ -451,6 +453,8 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
CHECK_COLO_ARGS(filter_sec_redirector1_indev);
CHECK_COLO_ARGS(filter_sec_redirector1_outdev);
CHECK_COLO_ARGS(filter_sec_rewriter0_queue);
+ CHECK_COLO_ARGS(checkpoint_host);
+ CHECK_COLO_ARGS(checkpoint_port);
#undef CHECK_COLO_ARGS
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 47e96b1..0b412ee 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -671,7 +671,9 @@ libxl_device_nic = Struct("device_nic", [
("colo_filter_sec_redirector1_queue", string),
("colo_filter_sec_redirector1_indev", string),
("colo_filter_sec_redirector1_outdev", string),
- ("colo_filter_sec_rewriter0_queue", string)
+ ("colo_filter_sec_rewriter0_queue", string),
+ ("colo_checkpoint_host", string),
+ ("colo_checkpoint_port", string)
])
libxl_device_pci = Struct("device_pci", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 32a47f6..ad5e193 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1145,6 +1145,10 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
replace_string(&nic->colo_filter_sec_redirector1_outdev, oparg);
} else if (MATCH_OPTION("colo_filter_sec_rewriter0_queue", token, oparg)) {
replace_string(&nic->colo_filter_sec_rewriter0_queue, oparg);
+ } else if (MATCH_OPTION("colo_checkpoint_host", token, oparg)) {
+ replace_string(&nic->colo_checkpoint_host, oparg);
+ } else if (MATCH_OPTION("colo_checkpoint_port", token, oparg)) {
+ replace_string(&nic->colo_checkpoint_port, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (6 preceding siblings ...)
2017-02-15 9:54 ` [PATCH RFC V2 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
@ 2017-02-15 16:18 ` Konrad Rzeszutek Wilk
2017-02-16 1:36 ` Zhang Chen
7 siblings, 1 reply; 12+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-15 16:18 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang, Xen devel,
Bian Naimeng
On Wed, Feb 15, 2017 at 05:54:26PM +0800, Zhang Chen wrote:
> Because of some reason, We no longer support COLO kernel proxy.
> So we send this patch set to make Xen use userspace colo-proxy in qemu.
How come it is RFC? It looks quite baked?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC V2 7/7] COLO-Proxy: Use socket to get checkpoint event.
2017-02-15 9:54 ` [PATCH RFC V2 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
@ 2017-02-15 16:24 ` Konrad Rzeszutek Wilk
2017-02-16 1:48 ` Zhang Chen
0 siblings, 1 reply; 12+ messages in thread
From: Konrad Rzeszutek Wilk @ 2017-02-15 16:24 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang, Xen devel,
Bian Naimeng
On Wed, Feb 15, 2017 at 05:54:33PM +0800, Zhang Chen wrote:
> We use kernel colo proxy's way to get the checkpoint event
> from qemu colo-compare.
> Qemu colo-compare need add a API to support this(I will add this in qemu).
>
> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> ---
> tools/libxl/libxl_colo.h | 2 +
> tools/libxl/libxl_colo_proxy.c | 84 +++++++++++++++++++++++++++++++++++++---
> tools/libxl/libxl_colo_restore.c | 11 ++++--
> tools/libxl/libxl_colo_save.c | 22 +++++++----
> tools/libxl/libxl_nic.c | 4 ++
> tools/libxl/libxl_types.idl | 4 +-
> tools/libxl/xl_cmdimpl.c | 4 ++
> 7 files changed, 113 insertions(+), 18 deletions(-)
>
> diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
> index 4746d8c..6c01b55 100644
> --- a/tools/libxl/libxl_colo.h
> +++ b/tools/libxl/libxl_colo.h
> @@ -69,6 +69,8 @@ struct libxl__colo_proxy_state {
> * False means use kernel colo proxy.
> */
> bool is_userspace_proxy;
> + const char *checkpoint_host;
> + const char *checkpoint_port;
> };
>
> struct libxl__colo_save_state {
> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
> index dd902fc..9d21cf1 100644
> --- a/tools/libxl/libxl_colo_proxy.c
> +++ b/tools/libxl/libxl_colo_proxy.c
> @@ -18,6 +18,9 @@
> #include "libxl_internal.h"
>
> #include <netlink/netlink.h>
> +#include <arpa/inet.h>
> +#include <sys/socket.h>
> +#include <netinet/in.h>
>
> /* Consistent with the new COLO netlink channel in kernel side */
> #define NETLINK_COLO 28
> @@ -76,6 +79,26 @@ static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
> return ret;
> }
>
> +static int colo_userspace_proxy_recv(libxl__colo_proxy_state *cps,
> + char *buff,
> + unsigned int timeout_us)
> +{
> + struct timeval tv;
> + int ret;
> +
> + STATE_AO_GC(cps->ao);
> +
> + if (timeout_us) {
> + tv.tv_sec = timeout_us / 1000000;
> + tv.tv_usec = timeout_us % 1000000;
> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
> + }
> +
> + ret = recv(cps->sock_fd, buff, sizeof(buff),0);
> +
> + return ret;
> +}
> +
> /* error: return -1, otherwise return 0 */
> static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
> unsigned int timeout_us)
> @@ -153,8 +176,45 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
> STATE_AO_GC(cps->ao);
>
> /* If enable userspace proxy mode, we don't need setup kernel proxy */
> - if (cps->is_userspace_proxy)
> + if (cps->is_userspace_proxy) {
> + struct sockaddr_in addr;
> + int port;
> + char recvbuff[1024];
> +
> + memset(&addr, 0, sizeof(addr));
> + port = atoi(cps->checkpoint_port);
> + addr.sin_family = AF_INET;
> + addr.sin_port = htons(port);
> + addr.sin_addr.s_addr = inet_addr(cps->checkpoint_host);
> +
> + skfd = socket(AF_INET, SOCK_STREAM, 0);
> + if (skfd < 0) {
> + LOGD(ERROR, ao->domid, "can not create a TCP socket: %s",
> + strerror(errno));
> + goto out;
> + }
> +
> + cps->sock_fd = skfd;
> +
> + if (connect(skfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> + LOGD(ERROR, ao->domid, "connect error");
> + goto out;
> + }
> +
> + char sendbuf[] = "COLO_USERSPACE_PROXY_INIT";
Um, could you move it right around 'char recvbuff' ?
> + ret = send(skfd, sendbuf, strlen(sendbuf),0);
> + if (ret < 0)
> + goto out;
> +
> + ret = colo_userspace_proxy_recv(cps, recvbuff, 500000);
That 500000 looks like a good candidate for a #define?
> + if (size < 0) {
> + LOGD(ERROR, ao->domid, "Can't recv msg from qemu colo-compare: %s",
> + strerror(errno));
> + goto out;
> + }
> +
> return 0;
> + }
>
> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
> if (skfd < 0) {
> @@ -247,8 +307,11 @@ void colo_proxy_preresume(libxl__colo_proxy_state *cps)
> * If enable userspace proxy mode,
> * we don't need preresume kernel proxy
> */
> - if (cps->is_userspace_proxy)
> + if (cps->is_userspace_proxy) {
> + char sendbuf[] = "COLO_CHECKPOINT";
> + send(cps->sock_fd, sendbuf, strlen(sendbuf),0);
> return;
> + }
>
> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
> /* TODO: need to handle if the call fails... */
> @@ -277,16 +340,25 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
> struct nlmsghdr *h;
> struct colo_msg *m;
> int ret = -1;
> + char recvbuff[1024];
>
> STATE_AO_GC(cps->ao);
>
> /*
> - * enable userspace proxy mode, tmp sleep.
> - * then we will add qemu API support this func.
> + * enable userspace proxy mode.
> + * Then we will add qemu API support for this func.
> */
> if (cps->is_userspace_proxy) {
> - sleep(timeout_us / 1000000);
> - return 0;
> + ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
> + if (ret <= 0)
> + return 0;
> +
> + if (!strcmp(recvbuff, "DO_CHECKPOINT")) {
> + return 1;
> + } else {
> + LOGD(ERROR, ao->domid, "receive qemu colo-compare checkpoint error");
> + return -1;
> + }
> }
>
> size = colo_proxy_recv(cps, &buff, timeout_us);
> diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
> index c6d239a..065ea00 100644
> --- a/tools/libxl/libxl_colo_restore.c
> +++ b/tools/libxl/libxl_colo_restore.c
> @@ -613,7 +613,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
> }
> }
>
> - colo_proxy_preresume(&crs->cps);
> + if (!crs->cps.is_userspace_proxy)
> + colo_proxy_preresume(&crs->cps);
>
> colo_restore_resume_vm(egc, crcs);
>
> @@ -786,9 +787,11 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
> cds->ops = colo_restore_ops;
>
> crs->cps.ao = ao;
> - if (colo_proxy_setup(&crs->cps)) {
> - LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
> - goto out;
> + if (!crs->cps.is_userspace_proxy) {
> + if (colo_proxy_setup(&crs->cps)) {
> + LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
> + goto out;
> + }
> }
>
> if (init_device_subkind(cds))
> diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
> index 91e3fce..a0cfc5a 100644
> --- a/tools/libxl/libxl_colo_save.c
> +++ b/tools/libxl/libxl_colo_save.c
> @@ -86,6 +86,7 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
> libxl__checkpoint_devices_state *const cds = &dss->cds;
> libxl__srm_save_autogen_callbacks *const callbacks =
> &dss->sws.shs.callbacks.save.a;
> + libxl_device_nic *nics;
>
> STATE_AO_GC(dss->ao);
>
> @@ -110,24 +111,31 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
> css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
> libxl__xen_script_dir_path());
>
> - /* If enable userspace proxy mode, we don't need VIF */
> - if (css->cps.is_userspace_proxy)
> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
> - else
> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
> - (1 << LIBXL__DEVICE_KIND_VBD);
> -
> cds->ops = colo_ops;
> cds->callback = colo_save_setup_done;
> cds->ao = ao;
> cds->domid = dss->domid;
> cds->concrete_data = css;
>
> + /* If enable userspace proxy mode, we don't need VIF */
> + if (css->cps.is_userspace_proxy) {
> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
> +
> + /* Use this args we can connect to qemu colo-compare */
> + nics = libxl_device_nic_list(CTX, cds->domid, &cds->num_nics);
> + css->cps.checkpoint_host = nics->colo_checkpoint_host;
> + css->cps.checkpoint_port = nics->colo_checkpoint_port;
> + } else {
> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
> + (1 << LIBXL__DEVICE_KIND_VBD);
> + }
> +
> css->srs.ao = ao;
> css->srs.fd = css->recv_fd;
> css->srs.back_channel = true;
> libxl__stream_read_start(egc, &css->srs);
> css->cps.ao = ao;
> +
> if (colo_proxy_setup(&css->cps)) {
> LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
> goto out;
> diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
> index 5e1fecd..6bc6146 100644
> --- a/tools/libxl/libxl_nic.c
> +++ b/tools/libxl/libxl_nic.c
> @@ -246,6 +246,8 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
> MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_indev);
> MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_outdev);
> MAYBE_ADD_COLO_ARGS(filter_sec_rewriter0_queue);
> + MAYBE_ADD_COLO_ARGS(checkpoint_host);
> + MAYBE_ADD_COLO_ARGS(checkpoint_port);
>
> #undef MAYBE_ADD_COLO_ARGS
>
> @@ -451,6 +453,8 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
> CHECK_COLO_ARGS(filter_sec_redirector1_indev);
> CHECK_COLO_ARGS(filter_sec_redirector1_outdev);
> CHECK_COLO_ARGS(filter_sec_rewriter0_queue);
> + CHECK_COLO_ARGS(checkpoint_host);
> + CHECK_COLO_ARGS(checkpoint_port);
>
> #undef CHECK_COLO_ARGS
>
> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index 47e96b1..0b412ee 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -671,7 +671,9 @@ libxl_device_nic = Struct("device_nic", [
> ("colo_filter_sec_redirector1_queue", string),
> ("colo_filter_sec_redirector1_indev", string),
> ("colo_filter_sec_redirector1_outdev", string),
> - ("colo_filter_sec_rewriter0_queue", string)
> + ("colo_filter_sec_rewriter0_queue", string),
> + ("colo_checkpoint_host", string),
> + ("colo_checkpoint_port", string)
> ])
>
> libxl_device_pci = Struct("device_pci", [
> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
> index 32a47f6..ad5e193 100644
> --- a/tools/libxl/xl_cmdimpl.c
> +++ b/tools/libxl/xl_cmdimpl.c
> @@ -1145,6 +1145,10 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
> replace_string(&nic->colo_filter_sec_redirector1_outdev, oparg);
> } else if (MATCH_OPTION("colo_filter_sec_rewriter0_queue", token, oparg)) {
> replace_string(&nic->colo_filter_sec_rewriter0_queue, oparg);
> + } else if (MATCH_OPTION("colo_checkpoint_host", token, oparg)) {
> + replace_string(&nic->colo_checkpoint_host, oparg);
> + } else if (MATCH_OPTION("colo_checkpoint_port", token, oparg)) {
> + replace_string(&nic->colo_checkpoint_port, oparg);
> } else if (MATCH_OPTION("accel", token, oparg)) {
> fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
> } else {
> --
> 2.7.4
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy
2017-02-15 16:18 ` [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Konrad Rzeszutek Wilk
@ 2017-02-16 1:36 ` Zhang Chen
0 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-16 1:36 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang, Xen devel,
Bian Naimeng
On 02/16/2017 12:18 AM, Konrad Rzeszutek Wilk wrote:
> On Wed, Feb 15, 2017 at 05:54:26PM +0800, Zhang Chen wrote:
>> Because of some reason, We no longer support COLO kernel proxy.
>> So we send this patch set to make Xen use userspace colo-proxy in qemu.
> How come it is RFC? It looks quite baked?
We just want to get more comments, maybe we should remove the 'RFC' tag
in next version.
Thanks
Zhang Chen
>
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH RFC V2 7/7] COLO-Proxy: Use socket to get checkpoint event.
2017-02-15 16:24 ` Konrad Rzeszutek Wilk
@ 2017-02-16 1:48 ` Zhang Chen
0 siblings, 0 replies; 12+ messages in thread
From: Zhang Chen @ 2017-02-16 1:48 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang, Xen devel,
Bian Naimeng
On 02/16/2017 12:24 AM, Konrad Rzeszutek Wilk wrote:
> On Wed, Feb 15, 2017 at 05:54:33PM +0800, Zhang Chen wrote:
>> We use kernel colo proxy's way to get the checkpoint event
>> from qemu colo-compare.
>> Qemu colo-compare need add a API to support this(I will add this in qemu).
>>
>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
>> ---
>> tools/libxl/libxl_colo.h | 2 +
>> tools/libxl/libxl_colo_proxy.c | 84 +++++++++++++++++++++++++++++++++++++---
>> tools/libxl/libxl_colo_restore.c | 11 ++++--
>> tools/libxl/libxl_colo_save.c | 22 +++++++----
>> tools/libxl/libxl_nic.c | 4 ++
>> tools/libxl/libxl_types.idl | 4 +-
>> tools/libxl/xl_cmdimpl.c | 4 ++
>> 7 files changed, 113 insertions(+), 18 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
>> index 4746d8c..6c01b55 100644
>> --- a/tools/libxl/libxl_colo.h
>> +++ b/tools/libxl/libxl_colo.h
>> @@ -69,6 +69,8 @@ struct libxl__colo_proxy_state {
>> * False means use kernel colo proxy.
>> */
>> bool is_userspace_proxy;
>> + const char *checkpoint_host;
>> + const char *checkpoint_port;
>> };
>>
>> struct libxl__colo_save_state {
>> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
>> index dd902fc..9d21cf1 100644
>> --- a/tools/libxl/libxl_colo_proxy.c
>> +++ b/tools/libxl/libxl_colo_proxy.c
>> @@ -18,6 +18,9 @@
>> #include "libxl_internal.h"
>>
>> #include <netlink/netlink.h>
>> +#include <arpa/inet.h>
>> +#include <sys/socket.h>
>> +#include <netinet/in.h>
>>
>> /* Consistent with the new COLO netlink channel in kernel side */
>> #define NETLINK_COLO 28
>> @@ -76,6 +79,26 @@ static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
>> return ret;
>> }
>>
>> +static int colo_userspace_proxy_recv(libxl__colo_proxy_state *cps,
>> + char *buff,
>> + unsigned int timeout_us)
>> +{
>> + struct timeval tv;
>> + int ret;
>> +
>> + STATE_AO_GC(cps->ao);
>> +
>> + if (timeout_us) {
>> + tv.tv_sec = timeout_us / 1000000;
>> + tv.tv_usec = timeout_us % 1000000;
>> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
>> + }
>> +
>> + ret = recv(cps->sock_fd, buff, sizeof(buff),0);
>> +
>> + return ret;
>> +}
>> +
>> /* error: return -1, otherwise return 0 */
>> static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
>> unsigned int timeout_us)
>> @@ -153,8 +176,45 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
>> STATE_AO_GC(cps->ao);
>>
>> /* If enable userspace proxy mode, we don't need setup kernel proxy */
>> - if (cps->is_userspace_proxy)
>> + if (cps->is_userspace_proxy) {
>> + struct sockaddr_in addr;
>> + int port;
>> + char recvbuff[1024];
>> +
>> + memset(&addr, 0, sizeof(addr));
>> + port = atoi(cps->checkpoint_port);
>> + addr.sin_family = AF_INET;
>> + addr.sin_port = htons(port);
>> + addr.sin_addr.s_addr = inet_addr(cps->checkpoint_host);
>> +
>> + skfd = socket(AF_INET, SOCK_STREAM, 0);
>> + if (skfd < 0) {
>> + LOGD(ERROR, ao->domid, "can not create a TCP socket: %s",
>> + strerror(errno));
>> + goto out;
>> + }
>> +
>> + cps->sock_fd = skfd;
>> +
>> + if (connect(skfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
>> + LOGD(ERROR, ao->domid, "connect error");
>> + goto out;
>> + }
>> +
>> + char sendbuf[] = "COLO_USERSPACE_PROXY_INIT";
> Um, could you move it right around 'char recvbuff' ?
OK, I will move it in next version.
>> + ret = send(skfd, sendbuf, strlen(sendbuf),0);
>> + if (ret < 0)
>> + goto out;
>> +
>> + ret = colo_userspace_proxy_recv(cps, recvbuff, 500000);
> That 500000 looks like a good candidate for a #define?
I will add a macro "#define COLO_DEFAULT_WAIT_TIME 500000".
Thanks
Zhang Chen
>
>> + if (size < 0) {
s/size/ret
>> + LOGD(ERROR, ao->domid, "Can't recv msg from qemu colo-compare: %s",
>> + strerror(errno));
>> + goto out;
>> + }
>> +
>> return 0;
>> + }
>>
>> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>> if (skfd < 0) {
>> @@ -247,8 +307,11 @@ void colo_proxy_preresume(libxl__colo_proxy_state *cps)
>> * If enable userspace proxy mode,
>> * we don't need preresume kernel proxy
>> */
>> - if (cps->is_userspace_proxy)
>> + if (cps->is_userspace_proxy) {
>> + char sendbuf[] = "COLO_CHECKPOINT";
>> + send(cps->sock_fd, sendbuf, strlen(sendbuf),0);
>> return;
>> + }
>>
>> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
>> /* TODO: need to handle if the call fails... */
>> @@ -277,16 +340,25 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
>> struct nlmsghdr *h;
>> struct colo_msg *m;
>> int ret = -1;
>> + char recvbuff[1024];
>>
>> STATE_AO_GC(cps->ao);
>>
>> /*
>> - * enable userspace proxy mode, tmp sleep.
>> - * then we will add qemu API support this func.
>> + * enable userspace proxy mode.
>> + * Then we will add qemu API support for this func.
>> */
>> if (cps->is_userspace_proxy) {
>> - sleep(timeout_us / 1000000);
>> - return 0;
>> + ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
>> + if (ret <= 0)
>> + return 0;
>> +
>> + if (!strcmp(recvbuff, "DO_CHECKPOINT")) {
>> + return 1;
>> + } else {
>> + LOGD(ERROR, ao->domid, "receive qemu colo-compare checkpoint error");
>> + return -1;
>> + }
>> }
>>
>> size = colo_proxy_recv(cps, &buff, timeout_us);
>> diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
>> index c6d239a..065ea00 100644
>> --- a/tools/libxl/libxl_colo_restore.c
>> +++ b/tools/libxl/libxl_colo_restore.c
>> @@ -613,7 +613,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
>> }
>> }
>>
>> - colo_proxy_preresume(&crs->cps);
>> + if (!crs->cps.is_userspace_proxy)
>> + colo_proxy_preresume(&crs->cps);
>>
>> colo_restore_resume_vm(egc, crcs);
>>
>> @@ -786,9 +787,11 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
>> cds->ops = colo_restore_ops;
>>
>> crs->cps.ao = ao;
>> - if (colo_proxy_setup(&crs->cps)) {
>> - LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
>> - goto out;
>> + if (!crs->cps.is_userspace_proxy) {
>> + if (colo_proxy_setup(&crs->cps)) {
>> + LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
>> + goto out;
>> + }
>> }
>>
>> if (init_device_subkind(cds))
>> diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
>> index 91e3fce..a0cfc5a 100644
>> --- a/tools/libxl/libxl_colo_save.c
>> +++ b/tools/libxl/libxl_colo_save.c
>> @@ -86,6 +86,7 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
>> libxl__checkpoint_devices_state *const cds = &dss->cds;
>> libxl__srm_save_autogen_callbacks *const callbacks =
>> &dss->sws.shs.callbacks.save.a;
>> + libxl_device_nic *nics;
>>
>> STATE_AO_GC(dss->ao);
>>
>> @@ -110,24 +111,31 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
>> css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
>> libxl__xen_script_dir_path());
>>
>> - /* If enable userspace proxy mode, we don't need VIF */
>> - if (css->cps.is_userspace_proxy)
>> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
>> - else
>> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
>> - (1 << LIBXL__DEVICE_KIND_VBD);
>> -
>> cds->ops = colo_ops;
>> cds->callback = colo_save_setup_done;
>> cds->ao = ao;
>> cds->domid = dss->domid;
>> cds->concrete_data = css;
>>
>> + /* If enable userspace proxy mode, we don't need VIF */
>> + if (css->cps.is_userspace_proxy) {
>> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
>> +
>> + /* Use this args we can connect to qemu colo-compare */
>> + nics = libxl_device_nic_list(CTX, cds->domid, &cds->num_nics);
>> + css->cps.checkpoint_host = nics->colo_checkpoint_host;
>> + css->cps.checkpoint_port = nics->colo_checkpoint_port;
>> + } else {
>> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
>> + (1 << LIBXL__DEVICE_KIND_VBD);
>> + }
>> +
>> css->srs.ao = ao;
>> css->srs.fd = css->recv_fd;
>> css->srs.back_channel = true;
>> libxl__stream_read_start(egc, &css->srs);
>> css->cps.ao = ao;
>> +
>> if (colo_proxy_setup(&css->cps)) {
>> LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
>> goto out;
>> diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
>> index 5e1fecd..6bc6146 100644
>> --- a/tools/libxl/libxl_nic.c
>> +++ b/tools/libxl/libxl_nic.c
>> @@ -246,6 +246,8 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
>> MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_indev);
>> MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_outdev);
>> MAYBE_ADD_COLO_ARGS(filter_sec_rewriter0_queue);
>> + MAYBE_ADD_COLO_ARGS(checkpoint_host);
>> + MAYBE_ADD_COLO_ARGS(checkpoint_port);
>>
>> #undef MAYBE_ADD_COLO_ARGS
>>
>> @@ -451,6 +453,8 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
>> CHECK_COLO_ARGS(filter_sec_redirector1_indev);
>> CHECK_COLO_ARGS(filter_sec_redirector1_outdev);
>> CHECK_COLO_ARGS(filter_sec_rewriter0_queue);
>> + CHECK_COLO_ARGS(checkpoint_host);
>> + CHECK_COLO_ARGS(checkpoint_port);
>>
>> #undef CHECK_COLO_ARGS
>>
>> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
>> index 47e96b1..0b412ee 100644
>> --- a/tools/libxl/libxl_types.idl
>> +++ b/tools/libxl/libxl_types.idl
>> @@ -671,7 +671,9 @@ libxl_device_nic = Struct("device_nic", [
>> ("colo_filter_sec_redirector1_queue", string),
>> ("colo_filter_sec_redirector1_indev", string),
>> ("colo_filter_sec_redirector1_outdev", string),
>> - ("colo_filter_sec_rewriter0_queue", string)
>> + ("colo_filter_sec_rewriter0_queue", string),
>> + ("colo_checkpoint_host", string),
>> + ("colo_checkpoint_port", string)
>> ])
>>
>> libxl_device_pci = Struct("device_pci", [
>> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
>> index 32a47f6..ad5e193 100644
>> --- a/tools/libxl/xl_cmdimpl.c
>> +++ b/tools/libxl/xl_cmdimpl.c
>> @@ -1145,6 +1145,10 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
>> replace_string(&nic->colo_filter_sec_redirector1_outdev, oparg);
>> } else if (MATCH_OPTION("colo_filter_sec_rewriter0_queue", token, oparg)) {
>> replace_string(&nic->colo_filter_sec_rewriter0_queue, oparg);
>> + } else if (MATCH_OPTION("colo_checkpoint_host", token, oparg)) {
>> + replace_string(&nic->colo_checkpoint_host, oparg);
>> + } else if (MATCH_OPTION("colo_checkpoint_port", token, oparg)) {
>> + replace_string(&nic->colo_checkpoint_port, oparg);
>> } else if (MATCH_OPTION("accel", token, oparg)) {
>> fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
>> } else {
>> --
>> 2.7.4
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-02-16 1:48 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-02-15 9:54 [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 3/7] tools/libxl: refactor do_domain_create() Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 5/7] COLO-Proxy: Add primary userspace colo proxy start args Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 6/7] COLO-Proxy: Add secondary userspace colo-proxy " Zhang Chen
2017-02-15 9:54 ` [PATCH RFC V2 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
2017-02-15 16:24 ` Konrad Rzeszutek Wilk
2017-02-16 1:48 ` Zhang Chen
2017-02-15 16:18 ` [PATCH RFC V2 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Konrad Rzeszutek Wilk
2017-02-16 1:36 ` Zhang Chen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).