* [PATCH v10 0/5] Remus/Libxl: Network buffering support
@ 2014-05-21 10:14 Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 1/5] libxl: introduce asynchronous execution API Yang Hongyang
` (6 more replies)
0 siblings, 7 replies; 9+ messages in thread
From: Yang Hongyang @ 2014-05-21 10:14 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, eddie.dong, rshriram, roger.pau,
laijs
This patch series adds support for network buffering in the Remus
codebase in libxl.
Changes in V10:
Restructured the whole patch series.
Introduce the remus device abstract layer.
Make remus checkpoint asynchronous.
Changes in V9:
Use async exec script api to exec scripts.
Changes in V8:
Applied some comments(by IanJ).
Merge some struct definitions to it's implementation.
(2/3/5 in V7 => 3 in V8)
Changes in V7:
Applied missing comments(by IanJ).
Applied Shriram comments.
merge netbufering tangled setup/teardown code into one patch.
(2/6/8 in V6 => 5 in V7. 9/10 in V6 => 7 in V7)
Changes in V6:
Applied Ian Jackson's comments of V5 series.
the [PATCH 2/4 V5] is split by small functionalities.
[PATCH 4/4 V5] --> [PATCH 13/13] netbuffer is default enabled.
Changes in V5:
Merge hotplug script patch (2/5) and hotplug script setup/teardown
patch (3/5) into a single patch.
Changes in V4:
[1/5] Remove check for libnl command line utils in autoconf checks
[2/5] minor nits
[3/5] define LIBXL_HAVE_REMUS_NETBUF in libxl.h
[4/5] clean ups. Make the usleep in checkpoint callback asynchronous
[5/5] minor nits
Changes in V3:
[1/5] Fix redundant checks in configure scripts
(based on Ian Campbell's suggestions)
[2/5] Introduce locking in the script, during IFB setup.
Add xenstore paths used by netbuf scripts
to xenstore-paths.markdown
[3/5] Hotplug scripts setup/teardown invocations are now asynchronous
following IanJ's feedback. However, the invocations are still
sequential.
[5/5] Allow per-domain specification of netbuffer scripts in xl remus
commmand.
And minor nits throughout the series based on feedback from
the last version
Changes in V2:
[1/5] Configure script will automatically enable/disable network
buffer support depending on the availability of the appropriate
libnl3 version. [If libnl3 is unavailable, a warning message will be
printed to let the user know that the feature has been disabled.]
use macros from pkg.m4 instead of pkg-config commands
removed redundant checks for libnl3 libraries.
[3,4/5] - Minor nits.
Version 1:
[1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
to libxl Makefile.
[2/5] External script to setup/teardown network buffering using libnl3's
CLI. This script will be invoked by libxl before starting Remus.
The script's main job is to bring up an IFB device with plug qdisc
attached to it. It then re-routes egress traffic from the guest's
vif to the IFB device.
[3/5] Libxl code to invoke the external setup script, followed by netlink
related setup to obtain a handle on the output buffers attached
to each vif.
[4/5] Libxl interaction with network buffer module in the kernel via
libnl3 API.
[5/5] xl cmdline switch to explicitly enable network buffering when
starting remus.
Few things to note(by shriram):
a) Based on previous email discussions, the setup/teardown task has
been moved to a hotplug style shell script which can be customized as
desired, instead of implementing it as C code inside libxl.
b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
(Linux). So I have made network buffering support an optional feature
so that it can be disabled if desired.
c) NetBSD does not have libnl3. So I have put the setup script under
tools/hotplug/Linux folder.
thanks
Shriram Rajagopalan (1):
libxl: network buffering cmdline switch
Yang Hongyang (4):
libxl: introduce asynchronous execution API
remus: add libnl3 dependency for network buffering support
remus: introduce remus device
remus: implement remus network buffering for nic devices
README | 4 +
config/Tools.mk.in | 4 +
docs/man/xl.conf.pod.5 | 6 +
docs/man/xl.pod.1 | 11 +-
docs/misc/xenstore-paths.markdown | 4 +
tools/configure.ac | 15 +
tools/hotplug/Linux/Makefile | 1 +
tools/hotplug/Linux/remus-netbuf-setup | 183 +++++++++++
tools/libxl/Makefile | 15 +
tools/libxl/libxl.c | 52 +++-
tools/libxl/libxl.h | 13 +
tools/libxl/libxl_aoutils.c | 89 ++++++
tools/libxl/libxl_device.c | 78 ++---
tools/libxl/libxl_dom.c | 132 +++++++-
tools/libxl/libxl_internal.h | 151 ++++++++-
tools/libxl/libxl_netbuffer.c | 550 +++++++++++++++++++++++++++++++++
tools/libxl/libxl_nonetbuffer.c | 98 ++++++
tools/libxl/libxl_remus_device.c | 323 +++++++++++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 2 +-
tools/libxl/libxl_types.idl | 3 +
tools/libxl/xl.c | 4 +
tools/libxl/xl.h | 1 +
tools/libxl/xl_cmdimpl.c | 28 +-
tools/libxl/xl_cmdtable.c | 3 +
tools/remus/README | 6 +
25 files changed, 1694 insertions(+), 82 deletions(-)
create mode 100644 tools/hotplug/Linux/remus-netbuf-setup
create mode 100644 tools/libxl/libxl_netbuffer.c
create mode 100644 tools/libxl/libxl_nonetbuffer.c
create mode 100644 tools/libxl/libxl_remus_device.c
--
1.9.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v10 1/5] libxl: introduce asynchronous execution API
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
@ 2014-05-21 10:14 ` Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 2/5] remus: add libnl3 dependency for network buffering support Yang Hongyang
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Yang Hongyang @ 2014-05-21 10:14 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, eddie.dong, rshriram, roger.pau,
laijs
1.introduce asynchronous execution API:
libxl__async_exec_init
libxl__async_exec_start
libxl__async_exec_inuse
2.use the async exec API to execute device hotplug scripts
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Acked-by: Roger Pau Monné <roger.pau@citrix.com
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
---
tools/libxl/libxl_aoutils.c | 89 ++++++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_device.c | 78 +++++++++++---------------------------
tools/libxl/libxl_internal.h | 34 +++++++++++++++--
3 files changed, 141 insertions(+), 60 deletions(-)
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index 1c9eb9e..b10d2e1 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -451,3 +451,92 @@ int libxl__openptys(libxl__openpty_state *op,
return rc;
}
+static void async_exec_timeout(libxl__egc *egc,
+ libxl__ev_time *ev,
+ const struct timeval *requested_abs)
+{
+ libxl__async_exec_state *aes = CONTAINER_OF(ev, *aes, time);
+ STATE_AO_GC(aes->ao);
+
+ libxl__ev_time_deregister(gc, &aes->time);
+
+ assert(libxl__ev_child_inuse(&aes->child));
+ LOG(ERROR, "killing execution of %s because of timeout", aes->what);
+
+ if (kill(aes->child.pid, SIGKILL)) {
+ LOGEV(ERROR, errno, "unable to kill %s [%ld]",
+ aes->what, (unsigned long)aes->child.pid);
+ }
+
+ return;
+}
+
+static void async_exec_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ pid_t pid, int status)
+{
+ libxl__async_exec_state *aes = CONTAINER_OF(child, *aes, child);
+ STATE_AO_GC(aes->ao);
+
+ libxl__ev_time_deregister(gc, &aes->time);
+
+ if (status) {
+ libxl_report_child_exitstatus(CTX, LIBXL__LOG_ERROR,
+ aes->what, pid, status);
+ }
+
+ aes->callback(egc, aes, status);
+}
+
+void libxl__async_exec_init(libxl__async_exec_state *aes)
+{
+ libxl__ev_time_init(&aes->time);
+ libxl__ev_child_init(&aes->child);
+}
+
+int libxl__async_exec_start(libxl__gc *gc, libxl__async_exec_state *aes)
+{
+ pid_t pid;
+
+ /* Convenience aliases */
+ libxl__ev_child *const child = &aes->child;
+ char ** const args = aes->args;
+
+ /* Set execution timeout */
+ if (libxl__ev_time_register_rel(gc, &aes->time,
+ async_exec_timeout,
+ aes->timeout_ms)) {
+ LOG(ERROR, "unable to register timeout for executing: %s", aes->what);
+ goto out;
+ }
+
+ LOG(DEBUG, "forking to execute: %s ", aes->what);
+
+ /* Fork and exec */
+ pid = libxl__ev_child_fork(gc, child, async_exec_done);
+ if (pid == -1) {
+ LOG(ERROR, "unable to fork");
+ goto out;
+ }
+
+ if (!pid) {
+ /* child */
+ libxl__exec(gc, aes->stdfds[0], aes->stdfds[1],
+ aes->stdfds[2], args[0], args, aes->env);
+ /* notreached */
+ abort();
+ }
+
+ return 0;
+
+out:
+ return ERROR_FAIL;
+}
+
+bool libxl__async_exec_inuse(const libxl__async_exec_state *aes)
+{
+ bool time_inuse = libxl__ev_time_isregistered(&aes->time);
+ bool child_inuse = libxl__ev_child_inuse(&aes->child);
+ assert(time_inuse == child_inuse);
+ return child_inuse;
+}
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index fa99f77..90ae564 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -430,7 +430,7 @@ void libxl__prepare_ao_device(libxl__ao *ao, libxl__ao_device *aodev)
aodev->rc = 0;
aodev->dev = NULL;
aodev->num_exec = 0;
- /* Initialize timer for QEMU Bodge and hotplug execution */
+ /* Initialize timer for QEMU Bodge */
libxl__ev_time_init(&aodev->timeout);
/*
* Initialize xs_watch, because it's not used on all possible
@@ -440,7 +440,7 @@ void libxl__prepare_ao_device(libxl__ao *ao, libxl__ao_device *aodev)
aodev->active = 1;
/* We init this here because we might call device_hotplug_done
* without actually calling any hotplug script */
- libxl__ev_child_init(&aodev->child);
+ libxl__async_exec_init(&aodev->aes);
}
/* multidev */
@@ -707,12 +707,9 @@ static void device_backend_cleanup(libxl__gc *gc,
static void device_hotplug(libxl__egc *egc, libxl__ao_device *aodev);
-static void device_hotplug_timeout_cb(libxl__egc *egc, libxl__ev_time *ev,
- const struct timeval *requested_abs);
-
static void device_hotplug_child_death_cb(libxl__egc *egc,
- libxl__ev_child *child,
- pid_t pid, int status);
+ libxl__async_exec_state *aes,
+ int status);
static void device_destroy_be_timeout_cb(libxl__egc *egc, libxl__ev_time *ev,
const struct timeval *requested_abs);
@@ -953,11 +950,11 @@ static void device_backend_cleanup(libxl__gc *gc, libxl__ao_device *aodev)
static void device_hotplug(libxl__egc *egc, libxl__ao_device *aodev)
{
STATE_AO_GC(aodev->ao);
+ libxl__async_exec_state *aes = &aodev->aes;
char *be_path = libxl__device_backend_path(gc, aodev->dev);
char **args = NULL, **env = NULL;
int rc = 0;
int hotplug, nullfd = -1;
- pid_t pid;
uint32_t domid;
/*
@@ -1009,16 +1006,6 @@ static void device_hotplug(libxl__egc *egc, libxl__ao_device *aodev)
goto out;
}
- /* Set hotplug timeout */
- rc = libxl__ev_time_register_rel(gc, &aodev->timeout,
- device_hotplug_timeout_cb,
- LIBXL_HOTPLUG_TIMEOUT * 1000);
- if (rc) {
- LOG(ERROR, "unable to register timeout for hotplug device %s", be_path);
- goto out;
- }
-
- aodev->what = GCSPRINTF("%s %s", args[0], args[1]);
LOG(DEBUG, "calling hotplug script: %s %s", args[0], args[1]);
nullfd = open("/dev/null", O_RDONLY);
@@ -1028,23 +1015,22 @@ static void device_hotplug(libxl__egc *egc, libxl__ao_device *aodev)
goto out;
}
- /* fork and execute hotplug script */
- pid = libxl__ev_child_fork(gc, &aodev->child, device_hotplug_child_death_cb);
- if (pid == -1) {
- LOG(ERROR, "unable to fork");
- rc = ERROR_FAIL;
+ aes->ao = ao;
+ aes->what = GCSPRINTF("%s %s", args[0], args[1]);
+ aes->env = env;
+ aes->args = args;
+ aes->callback = device_hotplug_child_death_cb;
+ aes->timeout_ms = LIBXL_HOTPLUG_TIMEOUT * 1000;
+ aes->stdfds[0] = nullfd;
+ aes->stdfds[1] = 2;
+ aes->stdfds[2] = -1;
+
+ rc = libxl__async_exec_start(gc, aes);
+ if (rc)
goto out;
- }
-
- if (!pid) {
- /* child */
- libxl__exec(gc, nullfd, 2, -1, args[0], args, env);
- /* notreached */
- abort();
- }
close(nullfd);
- assert(libxl__ev_child_inuse(&aodev->child));
+ assert(libxl__async_exec_inuse(&aodev->aes));
return;
@@ -1055,29 +1041,11 @@ out:
return;
}
-static void device_hotplug_timeout_cb(libxl__egc *egc, libxl__ev_time *ev,
- const struct timeval *requested_abs)
-{
- libxl__ao_device *aodev = CONTAINER_OF(ev, *aodev, timeout);
- STATE_AO_GC(aodev->ao);
-
- libxl__ev_time_deregister(gc, &aodev->timeout);
-
- assert(libxl__ev_child_inuse(&aodev->child));
- LOG(DEBUG, "killing hotplug script %s because of timeout", aodev->what);
- if (kill(aodev->child.pid, SIGKILL)) {
- LOGEV(ERROR, errno, "unable to kill hotplug script %s [%ld]",
- aodev->what, (unsigned long)aodev->child.pid);
- }
-
- return;
-}
-
static void device_hotplug_child_death_cb(libxl__egc *egc,
- libxl__ev_child *child,
- pid_t pid, int status)
+ libxl__async_exec_state *aes,
+ int status)
{
- libxl__ao_device *aodev = CONTAINER_OF(child, *aodev, child);
+ libxl__ao_device *aodev = CONTAINER_OF(aes, *aodev, aes);
STATE_AO_GC(aodev->ao);
char *be_path = libxl__device_backend_path(gc, aodev->dev);
char *hotplug_error;
@@ -1085,8 +1053,6 @@ static void device_hotplug_child_death_cb(libxl__egc *egc,
device_hotplug_clean(gc, aodev);
if (status) {
- libxl_report_child_exitstatus(CTX, LIBXL__LOG_ERROR,
- aodev->what, pid, status);
hotplug_error = libxl__xs_read(gc, XBT_NULL,
GCSPRINTF("%s/hotplug-error", be_path));
if (hotplug_error)
@@ -1178,7 +1144,7 @@ static void device_hotplug_clean(libxl__gc *gc, libxl__ao_device *aodev)
/* Clean events and check reentrancy */
libxl__ev_time_deregister(gc, &aodev->timeout);
libxl__ev_xswatch_deregister(gc, &aodev->xs_watch);
- assert(!libxl__ev_child_inuse(&aodev->child));
+ assert(!libxl__async_exec_inuse(&aodev->aes));
}
static void devices_remove_callback(libxl__egc *egc,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index c2b73c4..fe4a3d1 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2030,6 +2030,33 @@ _hidden const char *libxl__xen_script_dir_path(void);
_hidden const char *libxl__lock_dir_path(void);
_hidden const char *libxl__run_dir_path(void);
+/*----- subprocess execution with timeout -----*/
+
+typedef struct libxl__async_exec_state libxl__async_exec_state;
+
+typedef void libxl__async_exec_callback(libxl__egc *egc,
+ libxl__async_exec_state *aes, int status);
+
+struct libxl__async_exec_state {
+ /* caller must fill these in */
+ libxl__ao *ao;
+ const char *what; /* for error msgs, what we're executing */
+ int timeout_ms;
+ libxl__async_exec_callback *callback;
+ /* caller must fill in; as for libxl__exec */
+ int stdfds[3];
+ char **args; /* execution arguments */
+ char **env; /* execution environment */
+
+ /* private */
+ libxl__ev_time time;
+ libxl__ev_child child;
+};
+
+void libxl__async_exec_init(libxl__async_exec_state *aes);
+int libxl__async_exec_start(libxl__gc *gc, libxl__async_exec_state *aes);
+bool libxl__async_exec_inuse(const libxl__async_exec_state *aes);
+
/*----- device addition/removal -----*/
typedef struct libxl__ao_device libxl__ao_device;
@@ -2066,14 +2093,13 @@ struct libxl__ao_device {
libxl__multidev *multidev; /* reference to the containing multidev */
/* private for add/remove implementation */
libxl__ev_devstate backend_ds;
- /* Bodge for Qemu devices, also used for timeout of hotplug execution */
+ /* Bodge for Qemu devices */
libxl__ev_time timeout;
/* xenstore watch for backend path of driver domains */
libxl__ev_xswatch xs_watch;
- /* device hotplug execution */
- const char *what;
int num_exec;
- libxl__ev_child child;
+ /* for calling hotplug scripts */
+ libxl__async_exec_state aes;
};
/*
--
1.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v10 2/5] remus: add libnl3 dependency for network buffering support
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 1/5] libxl: introduce asynchronous execution API Yang Hongyang
@ 2014-05-21 10:14 ` Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 3/5] remus: introduce remus device Yang Hongyang
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Yang Hongyang @ 2014-05-21 10:14 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, eddie.dong, rshriram, roger.pau,
laijs
Libnl3 is required for controlling Remus network buffering.
This patch adds dependency on libnl3 (>= 3.2.8) to autoconf scripts.
Also provide ability to configure tools without libnl3 support, that
is without network buffering support.
when there's no network buffering support,libxl__netbuffer_enabled()
returns 0, otherwise returns 1.
Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca>
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Reviewed-by: Wen Congyang <wency@cn.fujitsu.com>
---
README | 4 ++++
config/Tools.mk.in | 4 ++++
tools/configure.ac | 15 +++++++++++++++
tools/libxl/Makefile | 13 +++++++++++++
tools/libxl/libxl_internal.h | 1 +
tools/libxl/libxl_netbuffer.c | 31 +++++++++++++++++++++++++++++++
tools/libxl/libxl_nonetbuffer.c | 31 +++++++++++++++++++++++++++++++
tools/remus/README | 6 ++++++
8 files changed, 105 insertions(+)
create mode 100644 tools/libxl/libxl_netbuffer.c
create mode 100644 tools/libxl/libxl_nonetbuffer.c
diff --git a/README b/README
index 9bbe734..e770932 100644
--- a/README
+++ b/README
@@ -72,6 +72,10 @@ disabled at compile time:
* cmake (if building vtpm stub domains)
* markdown
* figlet (for generating the traditional Xen start of day banner)
+ * Development install of libnl3 (e.g., libnl-3-200,
+ libnl-3-dev, etc). Required if network buffering is desired
+ when using Remus with libxl. See tools/remus/README for detailed
+ information.
Second, you need to acquire a suitable kernel for use in domain 0. If
possible you should use a kernel provided by your OS distributor. If
diff --git a/config/Tools.mk.in b/config/Tools.mk.in
index 84b2612..06c9d25 100644
--- a/config/Tools.mk.in
+++ b/config/Tools.mk.in
@@ -38,6 +38,9 @@ PTHREAD_LIBS := @PTHREAD_LIBS@
PTYFUNCS_LIBS := @PTYFUNCS_LIBS@
+LIBNL3_LIBS := @LIBNL3_LIBS@
+LIBNL3_CFLAGS := @LIBNL3_CFLAGS@
+
# Download GIT repositories via HTTP or GIT's own protocol?
# GIT's protocol is faster and more robust, when it works at all (firewalls
# may block it). We make it the default, but if your GIT repository downloads
@@ -56,6 +59,7 @@ CONFIG_QEMU_XEN := @qemu_xen@
CONFIG_BLKTAP1 := @blktap1@
CONFIG_VTPM := @vtpm@
CONFIG_QEMUU_EXTRA_ARGS:= @EXTRA_QEMUU_CONFIGURE_ARGS@
+CONFIG_REMUS_NETBUF := @remus_netbuf@
#System options
ZLIB := @zlib@
diff --git a/tools/configure.ac b/tools/configure.ac
index 25d7ca3..38d2d05 100644
--- a/tools/configure.ac
+++ b/tools/configure.ac
@@ -247,5 +247,20 @@ esac
# Checks for header files.
AC_CHECK_HEADERS([yajl/yajl_version.h sys/eventfd.h])
+# Check for libnl3 >=3.2.8. If present enable remus network buffering.
+PKG_CHECK_MODULES(LIBNL3, [libnl-3.0 >= 3.2.8 libnl-route-3.0 >= 3.2.8],
+ [libnl3_lib="y"], [libnl3_lib="n"])
+
+AS_IF([test "x$libnl3_lib" = "xn" ], [
+ AC_MSG_WARN([Disabling support for Remus network buffering.
+ Please install libnl3 libraries, command line tools and devel
+ headers - version 3.2.8 or higher])
+ AC_SUBST(remus_netbuf, [n])
+ ],[
+ AC_SUBST(LIBNL3_LIBS)
+ AC_SUBST(LIBNL3_CFLAGS)
+ AC_SUBST(remus_netbuf, [y])
+])
+
AC_OUTPUT()
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 4cfa275..a572dca 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -21,11 +21,17 @@ endif
LIBXL_LIBS =
LIBXL_LIBS = $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(LDLIBS_libblktapctl) $(PTYFUNCS_LIBS) $(LIBUUID_LIBS)
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+LIBXL_LIBS += $(LIBNL3_LIBS)
+endif
CFLAGS_LIBXL += $(CFLAGS_libxenctrl)
CFLAGS_LIBXL += $(CFLAGS_libxenguest)
CFLAGS_LIBXL += $(CFLAGS_libxenstore)
CFLAGS_LIBXL += $(CFLAGS_libblktapctl)
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+CFLAGS_LIBXL += $(LIBNL3_CFLAGS)
+endif
CFLAGS_LIBXL += -Wshadow
LIBXL_LIBS-$(CONFIG_ARM) += -lfdt
@@ -43,6 +49,13 @@ LIBXL_OBJS-y += libxl_blktap2.o
else
LIBXL_OBJS-y += libxl_noblktap2.o
endif
+
+ifeq ($(CONFIG_REMUS_NETBUF),y)
+LIBXL_OBJS-y += libxl_netbuffer.o
+else
+LIBXL_OBJS-y += libxl_nonetbuffer.o
+endif
+
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index fe4a3d1..de0a807 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2442,6 +2442,7 @@ typedef struct libxl__save_helper_state {
* marshalling and xc callback functions */
} libxl__save_helper_state;
+_hidden int libxl__netbuffer_enabled(libxl__gc *gc);
/*----- Domain suspend (save) state structure -----*/
diff --git a/tools/libxl/libxl_netbuffer.c b/tools/libxl/libxl_netbuffer.c
new file mode 100644
index 0000000..8e23d75
--- /dev/null
+++ b/tools/libxl/libxl_netbuffer.c
@@ -0,0 +1,31 @@
+/*
+ * Copyright (C) 2013
+ * Author Shriram Rajagopalan <rshriram@cs.ubc.ca>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+int libxl__netbuffer_enabled(libxl__gc *gc)
+{
+ return 1;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_nonetbuffer.c b/tools/libxl/libxl_nonetbuffer.c
new file mode 100644
index 0000000..6aa4bf1
--- /dev/null
+++ b/tools/libxl/libxl_nonetbuffer.c
@@ -0,0 +1,31 @@
+/*
+ * Copyright (C) 2013
+ * Author Shriram Rajagopalan <rshriram@cs.ubc.ca>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+int libxl__netbuffer_enabled(libxl__gc *gc)
+{
+ return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/remus/README b/tools/remus/README
index 9e8140b..4736252 100644
--- a/tools/remus/README
+++ b/tools/remus/README
@@ -2,3 +2,9 @@ Remus provides fault tolerance for virtual machines by sending continuous
checkpoints to a backup, which will activate if the target VM fails.
See the website at http://nss.cs.ubc.ca/remus/ for details.
+
+Using Remus with libxl on Xen 4.4 and higher:
+ To enable network buffering, you need libnl 3.2.8
+ or higher along with the development headers and command line utilities.
+ If your distro does not have the appropriate libnl3 version, you can find
+ the latest source tarball of libnl3 at http://www.carisma.slowglass.com/~tgr/libnl/
--
1.9.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v10 3/5] remus: introduce remus device
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 1/5] libxl: introduce asynchronous execution API Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 2/5] remus: add libnl3 dependency for network buffering support Yang Hongyang
@ 2014-05-21 10:14 ` Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 4/5] remus: implement remus network buffering for nic devices Yang Hongyang
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Yang Hongyang @ 2014-05-21 10:14 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, eddie.dong, rshriram, roger.pau,
laijs
introduce remus device, an abstract layer of remus devices(nic, disk,
etc).It provide the following APIs for libxl:
>libxl__remus_device_setup
setup remus devices, like attach qdisc, enable disk buffering, etc
>libxl__remus_device_teardown
teardown devices
>libxl__remus_device_postsuspend
>libxl__remus_device_preresume
>libxl__remus_device_commit
above three are for checkpoint.
through remus device layer, the remus execution flow will be like
this:
xl remus -> remus device setup
|-> remus checkpoint(postsuspend, commit, preresume)
...
|-> remus device teardown,failover or abort
the remus device layer provide an interface
libxl__remus_device_ops
which a remus device must implement.the whole remus structure:
|remus|
|
|remus device|
|
|nic| |drbd disks| |qemu disks| ...
a device(nic, drbd disks, qemu disks, etc) must implement
libxl__remus_device_ops to support remus.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
---
tools/libxl/Makefile | 2 +
tools/libxl/libxl.c | 34 ++++-
tools/libxl/libxl_dom.c | 132 ++++++++++++++--
tools/libxl/libxl_internal.h | 113 ++++++++++++++
tools/libxl/libxl_remus_device.c | 303 +++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 2 +-
tools/libxl/libxl_types.idl | 1 +
7 files changed, 572 insertions(+), 15 deletions(-)
create mode 100644 tools/libxl/libxl_remus_device.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index a572dca..7a722a8 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -56,6 +56,8 @@ else
LIBXL_OBJS-y += libxl_nonetbuffer.o
endif
+LIBXL_OBJS-y += libxl_remus_device.o
+
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index d59ce0c..2e7a6ea 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -713,6 +713,31 @@ out:
static void remus_failover_cb(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc);
+static void libxl__remus_setup_failed(libxl__egc *egc,
+ libxl__remus_state *rs, int rc)
+{
+ STATE_AO_GC(rs->ao);
+ libxl__ao_complete(egc,ao,rc);
+}
+
+static void libxl__remus_setup_done(libxl__egc *egc,
+ libxl__remus_state *rs, int rc)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(rs, *dss, rs);
+ STATE_AO_GC(rs->ao);
+
+ if (!rc) {
+ libxl__domain_suspend(egc, dss);
+ return;
+ }
+
+ LOG(ERROR, "Remus: failed to setup device for guest with domid %u",
+ dss->domid);
+ rs->saved_rc = rc;
+ rs->callback = libxl__remus_setup_failed;
+ libxl__remus_device_teardown(egc, rs);
+}
+
/* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
uint32_t domid, int send_fd, int recv_fd,
@@ -741,10 +766,15 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
assert(info);
- /* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */
+ /* Convenience aliases */
+ libxl__remus_state *const rs = &dss->rs;
+ rs->ao = ao;
+ rs->domid = domid;
+ rs->saved_rc = 0;
+ rs->callback = libxl__remus_setup_done;
/* Point of no return */
- libxl__domain_suspend(egc, dss);
+ libxl__remus_device_setup(egc, rs);
return AO_INPROGRESS;
out:
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 661999c..70765a3 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1444,31 +1444,65 @@ static void libxl__remus_domain_suspend_callback(void *data)
domain_suspend_callback_common(egc, dss);
}
+static void remus_device_postsuspend_cb(libxl__egc *egc,
+ libxl__remus_state *rs, int rc)
+{
+ int ok = 0;
+ libxl__domain_suspend_state *dss = CONTAINER_OF(rs, *dss, rs);
+
+ if (!rc)
+ ok = 1;
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, ok);
+}
+
static void remus_domain_suspend_callback_common_done(libxl__egc *egc,
libxl__domain_suspend_state *dss, int ok)
{
- /* REMUS TODO: Issue disk and network checkpoint reqs. */
+ if (!ok)
+ goto out;
+
+ libxl__remus_state *const rs = &dss->rs;
+ rs->callback = remus_device_postsuspend_cb;
+ libxl__remus_device_postsuspend(egc, rs);
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, ok);
+}
+
+static void remus_device_preresume_cb(libxl__egc *egc,
+ libxl__remus_state *rs, int rc)
+{
+ int ok = 0;
+ libxl__domain_suspend_state *dss = CONTAINER_OF(rs, *dss, rs);
+ STATE_AO_GC(dss->ao);
+
+ if (!rc) {
+ /* Resumes the domain and the device model */
+ if (!libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1))
+ ok = 1;
+ }
libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, ok);
}
-static int libxl__remus_domain_resume_callback(void *data)
+static void libxl__remus_domain_resume_callback(void *data)
{
libxl__save_helper_state *shs = data;
libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__egc *egc = shs->egc;
STATE_AO_GC(dss->ao);
- /* Resumes the domain and the device model */
- if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1))
- return 0;
-
- /* REMUS TODO: Deal with disk. Start a new network output buffer */
- return 1;
+ libxl__remus_state *const rs = &dss->rs;
+ rs->callback = remus_device_preresume_cb;
+ libxl__remus_device_preresume(egc, rs);
}
/*----- remus asynchronous checkpoint callback -----*/
static void remus_checkpoint_dm_saved(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc);
+static void remus_next_checkpoint(libxl__egc *egc, libxl__ev_time *ev,
+ const struct timeval *requested_abs);
static void libxl__remus_domain_checkpoint_callback(void *data)
{
@@ -1485,13 +1519,67 @@ static void libxl__remus_domain_checkpoint_callback(void *data)
}
}
+static void remus_device_commit_cb(libxl__egc *egc,
+ libxl__remus_state *rs, int rc)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(rs, *dss, rs);
+
+ STATE_AO_GC(dss->ao);
+
+ if (rc) {
+ LOG(ERROR, "Failed to do device commit op."
+ " Terminating Remus..");
+ goto out;
+ } else {
+ /* Set checkpoint interval timeout */
+ rc = libxl__ev_time_register_rel(gc, &rs->timeout,
+ remus_next_checkpoint,
+ dss->interval);
+ if (rc) {
+ LOG(ERROR, "unable to register timeout for next epoch."
+ " Terminating Remus..");
+ goto out;
+ }
+ }
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 0);
+}
+
static void remus_checkpoint_dm_saved(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc)
{
- /* REMUS TODO: Wait for disk and memory ack, release network buffer */
- /* REMUS TODO: make this asynchronous */
- assert(!rc); /* REMUS TODO handle this error properly */
- usleep(dss->interval * 1000);
+ /* Convenience aliases */
+ libxl__remus_state *const rs = &dss->rs;
+
+ STATE_AO_GC(dss->ao);
+
+ if (rc) {
+ LOG(ERROR, "Failed to save device model. Terminating Remus..");
+ goto out;
+ }
+
+ rs->callback = remus_device_commit_cb;
+ libxl__remus_device_commit(egc, rs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 0);
+}
+
+static void remus_next_checkpoint(libxl__egc *egc, libxl__ev_time *ev,
+ const struct timeval *requested_abs)
+{
+ libxl__remus_state *rs = CONTAINER_OF(ev, *rs, timeout);
+
+ /* Convenience aliases */
+ libxl__domain_suspend_state *const dss = CONTAINER_OF(rs, *dss, rs);
+
+ STATE_AO_GC(dss->ao);
+
+ libxl__ev_time_deregister(gc, &rs->timeout);
libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 1);
}
@@ -1716,6 +1804,13 @@ static void save_device_model_datacopier_done(libxl__egc *egc,
dss->save_dm_callback(egc, dss, our_rc);
}
+static void libxl__remus_teardown_done(libxl__egc *egc,
+ libxl__remus_state *rs, int rc)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(rs, *dss, rs);
+ dss->callback(egc, dss, rc);
+}
+
static void domain_suspend_done(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc)
{
@@ -1730,6 +1825,19 @@ static void domain_suspend_done(libxl__egc *egc,
xc_suspend_evtchn_release(CTX->xch, CTX->xce, domid,
dss->guest_evtchn.port, &dss->guest_evtchn_lockfd);
+ if (dss->remus) {
+ /*
+ * With Remus, if we reach this point, it means either
+ * backup died or some network error occurred preventing us
+ * from sending checkpoints. Teardown the network buffers and
+ * release netlink resources. This is an async op.
+ */
+ dss->rs.saved_rc = rc;
+ dss->rs.callback = libxl__remus_teardown_done;
+ libxl__remus_device_teardown(egc, &dss->rs);
+ return;
+ }
+
dss->callback(egc, dss, rc);
}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index de0a807..2fdbafa 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2442,6 +2442,118 @@ typedef struct libxl__save_helper_state {
* marshalling and xc callback functions */
} libxl__save_helper_state;
+/*----- remus device related state structure -----*/
+
+typedef enum libxl__remus_device_kind {
+ LIBXL__REMUS_DEVICE_NIC,
+ LIBXL__REMUS_DEVICE_DISK,
+} libxl__remus_device_kind;
+
+typedef struct libxl__remus_state libxl__remus_state;
+typedef struct libxl__remus_device libxl__remus_device;
+typedef struct libxl__remus_device_state libxl__remus_device_state;
+typedef struct libxl__remus_device_ops libxl__remus_device_ops;
+
+struct libxl__remus_device_ops {
+ /*
+ * init device ops private data, etc. must implenment
+ */
+ int (*init)(libxl__remus_device_ops *self,
+ libxl__remus_state *rs);
+ /*
+ * free device ops private data, etc. must implenment
+ */
+ void (*destroy)(libxl__remus_device_ops *self);
+ /* device ops's private data */
+ void *data;
+
+ /*
+ * checkpoint callbacks, async ops. may not implemented
+ */
+ void (*postsuspend)(libxl__remus_device *dev);
+ void (*preresume)(libxl__remus_device *dev);
+ void (*commit)(libxl__remus_device *dev);
+
+ /*
+ * check whether device ops match the device, async op. must implement
+ */
+ void (*match)(libxl__remus_device_ops *self,
+ libxl__remus_device *dev);
+ /*
+ * setup the remus device, async op. must implement
+ */
+ void (*setup)(libxl__remus_device *dev);
+
+ /*
+ * teardown the remus device, async op. must implement
+ */
+ void (*teardown)(libxl__remus_device *dev);
+};
+
+struct libxl__remus_device_state {
+ libxl__ao *ao;
+ libxl__egc *egc;
+
+ /* devices that have been setuped */
+ libxl__remus_device **dev;
+
+ int num_nics;
+ int num_disks;
+
+ /* for counting devices that have been handled */
+ int num_devices;
+ /* for counting devices that matched and setuped */
+ int num_setuped;
+};
+
+typedef void libxl__remus_device_callback(libxl__egc *,
+ libxl__remus_device *,
+ int rc);
+
+struct libxl__remus_device {
+ int devid;
+ /* libxl__device_* which this remus device related to */
+ const void *backend_dev;
+ libxl__remus_device_kind kind;
+ int ops_index;
+ libxl__remus_device_ops *ops;
+ libxl__remus_device_callback *callback;
+
+ /* *kind* of device's private data */
+ void *data;
+ libxl__remus_device_state *rds;
+ /* for calling scripts */
+ libxl__async_exec_state aes;
+ /* for async func calls */
+ libxl__ev_child child;
+};
+
+typedef void libxl__remus_callback(libxl__egc *,
+ libxl__remus_state *, int rc);
+
+struct libxl__remus_state {
+ libxl__ao *ao;
+ uint32_t domid;
+ libxl__remus_callback *callback;
+
+ /* private */
+ int saved_rc;
+ /* context containing device related stuff */
+ libxl__remus_device_state dev_state;
+
+ libxl__ev_time timeout; /* used for checkpoint */
+};
+
+_hidden void libxl__remus_device_setup(libxl__egc *egc,
+ libxl__remus_state *rs);
+_hidden void libxl__remus_device_teardown(libxl__egc *egc,
+ libxl__remus_state *rs);
+_hidden void libxl__remus_device_postsuspend(libxl__egc *egc,
+ libxl__remus_state *rs);
+_hidden void libxl__remus_device_preresume(libxl__egc *egc,
+ libxl__remus_state *rs);
+_hidden void libxl__remus_device_commit(libxl__egc *egc,
+ libxl__remus_state *rs);
_hidden int libxl__netbuffer_enabled(libxl__gc *gc);
/*----- Domain suspend (save) state structure -----*/
@@ -2472,6 +2584,7 @@ struct libxl__domain_suspend_state {
int live;
int debug;
const libxl_domain_remus_info *remus;
+ libxl__remus_state rs;
/* private */
libxl__ev_evtchn guest_evtchn;
int guest_evtchn_lockfd;
diff --git a/tools/libxl/libxl_remus_device.c b/tools/libxl/libxl_remus_device.c
new file mode 100644
index 0000000..62c0614
--- /dev/null
+++ b/tools/libxl/libxl_remus_device.c
@@ -0,0 +1,303 @@
+/*
+ * Copyright (C) 2014
+ * Author: Lai Jiangshan <laijs@cn.fujitsu.com>
+ * Yang Hongyang <yanghy@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+static libxl__remus_device_ops *dev_ops[] = {
+};
+
+static void device_common_cb(libxl__egc *egc,
+ libxl__remus_device *dev,
+ int rc)
+{
+ /* Convenience aliases */
+ libxl__remus_device_state *const rds = dev->rds;
+ libxl__remus_state *const rs = CONTAINER_OF(rds, *rs, dev_state);
+
+ STATE_AO_GC(rs->ao);
+
+ rds->num_devices++;
+
+ if (rc)
+ rs->saved_rc = ERROR_FAIL;
+
+ if (rds->num_devices == rds->num_setuped)
+ rs->callback(egc, rs, rs->saved_rc);
+}
+
+void libxl__remus_device_postsuspend(libxl__egc *egc, libxl__remus_state *rs)
+{
+ int i;
+ libxl__remus_device *dev;
+ STATE_AO_GC(rs->ao);
+
+ /* Convenience aliases */
+ libxl__remus_device_state *const rds = &rs->dev_state;
+
+ rds->num_devices = 0;
+ for (i = 0; i < rds->num_setuped; i++) {
+ dev = rds->dev[i];
+ dev->callback = device_common_cb;
+ if (dev->ops->postsuspend) {
+ dev->ops->postsuspend(dev);
+ } else {
+ rds->num_devices++;
+ }
+ }
+
+ if (rds->num_devices == rds->num_setuped)
+ rs->callback(egc, rs, rs->saved_rc);
+}
+
+void libxl__remus_device_preresume(libxl__egc *egc, libxl__remus_state *rs)
+{
+ int i;
+ libxl__remus_device *dev;
+ STATE_AO_GC(rs->ao);
+
+ /* Convenience aliases */
+ libxl__remus_device_state *const rds = &rs->dev_state;
+
+ rds->num_devices = 0;
+ for (i = 0; i < rds->num_setuped; i++) {
+ dev = rds->dev[i];
+ dev->callback = device_common_cb;
+ if (dev->ops->preresume) {
+ dev->ops->preresume(dev);
+ } else {
+ rds->num_devices++;
+ }
+ }
+
+ if (rds->num_devices == rds->num_setuped)
+ rs->callback(egc, rs, rs->saved_rc);
+}
+
+void libxl__remus_device_commit(libxl__egc *egc, libxl__remus_state *rs)
+{
+ int i;
+ libxl__remus_device *dev;
+ STATE_AO_GC(rs->ao);
+
+ /*
+ * REMUS TODO: Wait for disk and explicit memory ack (through restore
+ * callback from remote) before releasing network buffer.
+ */
+ /* Convenience aliases */
+ libxl__remus_device_state *const rds = &rs->dev_state;
+
+ rds->num_devices = 0;
+ for (i = 0; i < rds->num_setuped; i++) {
+ dev = rds->dev[i];
+ dev->callback = device_common_cb;
+ if (dev->ops->commit) {
+ dev->ops->commit(dev);
+ } else {
+ rds->num_devices++;
+ }
+ }
+
+ if (rds->num_devices == rds->num_setuped)
+ rs->callback(egc, rs, rs->saved_rc);
+}
+
+static void device_setup_cb(libxl__egc *egc,
+ libxl__remus_device *dev,
+ int rc)
+{
+ /* Convenience aliases */
+ libxl__remus_device_state *const rds = dev->rds;
+ libxl__remus_state *const rs = CONTAINER_OF(rds, *rs, dev_state);
+
+ STATE_AO_GC(rs->ao);
+
+ rds->num_devices++;
+ if (!rc) {
+ /* remus device has been setuped */
+ rds->dev[rds->num_setuped++] = dev;
+ } else {
+ /* setup failed */
+ rs->saved_rc = ERROR_FAIL;
+ }
+
+ if (rds->num_devices == (rds->num_nics + rds->num_disks))
+ rs->callback(egc, rs, rs->saved_rc);
+}
+
+static void device_match_cb(libxl__egc *egc,
+ libxl__remus_device *dev,
+ int rc)
+{
+ libxl__remus_device_state *const rds = dev->rds;
+ libxl__remus_state *rs = CONTAINER_OF(rds, *rs, dev_state);
+
+ STATE_AO_GC(rs->ao);
+
+ if (rc) {
+ if (++dev->ops_index >= ARRAY_SIZE(dev_ops) ||
+ rc != ERROR_NOT_MATCH) {
+ /* the device can not be matched */
+ rds->num_devices++;
+ rs->saved_rc = ERROR_FAIL;
+ goto out;
+ }
+ /* the ops does not match, try next ops */
+ dev->ops = dev_ops[dev->ops_index];
+ dev->ops->match(dev->ops, dev);
+ } else {
+ /* the ops matched, setup the device */
+ dev->callback = device_setup_cb;
+ dev->ops->setup(dev);
+ }
+
+out:
+ if (rds->num_devices == (rds->num_nics + rds->num_disks))
+ rs->callback(egc, rs, rs->saved_rc);
+}
+
+static void device_teardown_cb(libxl__egc *egc,
+ libxl__remus_device *dev,
+ int rc)
+{
+ int i;
+ libxl__remus_device_ops *ops;
+ libxl__remus_device_state *const rds = dev->rds;
+ libxl__remus_state *rs = CONTAINER_OF(rds, *rs, dev_state);
+
+ STATE_AO_GC(rs->ao);
+
+ /* ignore teardown errors to teardown as many devs as possible*/
+ rds->num_setuped--;
+
+ if (rds->num_setuped == 0) {
+ /* clean device ops */
+ for (i = 0; i < ARRAY_SIZE(dev_ops); i++) {
+ ops = dev_ops[i];
+ ops->destroy(ops);
+ }
+ rs->callback(egc, rs, rs->saved_rc);
+ }
+}
+
+static __attribute__((unused)) void libxl__remus_device_init(libxl__egc *egc,
+ libxl__remus_device_state *rds,
+ libxl__remus_device_kind kind,
+ void *libxl_dev)
+{
+ libxl__remus_device *dev = NULL;
+ libxl_device_nic *nic = NULL;
+ libxl_device_disk *disk = NULL;
+
+ STATE_AO_GC(rds->ao);
+ GCNEW(dev);
+ dev->ops_index = 0; /* we will match the ops later */
+ dev->backend_dev = libxl_dev;
+ dev->kind = kind;
+ dev->rds = rds;
+
+ switch (kind) {
+ case LIBXL__REMUS_DEVICE_NIC:
+ nic = libxl_dev;
+ dev->devid = nic->devid;
+ break;
+ case LIBXL__REMUS_DEVICE_DISK:
+ disk = libxl_dev;
+ /* there are no dev id for disk devices */
+ dev->devid = -1;
+ break;
+ default:
+ return;
+ }
+
+ libxl__async_exec_init(&dev->aes);
+ libxl__ev_child_init(&dev->child);
+
+ /* match the ops begin */
+ dev->callback = device_match_cb;
+ dev->ops = dev_ops[dev->ops_index];
+ dev->ops->match(dev->ops, dev);
+}
+
+void libxl__remus_device_setup(libxl__egc *egc, libxl__remus_state *rs)
+{
+ int i;
+ libxl__remus_device_ops *ops;
+
+ /* Convenience aliases */
+ libxl__remus_device_state *const rds = &rs->dev_state;
+
+ STATE_AO_GC(rs->ao);
+
+ if (ARRAY_SIZE(dev_ops) == 0)
+ goto out;
+
+ for (i = 0; i < ARRAY_SIZE(dev_ops); i++) {
+ ops = dev_ops[i];
+ if (ops->init(ops, rs)) {
+ rs->saved_rc = ERROR_FAIL;
+ goto out;
+ }
+ }
+
+ rds->ao = rs->ao;
+ rds->egc = egc;
+ rds->num_devices = 0;
+ rds->num_nics = 0;
+ rds->num_disks = 0;
+
+ /* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */
+
+ GCNEW_ARRAY(rds->dev, rds->num_nics + rds->num_disks);
+
+ /* TBD: CALL libxl__remus_device_init to init remus devices */
+
+ if (rds->num_nics == 0 && rds->num_disks == 0)
+ goto out;
+
+ return;
+
+out:
+ rs->callback(egc, rs, rs->saved_rc);
+ return;
+}
+
+void libxl__remus_device_teardown(libxl__egc *egc, libxl__remus_state *rs)
+{
+ int i;
+ libxl__remus_device *dev;
+
+ STATE_AO_GC(rs->ao);
+
+ /* Convenience aliases */
+ libxl__remus_device_state *const rds = &rs->dev_state;
+
+ if (rds->num_setuped == 0)
+ goto out;
+
+ for (i = 0; i < rds->num_setuped; i++) {
+ dev = rds->dev[i];
+ dev->callback = device_teardown_cb;
+ dev->ops->teardown(dev);
+ }
+
+ return;
+
+out:
+ rs->callback(egc, rs, rs->saved_rc);
+ return;
+}
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 745e2ac..36bae04 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -24,7 +24,7 @@ our @msgs = (
'unsigned long', 'done',
'unsigned long', 'total'] ],
[ 3, 'scxA', "suspend", [] ],
- [ 4, 'scxW', "postcopy", [] ],
+ [ 4, 'scxA', "postcopy", [] ],
[ 5, 'scxA', "checkpoint", [] ],
[ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 8944686..c757f17 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -43,6 +43,7 @@ libxl_error = Enumeration("error", [
(-12, "OSEVENT_REG_FAIL"),
(-13, "BUFFERFULL"),
(-14, "UNKNOWN_CHILD"),
+ (-15, "NOT_MATCH"),
], value_namespace = "")
libxl_domain_type = Enumeration("domain_type", [
--
1.9.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v10 4/5] remus: implement remus network buffering for nic devices
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
` (2 preceding siblings ...)
2014-05-21 10:14 ` [PATCH v10 3/5] remus: introduce remus device Yang Hongyang
@ 2014-05-21 10:14 ` Yang Hongyang
2014-05-21 10:15 ` [PATCH v10 5/5] libxl: network buffering cmdline switch Yang Hongyang
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Yang Hongyang @ 2014-05-21 10:14 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, eddie.dong, rshriram, roger.pau,
laijs
1.Add two members in libxl_domain_remus_info:
netbuf: whether netbuf is enabled
netbufscript: the path of the script which will be run to setup
and tear down the guest's interface.
2.introduces remus-netbuf-setup hotplug script responsible for
setting up and tearing down the necessary infrastructure required for
network output buffering in Remus. This script is intended to be invoked
by libxl for each guest interface, when starting or stopping Remus.
Apart from returning success/failure indication via the usual hotplug
entries in xenstore, this script also writes to xenstore, the name of
the IFB device to be used to control the vif's network output.
The script relies on libnl3 command line utilities to perform various
setup/teardown functions. The script is confined to Linux platforms only
since NetBSD does not seem to have libnl3.
The following steps are taken during init:
a) establish a dedicated remus context containing libnl related
state (netlink sockets, qdisc caches, etc.,)
The following steps are taken for each vif during setup:
a) call the hotplug script to setup its network buffer
b) Obtain handles to plug qdiscs installed on the IFB devices
chosen by the hotplug scripts.
And during teardown, the netlink resources are released, followed by
invocation of hotplug scripts to remove the ifb devices.
3.implement the remus device interface. setup, teardown, etc.
Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Wen Congyang <wency@cn.fujitsu.com>
---
docs/misc/xenstore-paths.markdown | 4 +
tools/hotplug/Linux/Makefile | 1 +
tools/hotplug/Linux/remus-netbuf-setup | 183 ++++++++++++
tools/libxl/libxl.c | 18 ++
tools/libxl/libxl.h | 13 +
tools/libxl/libxl_internal.h | 3 +
tools/libxl/libxl_netbuffer.c | 519 +++++++++++++++++++++++++++++++++
tools/libxl/libxl_nonetbuffer.c | 67 +++++
tools/libxl/libxl_remus_device.c | 22 +-
tools/libxl/libxl_types.idl | 2 +
10 files changed, 831 insertions(+), 1 deletion(-)
create mode 100644 tools/hotplug/Linux/remus-netbuf-setup
diff --git a/docs/misc/xenstore-paths.markdown b/docs/misc/xenstore-paths.markdown
index 70ab7f4..039eaea 100644
--- a/docs/misc/xenstore-paths.markdown
+++ b/docs/misc/xenstore-paths.markdown
@@ -385,6 +385,10 @@ The guest's virtual time offset from UTC in seconds.
The device model version for a domain.
+#### /libxl/$DOMID/remus/netbuf/$DEVID/ifb = STRING [n,INTERNAL]
+
+ifb device used by Remus to buffer network output from the associated vif.
+
[BLKIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,blkif.h.html
[FBIF]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,io,fbif.h.html
[HVMPARAMS]: http://xenbits.xen.org/docs/unstable/hypercall/include,public,hvm,params.h.html
diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile
index 4874ec5..13e1f5f 100644
--- a/tools/hotplug/Linux/Makefile
+++ b/tools/hotplug/Linux/Makefile
@@ -15,6 +15,7 @@ XEN_SCRIPTS += vif-nat
XEN_SCRIPTS += vif-openvswitch
XEN_SCRIPTS += vif2
XEN_SCRIPTS += vif-setup
+XEN_SCRIPTS-$(CONFIG_REMUS_NETBUF) += remus-netbuf-setup
XEN_SCRIPTS += block
XEN_SCRIPTS += block-enbd block-nbd
XEN_SCRIPTS-$(CONFIG_BLKTAP1) += blktap
diff --git a/tools/hotplug/Linux/remus-netbuf-setup b/tools/hotplug/Linux/remus-netbuf-setup
new file mode 100644
index 0000000..aed2583
--- /dev/null
+++ b/tools/hotplug/Linux/remus-netbuf-setup
@@ -0,0 +1,183 @@
+#!/bin/bash
+#============================================================================
+# ${XEN_SCRIPT_DIR}/remus-netbuf-setup
+#
+# Script for attaching a network buffer to the specified vif (in any mode).
+# The hotplugging system will call this script when starting remus via libxl
+# API, libxl_domain_remus_start.
+#
+# Usage:
+# remus-netbuf-setup (setup|teardown)
+#
+# Environment vars:
+# vifname vif interface name (required).
+# XENBUS_PATH path in Xenstore, where the IFB device details will be stored
+# or read from (required).
+# (libxl passes /libxl/<domid>/remus/netbuf/<devid>)
+# IFB ifb interface to be cleaned up (required). [for teardown op only]
+
+# Written to the store: (setup operation)
+# XENBUS_PATH/ifb=<ifbdevName> the IFB device serving
+# as the intermediate buffer through which the interface's network output
+# can be controlled.
+#
+# To install a network buffer on a guest vif (vif1.0) using ifb (ifb0)
+# we need to do the following
+#
+# ip link set dev ifb0 up
+# tc qdisc add dev vif1.0 ingress
+# tc filter add dev vif1.0 parent ffff: proto ip \
+# prio 10 u32 match u32 0 0 action mirred egress redirect dev ifb0
+# nl-qdisc-add --dev=ifb0 --parent root plug
+# nl-qdisc-add --dev=ifb0 --parent root --update plug --limit=10000000
+# (10MB limit on buffer)
+#
+# So order of operations when installing a network buffer on vif1.0
+# 1. find a free ifb and bring up the device
+# 2. redirect traffic from vif1.0 to ifb:
+# 2.1 add ingress qdisc to vif1.0 (to capture outgoing packets from guest)
+# 2.2 use tc filter command with actions mirred egress + redirect
+# 3. install plug_qdisc on ifb device, with which we can buffer/release
+# guest's network output from vif1.0
+#
+#
+
+#============================================================================
+
+# Unlike other vif scripts, vif-common is not needed here as it executes vif
+#specific setup code such as renaming.
+dir=$(dirname "$0")
+. "$dir/xen-hotplug-common.sh"
+
+findCommand "$@"
+
+if [ "$command" != "setup" -a "$command" != "teardown" ]
+then
+ echo "Invalid command: $command"
+ log err "Invalid command: $command"
+ exit 1
+fi
+
+evalVariables "$@"
+
+: ${vifname:?}
+: ${XENBUS_PATH:?}
+
+check_libnl_tools() {
+ if ! command -v nl-qdisc-list > /dev/null 2>&1; then
+ fatal "Unable to find nl-qdisc-list tool"
+ fi
+ if ! command -v nl-qdisc-add > /dev/null 2>&1; then
+ fatal "Unable to find nl-qdisc-add tool"
+ fi
+ if ! command -v nl-qdisc-delete > /dev/null 2>&1; then
+ fatal "Unable to find nl-qdisc-delete tool"
+ fi
+}
+
+# We only check for modules. We don't load them.
+# User/Admin is supposed to load ifb during boot time,
+# ensuring that there are enough free ifbs in the system.
+# Other modules will be loaded automatically by tc commands.
+check_modules() {
+ for m in ifb sch_plug sch_ingress act_mirred cls_u32
+ do
+ if ! modinfo $m > /dev/null 2>&1; then
+ fatal "Unable to find $m kernel module"
+ fi
+ done
+}
+
+setup_ifb() {
+
+ for ifb in `ifconfig -a -s|egrep ^ifb|cut -d ' ' -f1`
+ do
+ local installed=`nl-qdisc-list -d $ifb`
+ [ -n "$installed" ] && continue
+ IFB="$ifb"
+ break
+ done
+
+ if [ -z "$IFB" ]
+ then
+ fatal "Unable to find a free IFB device for $vifname"
+ fi
+
+ do_or_die ip link set dev "$IFB" up
+}
+
+redirect_vif_traffic() {
+ local vif=$1
+ local ifb=$2
+
+ do_or_die tc qdisc add dev "$vif" ingress
+
+ tc filter add dev "$vif" parent ffff: proto ip prio 10 \
+ u32 match u32 0 0 action mirred egress redirect dev "$ifb" >/dev/null 2>&1
+
+ if [ $? -ne 0 ]
+ then
+ do_without_error tc qdisc del dev "$vif" ingress
+ fatal "Failed to redirect traffic from $vif to $ifb"
+ fi
+}
+
+add_plug_qdisc() {
+ local vif=$1
+ local ifb=$2
+
+ nl-qdisc-add --dev="$ifb" --parent root plug >/dev/null 2>&1
+ if [ $? -ne 0 ]
+ then
+ do_without_error tc qdisc del dev "$vif" ingress
+ fatal "Failed to add plug qdisc to $ifb"
+ fi
+
+ #set ifb buffering limit in bytes. Its okay if this command fails
+ nl-qdisc-add --dev="$ifb" --parent root \
+ --update plug --limit=10000000 >/dev/null 2>&1 || true
+}
+
+teardown_netbuf() {
+ local vif=$1
+ local ifb=$2
+
+ if [ "$ifb" ]; then
+ do_without_error ip link set dev "$ifb" down
+ do_without_error nl-qdisc-delete --dev="$ifb" --parent root plug >/dev/null 2>&1
+ xenstore-rm -t "$XENBUS_PATH/ifb" 2>/dev/null || true
+ fi
+ do_without_error tc qdisc del dev "$vif" ingress
+ xenstore-rm -t "$XENBUS_PATH/hotplug-status" 2>/dev/null || true
+ xenstore-rm -t "$XENBUS_PATH/hotplug-error" 2>/dev/null || true
+}
+
+xs_write_failed() {
+ local vif=$1
+ local ifb=$2
+ teardown_netbuf "$vifname" "$IFB"
+ fatal "failed to write ifb name to xenstore"
+}
+
+case "$command" in
+ setup)
+ check_libnl_tools
+ check_modules
+
+ claim_lock "pickifb"
+ setup_ifb
+ redirect_vif_traffic "$vifname" "$IFB"
+ add_plug_qdisc "$vifname" "$IFB"
+ release_lock "pickifb"
+
+ #not using xenstore_write that automatically exits on error
+ #because we need to cleanup
+ _xenstore_write "$XENBUS_PATH/ifb" "$IFB" || xs_write_failed "$vifname" "$IFB"
+ success
+ ;;
+ teardown)
+ teardown_netbuf "$vifname" "$IFB"
+ ;;
+esac
+
+log debug "Successful remus-netbuf-setup $command for $vifname, ifb $IFB."
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 2e7a6ea..4e7245f 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -768,6 +768,24 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
/* Convenience aliases */
libxl__remus_state *const rs = &dss->rs;
+
+ /* Setup network buffering */
+ if (info->netbuf) {
+ if (!libxl__netbuffer_enabled(gc)) {
+ LOG(ERROR, "Remus: No support for network buffering");
+ goto out;
+ }
+
+ if (info->netbufscript) {
+ rs->netbufscript =
+ libxl__strdup(gc, info->netbufscript);
+ } else {
+ rs->netbufscript =
+ GCSPRINTF("%s/remus-netbuf-setup",
+ libxl__xen_script_dir_path());
+ }
+ }
+
rs->ao = ao;
rs->domid = domid;
rs->saved_rc = 0;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 84f9c0e..34af808 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -417,6 +417,19 @@
#define LIBXL_HAVE_DRIVER_DOMAIN_CREATION 1
/*
+ * LIBXL_HAVE_REMUS_NETBUF 1
+ *
+ * If this is defined, then the libxl_domain_remus_info structure will
+ * have a boolean field (netbuf) and a string field (netbufscript).
+ *
+ * netbuf, if true, indicates that network buffering should be enabled.
+ *
+ * netbufscript, if set, indicates the path to the hotplug script to
+ * setup or teardown network buffers.
+ */
+#define LIBXL_HAVE_REMUS_NETBUF 1
+
+/*
* LIBXL_HAVE_SIGCHLD_SELECTIVE_REAP
*
* If this is defined:
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 2fdbafa..85fa713 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2497,6 +2497,7 @@ struct libxl__remus_device_state {
/* devices that have been setuped */
libxl__remus_device **dev;
+ libxl_device_nic *nics;
int num_nics;
int num_disks;
@@ -2535,6 +2536,8 @@ struct libxl__remus_state {
libxl__ao *ao;
uint32_t domid;
libxl__remus_callback *callback;
+ /* Script to setup/teardown network buffers */
+ const char *netbufscript;
/* private */
int saved_rc;
diff --git a/tools/libxl/libxl_netbuffer.c b/tools/libxl/libxl_netbuffer.c
index 8e23d75..8729a3f 100644
--- a/tools/libxl/libxl_netbuffer.c
+++ b/tools/libxl/libxl_netbuffer.c
@@ -17,11 +17,530 @@
#include "libxl_internal.h"
+#include <netlink/cache.h>
+#include <netlink/socket.h>
+#include <netlink/attr.h>
+#include <netlink/route/link.h>
+#include <netlink/route/route.h>
+#include <netlink/route/qdisc.h>
+#include <netlink/route/qdisc/plug.h>
+
+typedef struct libxl__remus_netbuf_state {
+ libxl__ao *ao;
+ uint32_t domid;
+ const char *netbufscript;
+
+ struct nl_sock *nlsock;
+ struct nl_cache *qdisc_cache;
+} libxl__remus_netbuf_state;
+
+typedef struct libxl__remus_device_nic {
+ const char *vif;
+ const char *ifb;
+ struct rtnl_qdisc *qdisc;
+} libxl__remus_device_nic;
+
int libxl__netbuffer_enabled(libxl__gc *gc)
{
return 1;
}
+/* If the device has a vifname, then use that instead of
+ * the vifX.Y format.
+ */
+static const char *get_vifname(libxl__remus_device *dev,
+ const libxl_device_nic *nic)
+{
+ libxl__remus_netbuf_state *netbuf_state = dev->ops->data;
+ const char *vifname = NULL;
+ const char *path;
+ int rc;
+
+ STATE_AO_GC(netbuf_state->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = netbuf_state->domid;
+
+ path = libxl__sprintf(gc, "%s/backend/vif/%d/%d/vifname",
+ libxl__xs_get_dompath(gc, 0), domid, nic->devid);
+ rc = libxl__xs_read_checked(gc, XBT_NULL, path, &vifname);
+ if (!rc && !vifname) {
+ /* use the default name */
+ vifname = libxl__device_nic_devname(gc, domid,
+ nic->devid,
+ nic->nictype);
+ }
+
+ return vifname;
+}
+
+static void free_qdisc(libxl__remus_device_nic *remus_nic)
+{
+ /* free qdiscs */
+ if (remus_nic->qdisc == NULL)
+ return;
+
+ nl_object_put((struct nl_object *)(remus_nic->qdisc));
+ remus_nic->qdisc = NULL;
+}
+
+static int init_qdisc(libxl__remus_netbuf_state *netbuf_state,
+ libxl__remus_device_nic *remus_nic)
+{
+ int ret, ifindex;
+ struct rtnl_link *ifb = NULL;
+ struct rtnl_qdisc *qdisc = NULL;
+
+ STATE_AO_GC(netbuf_state->ao);
+
+ /* Now that we have brought up IFB device with plug qdisc for
+ * this vif, so we need to refill the qdisc cache.
+ */
+ ret = nl_cache_refill(netbuf_state->nlsock, netbuf_state->qdisc_cache);
+ if (ret < 0) {
+ LOG(ERROR, "cannot refill qdisc cache");
+ goto out;
+ }
+
+ /* get a handle to the IFB interface */
+ ifb = NULL;
+ ret = rtnl_link_get_kernel(netbuf_state->nlsock, 0,
+ remus_nic->ifb, &ifb);
+ if (ret) {
+ LOG(ERROR, "cannot obtain handle for %s: %s", remus_nic->ifb,
+ nl_geterror(ret));
+ ret = ERROR_FAIL;
+ goto out;
+ }
+
+ ret = ERROR_FAIL;
+ ifindex = rtnl_link_get_ifindex(ifb);
+ if (!ifindex) {
+ LOG(ERROR, "interface %s has no index", remus_nic->ifb);
+ goto out;
+ }
+
+ /* Get a reference to the root qdisc installed on the IFB, by
+ * querying the qdisc list we obtained earlier. The netbufscript
+ * sets up the plug qdisc as the root qdisc, so we don't have to
+ * search the entire qdisc tree on the IFB dev.
+
+ * There is no need to explicitly free this qdisc as its just a
+ * reference from the qdisc cache we allocated earlier.
+ */
+ qdisc = rtnl_qdisc_get_by_parent(netbuf_state->qdisc_cache, ifindex,
+ TC_H_ROOT);
+
+ if (qdisc) {
+ const char *tc_kind = rtnl_tc_get_kind(TC_CAST(qdisc));
+ /* Sanity check: Ensure that the root qdisc is a plug qdisc. */
+ if (!tc_kind || strcmp(tc_kind, "plug")) {
+ nl_object_put((struct nl_object *)qdisc);
+ LOG(ERROR, "plug qdisc is not installed on %s", remus_nic->ifb);
+ goto out;
+ }
+ remus_nic->qdisc = qdisc;
+ ret = 0;
+ } else {
+ LOG(ERROR, "Cannot get qdisc handle from ifb %s", remus_nic->ifb);
+ }
+
+out:
+ if (ifb)
+ rtnl_link_put(ifb);
+
+ return ret;
+}
+
+/*
+ * In return, the script writes the name of IFB device (during setup) to be
+ * used for output buffering into XENBUS_PATH/ifb
+ */
+static void netbuf_setup_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(aes, *dev, aes);
+ libxl__remus_device_nic *remus_nic = dev->data;
+ libxl__remus_netbuf_state *netbuf_state = dev->ops->data;
+ const char *out_path_base, *hotplug_error = NULL;
+ int rc;
+
+ /* Convenience aliases */
+ const uint32_t domid = netbuf_state->domid;
+ const int devid = dev->devid;
+ const char *const vif = remus_nic->vif;
+ const char **const ifb = &remus_nic->ifb;
+
+ STATE_AO_GC(netbuf_state->ao);
+
+ if (status) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ out_path_base = GCSPRINTF("%s/remus/netbuf/%d",
+ libxl__xs_libxl_path(gc, domid), devid);
+
+ rc = libxl__xs_read_checked(gc, XBT_NULL,
+ GCSPRINTF("%s/hotplug-error", out_path_base),
+ &hotplug_error);
+ if (rc) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (hotplug_error) {
+ LOG(ERROR, "netbuf script %s setup failed for vif %s: %s",
+ netbuf_state->netbufscript, vif, hotplug_error);
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ rc = libxl__xs_read_checked(gc, XBT_NULL,
+ GCSPRINTF("%s/remus/netbuf/%d/ifb",
+ libxl__xs_libxl_path(gc, domid),
+ devid),
+ ifb);
+ if (rc) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (!(*ifb)) {
+ LOG(ERROR, "Cannot get ifb dev name for domain %u dev %s",
+ domid, vif);
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ LOG(DEBUG, "%s will buffer packets from vif %s", *ifb, vif);
+ rc = init_qdisc(netbuf_state, remus_nic);
+
+out:
+ dev->callback(egc, dev, rc);
+}
+
+static void netbuf_teardown_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int status)
+{
+ int rc;
+ libxl__remus_device *dev = CONTAINER_OF(aes, *dev, aes);
+ libxl__remus_device_nic *remus_nic = dev->data;
+
+ if (status)
+ rc = ERROR_FAIL;
+ else
+ rc = 0;
+
+ free_qdisc(remus_nic);
+
+ dev->callback(egc, dev, rc);
+}
+
+/* the script needs the following env & args
+ * $vifname
+ * $XENBUS_PATH (/libxl/<domid>/remus/netbuf/<devid>/)
+ * $IFB (for teardown)
+ * setup/teardown as command line arg.
+ */
+static void setup_async_exec(libxl__async_exec_state *aes,
+ char *op, libxl__remus_device *dev)
+{
+ int arraysize, nr = 0;
+ char **env = NULL, **args = NULL;
+ libxl__remus_device_nic *remus_nic = dev->data;
+ libxl__remus_netbuf_state *ns = dev->ops->data;
+ STATE_AO_GC(ns->ao);
+
+ /* Convenience aliases */
+ char *const script = libxl__strdup(gc, ns->netbufscript);
+ const uint32_t domid = ns->domid;
+ const int dev_id = dev->devid;
+ const char *const vif = remus_nic->vif;
+ const char *const ifb = remus_nic->ifb;
+
+ arraysize = 7;
+ GCNEW_ARRAY(env, arraysize);
+ env[nr++] = "vifname";
+ env[nr++] = libxl__strdup(gc, vif);
+ env[nr++] = "XENBUS_PATH";
+ env[nr++] = GCSPRINTF("%s/remus/netbuf/%d",
+ libxl__xs_libxl_path(gc, domid), dev_id);
+ if (!strcmp(op, "teardown") && ifb) {
+ env[nr++] = "IFB";
+ env[nr++] = libxl__strdup(gc, ifb);
+ }
+ env[nr++] = NULL;
+ assert(nr <= arraysize);
+
+ arraysize = 3; nr = 0;
+ GCNEW_ARRAY(args, arraysize);
+ args[nr++] = script;
+ args[nr++] = op;
+ args[nr++] = NULL;
+ assert(nr == arraysize);
+
+ aes->ao = ns->ao;
+ aes->what = GCSPRINTF("%s %s", args[0], args[1]);
+ aes->env = env;
+ aes->args = args;
+ aes->timeout_ms = LIBXL_HOTPLUG_TIMEOUT * 1000;
+ aes->stdfds[0] = -1;
+ aes->stdfds[1] = -1;
+ aes->stdfds[2] = -1;
+
+ if (!strcmp(op, "teardown"))
+ aes->callback = netbuf_teardown_script_cb;
+ else
+ aes->callback = netbuf_setup_script_cb;
+}
+
+static int nic_init(libxl__remus_device_ops *self,
+ libxl__remus_state *rs)
+{
+ int rc;
+ libxl__remus_netbuf_state *ns;
+
+ STATE_AO_GC(rs->ao);
+
+ GCNEW(ns);
+ self->data = ns;
+
+ ns->nlsock = nl_socket_alloc();
+ if (!ns->nlsock) {
+ LOG(ERROR, "cannot allocate nl socket");
+ goto out;
+ }
+
+ rc = nl_connect(ns->nlsock, NETLINK_ROUTE);
+ if (rc) {
+ LOG(ERROR, "failed to open netlink socket: %s",
+ nl_geterror(rc));
+ goto out;
+ }
+
+ /* get list of all qdiscs installed on network devs. */
+ rc = rtnl_qdisc_alloc_cache(ns->nlsock, &ns->qdisc_cache);
+ if (rc) {
+ LOG(ERROR, "failed to allocate qdisc cache: %s",
+ nl_geterror(rc));
+ goto out;
+ }
+
+ ns->ao = rs->ao;
+ ns->domid = rs->domid;
+ ns->netbufscript = rs->netbufscript;
+
+ return 0;
+
+out:
+ return ERROR_FAIL;
+}
+
+static void nic_destroy(libxl__remus_device_ops *self)
+{
+ libxl__remus_netbuf_state *ns = self->data;
+
+ if (!self->data)
+ return;
+
+ /* free qdisc cache */
+ if (ns->qdisc_cache) {
+ nl_cache_clear(ns->qdisc_cache);
+ nl_cache_free(ns->qdisc_cache);
+ ns->qdisc_cache = NULL;
+ }
+
+ /* close & free nlsock */
+ if (ns->nlsock) {
+ nl_close(ns->nlsock);
+ nl_socket_free(ns->nlsock);
+ ns->nlsock = NULL;
+ }
+}
+
+static void async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ pid_t pid, int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(child, *dev, child);
+ libxl__remus_device_state *rds = dev->rds;
+ STATE_AO_GC(rds->ao);
+
+ if (WIFEXITED(status)) {
+ dev->callback(egc, dev, -WEXITSTATUS(status));
+ } else {
+ dev->callback(egc, dev, ERROR_FAIL);
+ }
+}
+
+static void nic_match_async(const libxl__remus_device_ops *self,
+ libxl__remus_device *dev)
+{
+ if (dev->kind == LIBXL__REMUS_DEVICE_NIC)
+ _exit(0);
+
+ _exit(-ERROR_NOT_MATCH);
+}
+
+static void nic_match(libxl__remus_device_ops *self,
+ libxl__remus_device *dev)
+{
+ int pid = -1;
+ STATE_AO_GC(dev->rds->ao);
+
+ /* Fork and call */
+ pid = libxl__ev_child_fork(gc, &dev->child, async_call_done);
+ if (pid == -1) {
+ LOG(ERROR, "unable to fork");
+ goto out;
+ }
+
+ if (!pid) {
+ /* child */
+ nic_match_async(self, dev);
+ /* notreached */
+ abort();
+ }
+
+ return;
+
+out:
+ dev->callback(dev->rds->egc, dev, ERROR_FAIL);
+}
+
+static void nic_setup(libxl__remus_device *dev)
+{
+ libxl__remus_device_nic *remus_nic;
+ libxl__remus_netbuf_state *ns = dev->ops->data;
+ const libxl_device_nic *nic = dev->backend_dev;
+
+ STATE_AO_GC(ns->ao);
+
+ GCNEW(remus_nic);
+ dev->data = remus_nic;
+ remus_nic->vif = get_vifname(dev, nic);
+
+ setup_async_exec(&dev->aes, "setup", dev);
+ if (libxl__async_exec_start(gc, &dev->aes)) {
+ goto out;
+ }
+
+ return;
+
+out:
+ dev->callback(dev->rds->egc, dev, ERROR_FAIL);
+}
+
+/*
+ * Note: This function will be called in the same gc context as
+ * libxl__remus_netbuf_setup, created during the libxl_domain_remus_start
+ * API call.
+ */
+static void nic_teardown(libxl__remus_device *dev)
+{
+ libxl__remus_netbuf_state *ns = dev->ops->data;
+
+ STATE_AO_GC(ns->ao);
+
+ setup_async_exec(&dev->aes, "teardown", dev);
+
+ if (libxl__async_exec_start(gc, &dev->aes)) {
+ goto out;
+ }
+
+ return;
+
+out:
+ dev->callback(dev->rds->egc, dev, ERROR_FAIL);
+}
+
+/* The buffer_op's value, not the value passed to kernel */
+enum {
+ tc_buffer_start,
+ tc_buffer_release
+};
+
+static void remus_netbuf_op_async(libxl__remus_device_nic *remus_nic,
+ libxl__remus_netbuf_state *netbuf_state,
+ int buffer_op)
+{
+ int ret;
+
+ STATE_AO_GC(netbuf_state->ao);
+
+ if (buffer_op == tc_buffer_start)
+ ret = rtnl_qdisc_plug_buffer(remus_nic->qdisc);
+ else
+ ret = rtnl_qdisc_plug_release_one(remus_nic->qdisc);
+
+ if (!ret) {
+ ret = rtnl_qdisc_add(netbuf_state->nlsock,
+ remus_nic->qdisc,
+ NLM_F_REQUEST);
+ if (ret)
+ goto out;
+ }
+
+ _exit(0);
+
+out:
+ LOG(ERROR, "Remus: cannot do netbuf op %s on %s:%s",
+ ((buffer_op == tc_buffer_start) ?
+ "start_new_epoch" : "release_prev_epoch"),
+ remus_nic->ifb, nl_geterror(ret));
+ _exit(-ERROR_FAIL);
+}
+
+static void netbuf_epoch_op(libxl__remus_device *dev, int buffer_op)
+{
+ int pid = -1;
+ libxl__remus_device_nic *remus_nic = dev->data;
+ libxl__remus_netbuf_state *ns = dev->ops->data;
+ STATE_AO_GC(dev->rds->ao);
+
+ /* Fork and call */
+ pid = libxl__ev_child_fork(gc, &dev->child, async_call_done);
+ if (pid == -1) {
+ LOG(ERROR, "unable to fork");
+ goto out;
+ }
+
+ if (!pid) {
+ /* child */
+ remus_netbuf_op_async(remus_nic, ns, buffer_op);
+ /* notreached */
+ abort();
+ }
+
+ return;
+
+out:
+ dev->callback(dev->rds->egc, dev, ERROR_FAIL);
+}
+
+static void nic_postsuspend(libxl__remus_device *dev)
+{
+ netbuf_epoch_op(dev, tc_buffer_start);
+}
+
+static void nic_commit(libxl__remus_device *dev)
+{
+ netbuf_epoch_op(dev, tc_buffer_release);
+}
+
+libxl__remus_device_ops remus_device_nic = {
+ .init = nic_init,
+ .destroy = nic_destroy,
+ .postsuspend = nic_postsuspend,
+ .commit = nic_commit,
+ .match = nic_match,
+ .setup = nic_setup,
+ .teardown = nic_teardown,
+};
+
/*
* Local variables:
* mode: C
diff --git a/tools/libxl/libxl_nonetbuffer.c b/tools/libxl/libxl_nonetbuffer.c
index 6aa4bf1..7fe288a 100644
--- a/tools/libxl/libxl_nonetbuffer.c
+++ b/tools/libxl/libxl_nonetbuffer.c
@@ -22,6 +22,73 @@ int libxl__netbuffer_enabled(libxl__gc *gc)
return 0;
}
+static void async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ pid_t pid, int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(child, *dev, child);
+ libxl__remus_device_state *rds = dev->rds;
+ STATE_AO_GC(rds->ao);
+
+ if (WIFEXITED(status)) {
+ dev->callback(egc, dev, -WEXITSTATUS(status));
+ } else {
+ dev->callback(egc, dev, ERROR_FAIL);
+ }
+}
+
+static void nic_match_async(const libxl__remus_device_ops *self,
+ libxl__remus_device *dev)
+{
+ if (dev->kind == LIBXL__REMUS_DEVICE_NIC)
+ _exit(-ERROR_FAIL);
+
+ _exit(-ERROR_NOT_MATCH);
+}
+
+static void nic_match(libxl__remus_device_ops *self,
+ libxl__remus_device *dev)
+{
+ int pid = -1;
+ STATE_AO_GC(dev->rds->ao);
+
+ /* Fork and call */
+ pid = libxl__ev_child_fork(gc, &dev->child, async_call_done);
+ if (pid == -1) {
+ LOG(ERROR, "unable to fork");
+ goto out;
+ }
+
+ if (!pid) {
+ /* child */
+ nic_match_async(self, dev);
+ /* notreached */
+ abort();
+ }
+
+ return;
+
+out:
+ dev->callback(dev->rds->egc, dev, ERROR_FAIL);
+}
+
+static int nic_init(libxl__remus_device_ops *self,
+ libxl__remus_state *rs)
+{
+ return 0;
+}
+
+static void nic_destroy(libxl__remus_device_ops *self)
+{
+ return;
+}
+
+libxl__remus_device_ops remus_device_nic = {
+ .init = nic_init,
+ .destroy = nic_destroy,
+ .match = nic_match,
+};
+
/*
* Local variables:
* mode: C
diff --git a/tools/libxl/libxl_remus_device.c b/tools/libxl/libxl_remus_device.c
index 62c0614..5f07266 100644
--- a/tools/libxl/libxl_remus_device.c
+++ b/tools/libxl/libxl_remus_device.c
@@ -18,7 +18,9 @@
#include "libxl_internal.h"
+extern libxl__remus_device_ops remus_device_nic;
static libxl__remus_device_ops *dev_ops[] = {
+ &remus_device_nic,
};
static void device_common_cb(libxl__egc *egc,
@@ -185,6 +187,13 @@ static void device_teardown_cb(libxl__egc *egc,
rds->num_setuped--;
if (rds->num_setuped == 0) {
+ /* clean nic */
+ for (i = 0; i < rds->num_nics; i++)
+ libxl_device_nic_dispose(&rds->nics[i]);
+ free(rds->nics);
+ rds->nics = NULL;
+ rds->num_nics = 0;
+
/* clean device ops */
for (i = 0; i < ARRAY_SIZE(dev_ops); i++) {
ops = dev_ops[i];
@@ -194,7 +203,7 @@ static void device_teardown_cb(libxl__egc *egc,
}
}
-static __attribute__((unused)) void libxl__remus_device_init(libxl__egc *egc,
+static void libxl__remus_device_init(libxl__egc *egc,
libxl__remus_device_state *rds,
libxl__remus_device_kind kind,
void *libxl_dev)
@@ -262,10 +271,21 @@ void libxl__remus_device_setup(libxl__egc *egc, libxl__remus_state *rs)
/* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */
+ if (rs->netbufscript) {
+ rds->nics = libxl_device_nic_list(CTX, rs->domid, &rds->num_nics);
+ }
+
GCNEW_ARRAY(rds->dev, rds->num_nics + rds->num_disks);
/* TBD: CALL libxl__remus_device_init to init remus devices */
+ if (rs->netbufscript && rds->nics) {
+ for (i = 0; i < rds->num_nics; i++) {
+ libxl__remus_device_init(egc, rds,
+ LIBXL__REMUS_DEVICE_NIC, &rds->nics[i]);
+ }
+ }
+
if (rds->num_nics == 0 && rds->num_disks == 0)
goto out;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index c757f17..d3122fa 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -565,6 +565,8 @@ libxl_domain_remus_info = Struct("domain_remus_info",[
("interval", integer),
("blackhole", bool),
("compression", bool),
+ ("netbuf", bool),
+ ("netbufscript", string),
])
libxl_event_type = Enumeration("event_type", [
--
1.9.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v10 5/5] libxl: network buffering cmdline switch
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
` (3 preceding siblings ...)
2014-05-21 10:14 ` [PATCH v10 4/5] remus: implement remus network buffering for nic devices Yang Hongyang
@ 2014-05-21 10:15 ` Yang Hongyang
2014-05-28 9:22 ` [PATCH v10] remus drbd: Implement remus drbd replicated disk Yang Hongyang
2014-05-26 1:11 ` [PATCH v10 0/5] Remus/Libxl: Network buffering support Hongyang Yang
2014-05-28 10:01 ` Hongyang Yang
6 siblings, 1 reply; 9+ messages in thread
From: Yang Hongyang @ 2014-05-21 10:15 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, eddie.dong, rshriram, roger.pau,
laijs
From: Shriram Rajagopalan <rshriram@cs.ubc.ca>
Command line switch to 'xl remus' command, to enable network buffering.
Pass on this flag to libxl so that it can act accordingly.
Also update man pages to reflect the addition of a new option to
'xl remus' command.
Note: the network buffering is enabled as default. If you want to
disable it, please use -n option.
Signed-off-by: Shriram Rajagopalan <rshriram@cs.ubc.ca>
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Wen Congyang <wency@cn.fujitsu.com>
---
docs/man/xl.conf.pod.5 | 6 ++++++
docs/man/xl.pod.1 | 11 ++++++++++-
tools/libxl/xl.c | 4 ++++
tools/libxl/xl.h | 1 +
tools/libxl/xl_cmdimpl.c | 28 ++++++++++++++++++++++------
tools/libxl/xl_cmdtable.c | 3 +++
6 files changed, 46 insertions(+), 7 deletions(-)
diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
index 7c43bde..8ae19bb 100644
--- a/docs/man/xl.conf.pod.5
+++ b/docs/man/xl.conf.pod.5
@@ -105,6 +105,12 @@ Configures the default gateway device to set for virtual network devices.
Default: C<None>
+=item B<remus.default.netbufscript="PATH">
+
+Configures the default script used by Remus to setup network buffering.
+
+Default: C</etc/xen/scripts/remus-netbuf-setup>
+
=item B<output_format="json|sxp">
Configures the default output format used by xl when printing "machine
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 30bd4bf..8b0c012 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -431,7 +431,7 @@ Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
mechanism between the two hosts.
N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
- There is no support for network or disk buffering at the moment.
+ There is no support for disk buffering at the moment.
B<OPTIONS>
@@ -450,6 +450,15 @@ Generally useful for debugging.
Disable memory checkpoint compression.
+=item B<-n>
+
+Disable network output buffering.
+
+=item B<-N> I<netbufscript>
+
+Use <netbufscript> to setup network buffering instead of the instead of
+the default (/etc/xen/scripts/remus-netbuf-setup).
+
=item B<-s> I<sshcommand>
Use <sshcommand> instead of ssh. String will be passed to sh.
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index 4c5a5ee..f014306 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -44,6 +44,7 @@ char *default_vifscript = NULL;
char *default_bridge = NULL;
char *default_gatewaydev = NULL;
char *default_vifbackend = NULL;
+char *default_remus_netbufscript = NULL;
enum output_format default_output_format = OUTPUT_FORMAT_JSON;
int claim_mode = 1;
bool progress_use_cr = 0;
@@ -176,6 +177,9 @@ static void parse_global_config(const char *configfile,
if (!xlu_cfg_get_long (config, "claim_mode", &l, 0))
claim_mode = l;
+ xlu_cfg_replace_string (config, "remus.default.netbufscript",
+ &default_remus_netbufscript, 0);
+
xlu_cfg_destroy(config);
}
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index 10a2e66..087eb8c 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -170,6 +170,7 @@ extern char *default_vifscript;
extern char *default_bridge;
extern char *default_gatewaydev;
extern char *default_vifbackend;
+extern char *default_remus_netbufscript;
extern char *blkdev_start;
enum output_format {
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 5195914..ce06e82 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -7284,8 +7284,9 @@ int main_remus(int argc, char **argv)
r_info.interval = 200;
r_info.blackhole = 0;
r_info.compression = 1;
+ r_info.netbuf = 1;
- SWITCH_FOREACH_OPT(opt, "bui:s:e", NULL, "remus", 2) {
+ SWITCH_FOREACH_OPT(opt, "buni:s:N:e", NULL, "remus", 2) {
case 'i':
r_info.interval = atoi(optarg);
break;
@@ -7295,6 +7296,12 @@ int main_remus(int argc, char **argv)
case 'u':
r_info.compression = 0;
break;
+ case 'n':
+ r_info.netbuf = 0;
+ break;
+ case 'N':
+ r_info.netbufscript = optarg;
+ break;
case 's':
ssh_command = optarg;
break;
@@ -7306,6 +7313,9 @@ int main_remus(int argc, char **argv)
domid = find_domain(argv[optind]);
host = argv[optind + 1];
+ if (!r_info.netbufscript)
+ r_info.netbufscript = default_remus_netbufscript;
+
if (r_info.blackhole) {
send_fd = open("/dev/null", O_RDWR, 0644);
if (send_fd < 0) {
@@ -7343,13 +7353,19 @@ int main_remus(int argc, char **argv)
/* Point of no return */
rc = libxl_domain_remus_start(ctx, &r_info, domid, send_fd, recv_fd, 0);
- /* If we are here, it means backup has failed/domain suspend failed.
- * Try to resume the domain and exit gracefully.
- * TODO: Split-Brain check.
+ /* check if the domain exists. User may have xl destroyed the
+ * domain to force failover
*/
- fprintf(stderr, "remus sender: libxl_domain_suspend failed"
- " (rc=%d)\n", rc);
+ if (libxl_domain_info(ctx, 0, domid)) {
+ fprintf(stderr, "Remus: Primary domain has been destroyed.\n");
+ close(send_fd);
+ return 0;
+ }
+ /* If we are here, it means remus setup/domain suspend/backup has
+ * failed. Try to resume the domain and exit gracefully.
+ * TODO: Split-Brain check.
+ */
if (rc == ERROR_GUEST_TIMEDOUT)
fprintf(stderr, "Failed to suspend domain at primary.\n");
else {
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 4279b9f..3f7520d 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -487,6 +487,9 @@ struct cmd_spec cmd_table[] = {
"-i MS Checkpoint domain memory every MS milliseconds (def. 200ms).\n"
"-b Replicate memory checkpoints to /dev/null (blackhole)\n"
"-u Disable memory checkpoint compression.\n"
+ "-n Disable network output buffering.\n"
+ "-N <netbufscript> Use netbufscript to setup network buffering instead of the\n"
+ " instead of the default (/etc/xen/scripts/remus-netbuf-setup).\n"
"-s <sshcommand> Use <sshcommand> instead of ssh. String will be passed\n"
" to sh. If empty, run <host> instead of \n"
" ssh <host> xl migrate-receive -r [-e]\n"
--
1.9.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v10 0/5] Remus/Libxl: Network buffering support
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
` (4 preceding siblings ...)
2014-05-21 10:15 ` [PATCH v10 5/5] libxl: network buffering cmdline switch Yang Hongyang
@ 2014-05-26 1:11 ` Hongyang Yang
2014-05-28 10:01 ` Hongyang Yang
6 siblings, 0 replies; 9+ messages in thread
From: Hongyang Yang @ 2014-05-26 1:11 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, laijs, eddie.dong, rshriram,
roger.pau
Ping!
On 05/21/2014 06:14 PM, Yang Hongyang wrote:
> This patch series adds support for network buffering in the Remus
> codebase in libxl.
>
> Changes in V10:
> Restructured the whole patch series.
> Introduce the remus device abstract layer.
> Make remus checkpoint asynchronous.
>
> Changes in V9:
> Use async exec script api to exec scripts.
>
> Changes in V8:
> Applied some comments(by IanJ).
> Merge some struct definitions to it's implementation.
> (2/3/5 in V7 => 3 in V8)
>
> Changes in V7:
> Applied missing comments(by IanJ).
> Applied Shriram comments.
>
> merge netbufering tangled setup/teardown code into one patch.
> (2/6/8 in V6 => 5 in V7. 9/10 in V6 => 7 in V7)
>
> Changes in V6:
> Applied Ian Jackson's comments of V5 series.
> the [PATCH 2/4 V5] is split by small functionalities.
>
> [PATCH 4/4 V5] --> [PATCH 13/13] netbuffer is default enabled.
>
> Changes in V5:
>
> Merge hotplug script patch (2/5) and hotplug script setup/teardown
> patch (3/5) into a single patch.
>
> Changes in V4:
>
> [1/5] Remove check for libnl command line utils in autoconf checks
>
> [2/5] minor nits
>
> [3/5] define LIBXL_HAVE_REMUS_NETBUF in libxl.h
>
> [4/5] clean ups. Make the usleep in checkpoint callback asynchronous
>
> [5/5] minor nits
>
> Changes in V3:
> [1/5] Fix redundant checks in configure scripts
> (based on Ian Campbell's suggestions)
>
> [2/5] Introduce locking in the script, during IFB setup.
> Add xenstore paths used by netbuf scripts
> to xenstore-paths.markdown
>
> [3/5] Hotplug scripts setup/teardown invocations are now asynchronous
> following IanJ's feedback. However, the invocations are still
> sequential.
>
> [5/5] Allow per-domain specification of netbuffer scripts in xl remus
> commmand.
>
> And minor nits throughout the series based on feedback from
> the last version
>
> Changes in V2:
> [1/5] Configure script will automatically enable/disable network
> buffer support depending on the availability of the appropriate
> libnl3 version. [If libnl3 is unavailable, a warning message will be
> printed to let the user know that the feature has been disabled.]
>
> use macros from pkg.m4 instead of pkg-config commands
> removed redundant checks for libnl3 libraries.
>
> [3,4/5] - Minor nits.
>
> Version 1:
>
> [1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
> to libxl Makefile.
>
> [2/5] External script to setup/teardown network buffering using libnl3's
> CLI. This script will be invoked by libxl before starting Remus.
> The script's main job is to bring up an IFB device with plug qdisc
> attached to it. It then re-routes egress traffic from the guest's
> vif to the IFB device.
>
> [3/5] Libxl code to invoke the external setup script, followed by netlink
> related setup to obtain a handle on the output buffers attached
> to each vif.
>
> [4/5] Libxl interaction with network buffer module in the kernel via
> libnl3 API.
>
> [5/5] xl cmdline switch to explicitly enable network buffering when
> starting remus.
>
>
> Few things to note(by shriram):
>
> a) Based on previous email discussions, the setup/teardown task has
> been moved to a hotplug style shell script which can be customized as
> desired, instead of implementing it as C code inside libxl.
>
> b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
> (Linux). So I have made network buffering support an optional feature
> so that it can be disabled if desired.
>
> c) NetBSD does not have libnl3. So I have put the setup script under
> tools/hotplug/Linux folder.
>
> thanks
>
> Shriram Rajagopalan (1):
> libxl: network buffering cmdline switch
>
> Yang Hongyang (4):
> libxl: introduce asynchronous execution API
> remus: add libnl3 dependency for network buffering support
> remus: introduce remus device
> remus: implement remus network buffering for nic devices
>
> README | 4 +
> config/Tools.mk.in | 4 +
> docs/man/xl.conf.pod.5 | 6 +
> docs/man/xl.pod.1 | 11 +-
> docs/misc/xenstore-paths.markdown | 4 +
> tools/configure.ac | 15 +
> tools/hotplug/Linux/Makefile | 1 +
> tools/hotplug/Linux/remus-netbuf-setup | 183 +++++++++++
> tools/libxl/Makefile | 15 +
> tools/libxl/libxl.c | 52 +++-
> tools/libxl/libxl.h | 13 +
> tools/libxl/libxl_aoutils.c | 89 ++++++
> tools/libxl/libxl_device.c | 78 ++---
> tools/libxl/libxl_dom.c | 132 +++++++-
> tools/libxl/libxl_internal.h | 151 ++++++++-
> tools/libxl/libxl_netbuffer.c | 550 +++++++++++++++++++++++++++++++++
> tools/libxl/libxl_nonetbuffer.c | 98 ++++++
> tools/libxl/libxl_remus_device.c | 323 +++++++++++++++++++
> tools/libxl/libxl_save_msgs_gen.pl | 2 +-
> tools/libxl/libxl_types.idl | 3 +
> tools/libxl/xl.c | 4 +
> tools/libxl/xl.h | 1 +
> tools/libxl/xl_cmdimpl.c | 28 +-
> tools/libxl/xl_cmdtable.c | 3 +
> tools/remus/README | 6 +
> 25 files changed, 1694 insertions(+), 82 deletions(-)
> create mode 100644 tools/hotplug/Linux/remus-netbuf-setup
> create mode 100644 tools/libxl/libxl_netbuffer.c
> create mode 100644 tools/libxl/libxl_nonetbuffer.c
> create mode 100644 tools/libxl/libxl_remus_device.c
>
--
Thanks,
Yang.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v10] remus drbd: Implement remus drbd replicated disk
2014-05-21 10:15 ` [PATCH v10 5/5] libxl: network buffering cmdline switch Yang Hongyang
@ 2014-05-28 9:22 ` Yang Hongyang
0 siblings, 0 replies; 9+ messages in thread
From: Yang Hongyang @ 2014-05-28 9:22 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, eddie.dong, rshriram, roger.pau,
laijs
Implement remus-drbd-replicated-checkpointing-disk based on
generic remus devices framework.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
tools/hotplug/Linux/Makefile | 1 +
tools/hotplug/Linux/block-drbd-probe | 84 ++++++++++
tools/libxl/Makefile | 2 +-
tools/libxl/libxl_internal.h | 1 +
tools/libxl/libxl_remus_device.c | 23 ++-
tools/libxl/libxl_remus_disk_drbd.c | 290 +++++++++++++++++++++++++++++++++++
6 files changed, 394 insertions(+), 7 deletions(-)
create mode 100755 tools/hotplug/Linux/block-drbd-probe
create mode 100644 tools/libxl/libxl_remus_disk_drbd.c
diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile
index 13e1f5f..5dd8599 100644
--- a/tools/hotplug/Linux/Makefile
+++ b/tools/hotplug/Linux/Makefile
@@ -23,6 +23,7 @@ XEN_SCRIPTS += xen-hotplug-cleanup
XEN_SCRIPTS += external-device-migrate
XEN_SCRIPTS += vscsi
XEN_SCRIPTS += block-iscsi
+XEN_SCRIPTS += block-drbd-probe
XEN_SCRIPTS += $(XEN_SCRIPTS-y)
XEN_SCRIPT_DATA = xen-script-common.sh locking.sh logging.sh
diff --git a/tools/hotplug/Linux/block-drbd-probe b/tools/hotplug/Linux/block-drbd-probe
new file mode 100755
index 0000000..163ad04
--- /dev/null
+++ b/tools/hotplug/Linux/block-drbd-probe
@@ -0,0 +1,84 @@
+#! /bin/bash
+#
+# Copyright (C) 2014 FUJITSU LIMITED
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of version 2.1 of the GNU Lesser General Public
+# License as published by the Free Software Foundation.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+# Usage:
+# block-drbd-probe devicename
+#
+# Return value:
+# 0: the device is drbd device
+# 1: the device is not drbd device
+# 2: unkown error
+# 3: the drbd device does not use protocol D
+# 4: the drbd device is not ready
+
+drbd_res=
+
+function get_res_name()
+{
+ local drbd_dev=$1
+ local drbd_dev_list=($(drbdadm sh-dev all))
+ local drbd_res_list=($(drbdadm sh-resource all))
+ local temp_drbd_dev temp_drbd_res
+ local found=0
+
+ for temp_drbd_dev in ${drbd_dev_list[@]}; do
+ if [[ "$temp_drbd_dev" == "$drbd_dev" ]]; then
+ found=1
+ break
+ fi
+ done
+
+ if [[ $found -eq 0 ]]; then
+ return 1
+ fi
+
+ for temp_drbd_res in ${drbd_res_list[@]}; do
+ temp_drbd_dev=$(drbdadm sh-dev $temp_drbd_res)
+ if [[ "$temp_drbd_dev" == "$drbd_dev" ]]; then
+ drbd_res="$temp_drbd_res"
+ return 0
+ fi
+ done
+
+ # OOPS
+ return 2
+}
+
+get_res_name $1
+if [[ $? -ne 0 ]]; then
+ exit $?
+fi
+
+# check protocol
+drbdsetup $1 show | grep -q "protocol D;"
+if [[ $? -ne 0 ]]; then
+ exit 3
+fi
+
+# check connect status
+state=$(drbdadm cstate "$drbd_res")
+if [[ "$state" != "Connected" ]]; then
+ exit 4
+fi
+
+# check role
+role=$(drbdadm role "$drbd_res")
+if [[ "$role" != "Primary/Secondary" ]]; then
+ exit 4
+fi
+
+exit 0
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 7a722a8..6f4d9b4 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -56,7 +56,7 @@ else
LIBXL_OBJS-y += libxl_nonetbuffer.o
endif
-LIBXL_OBJS-y += libxl_remus_device.o
+LIBXL_OBJS-y += libxl_remus_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 85fa713..9d344e8 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2499,6 +2499,7 @@ struct libxl__remus_device_state {
libxl_device_nic *nics;
int num_nics;
+ libxl_device_disk *disks;
int num_disks;
/* for counting devices that have been handled */
diff --git a/tools/libxl/libxl_remus_device.c b/tools/libxl/libxl_remus_device.c
index 5f07266..040441a 100644
--- a/tools/libxl/libxl_remus_device.c
+++ b/tools/libxl/libxl_remus_device.c
@@ -19,8 +19,10 @@
#include "libxl_internal.h"
extern libxl__remus_device_ops remus_device_nic;
+extern libxl__remus_device_ops remus_device_drbd_disk;
static libxl__remus_device_ops *dev_ops[] = {
&remus_device_nic,
+ &remus_device_drbd_disk,
};
static void device_common_cb(libxl__egc *egc,
@@ -194,6 +196,13 @@ static void device_teardown_cb(libxl__egc *egc,
rds->nics = NULL;
rds->num_nics = 0;
+ /* clean disk */
+ for (i = 0; i < rds->num_disks; i++)
+ libxl_device_disk_dispose(&rds->disks[i]);
+ free(rds->disks);
+ rds->disks = NULL;
+ rds->num_disks = 0;
+
/* clean device ops */
for (i = 0; i < ARRAY_SIZE(dev_ops); i++) {
ops = dev_ops[i];
@@ -269,15 +278,15 @@ void libxl__remus_device_setup(libxl__egc *egc, libxl__remus_state *rs)
rds->num_nics = 0;
rds->num_disks = 0;
- /* TBD: Remus setup - i.e. attach qdisc, enable disk buffering, etc */
-
if (rs->netbufscript) {
rds->nics = libxl_device_nic_list(CTX, rs->domid, &rds->num_nics);
}
+ rds->disks = libxl_device_disk_list(CTX, rs->domid, &rds->num_disks);
- GCNEW_ARRAY(rds->dev, rds->num_nics + rds->num_disks);
+ if (rds->num_nics == 0 && rds->num_disks == 0)
+ goto out;
- /* TBD: CALL libxl__remus_device_init to init remus devices */
+ GCNEW_ARRAY(rds->dev, rds->num_nics + rds->num_disks);
if (rs->netbufscript && rds->nics) {
for (i = 0; i < rds->num_nics; i++) {
@@ -286,8 +295,10 @@ void libxl__remus_device_setup(libxl__egc *egc, libxl__remus_state *rs)
}
}
- if (rds->num_nics == 0 && rds->num_disks == 0)
- goto out;
+ for (i = 0; i < rds->num_disks; i++) {
+ libxl__remus_device_init(egc, rds,
+ LIBXL__REMUS_DEVICE_DISK, &rds->disks[i]);
+ }
return;
diff --git a/tools/libxl/libxl_remus_disk_drbd.c b/tools/libxl/libxl_remus_disk_drbd.c
new file mode 100644
index 0000000..f35a406
--- /dev/null
+++ b/tools/libxl/libxl_remus_disk_drbd.c
@@ -0,0 +1,290 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author Lai Jiangshan <laijs@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*** drbd implementation ***/
+const int DRBD_SEND_CHECKPOINT = 20;
+const int DRBD_WAIT_CHECKPOINT_ACK = 30;
+
+typedef struct libxl__remus_drbd_disk {
+ libxl__remus_device remus_dev;
+ int ctl_fd;
+ int ackwait;
+ const char *path;
+} libxl__remus_drbd_disk;
+
+typedef struct libxl__remus_drbd_state {
+ libxl__ao *ao;
+ char *drbd_probe_script;
+} libxl__remus_drbd_state;
+
+static void drbd_async_call(libxl__remus_device *dev,
+ void func(libxl__remus_device *),
+ libxl__ev_child_callback callback)
+{
+ int pid = -1;
+ STATE_AO_GC(dev->rds->ao);
+
+ /* Fork and call */
+ pid = libxl__ev_child_fork(gc, &dev->child, callback);
+ if (pid == -1) {
+ LOG(ERROR, "unable to fork");
+ goto out;
+ }
+
+ if (!pid) {
+ /* child */
+ func(dev);
+ /* notreached */
+ abort();
+ }
+
+ return;
+
+out:
+ dev->callback(dev->rds->egc, dev, ERROR_FAIL);
+}
+
+static void chekpoint_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ pid_t pid, int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(child, *dev, child);
+ libxl__remus_drbd_disk *rdd = dev->data;
+ STATE_AO_GC(dev->rds->ao);
+
+ if (WIFEXITED(status)) {
+ rdd->ackwait = WEXITSTATUS(status);
+ dev->callback(egc, dev, 0);
+ } else {
+ dev->callback(egc, dev, ERROR_FAIL);
+ }
+}
+
+static void drbd_postsuspend_async(libxl__remus_device *dev)
+{
+ libxl__remus_drbd_disk *rdd = dev->data;
+ int ackwait = rdd->ackwait;
+
+ if (!ackwait) {
+ if (ioctl(rdd->ctl_fd, DRBD_SEND_CHECKPOINT, 0) <= 0)
+ ackwait = 1;
+ }
+
+ _exit(ackwait);
+}
+
+static void drbd_postsuspend(libxl__remus_device *dev)
+{
+ drbd_async_call(dev, drbd_postsuspend_async, chekpoint_async_call_done);
+}
+
+static void drbd_preresume_async(libxl__remus_device *dev)
+{
+ libxl__remus_drbd_disk *rdd = dev->data;
+ int ackwait = rdd->ackwait;
+
+ if (ackwait) {
+ ioctl(rdd->ctl_fd, DRBD_WAIT_CHECKPOINT_ACK, 0);
+ ackwait = 0;
+ }
+
+ _exit(ackwait);
+}
+
+static void drbd_preresume(libxl__remus_device *dev)
+{
+ drbd_async_call(dev, drbd_preresume_async, chekpoint_async_call_done);
+}
+
+static int drbd_init(libxl__remus_device_ops *self,
+ libxl__remus_state *rs)
+{
+ libxl__remus_drbd_state *drbd_state;
+
+ STATE_AO_GC(rs->ao);
+
+ GCNEW(drbd_state);
+ self->data = drbd_state;
+ drbd_state->ao = ao;
+ drbd_state->drbd_probe_script = GCSPRINTF("%s/block-drbd-probe",
+ libxl__xen_script_dir_path());
+
+
+ return 0;
+}
+
+static void drbd_destroy(libxl__remus_device_ops *self)
+{
+ return;
+}
+
+static void match_async_exec_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(aes, *dev, aes);
+
+ if (status) {
+ dev->callback(egc, dev, ERROR_NOT_MATCH);
+ } else {
+ dev->callback(egc, dev, 0);
+ }
+}
+
+static void match_async_exec(libxl__egc *egc, libxl__remus_device *dev)
+{
+ int arraysize, nr = 0;
+ const libxl_device_disk *disk = dev->backend_dev;
+ libxl__remus_drbd_state *drbd_state = dev->ops->data;
+ libxl__async_exec_state *aes = &dev->aes;
+ STATE_AO_GC(drbd_state->ao);
+
+ /* setup env & args */
+ arraysize = 1;
+ GCNEW_ARRAY(aes->env, arraysize);
+ aes->env[nr++] = NULL;
+ assert(nr <= arraysize);
+
+ arraysize = 3;
+ nr = 0;
+ GCNEW_ARRAY(aes->args, arraysize);
+ aes->args[nr++] = drbd_state->drbd_probe_script;
+ aes->args[nr++] = disk->pdev_path;
+ aes->args[nr++] = NULL;
+ assert(nr <= arraysize);
+
+ aes->ao = drbd_state->ao;
+ aes->what = GCSPRINTF("%s %s", aes->args[0], aes->args[1]);
+ aes->timeout_ms = LIBXL_HOTPLUG_TIMEOUT * 1000;
+ aes->callback = match_async_exec_cb;
+ aes->stdfds[0] = -1;
+ aes->stdfds[1] = -1;
+ aes->stdfds[2] = -1;
+
+ if (libxl__async_exec_start(gc, aes))
+ goto out;
+
+ return;
+
+out:
+ dev->callback(egc, dev, ERROR_FAIL);
+}
+
+static void match_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ pid_t pid, int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(child, *dev, child);
+ STATE_AO_GC(dev->rds->ao);
+
+ if (WIFEXITED(status)) {
+ if (-WEXITSTATUS(status) == ERROR_NOT_MATCH) {
+ dev->callback(egc, dev, ERROR_NOT_MATCH);
+ } else {
+ match_async_exec(egc, dev);
+ }
+ } else {
+ dev->callback(egc, dev, ERROR_FAIL);
+ }
+}
+
+static void drbd_match_async(libxl__remus_device *dev)
+{
+ if (dev->kind != LIBXL__REMUS_DEVICE_DISK)
+ _exit(-ERROR_NOT_MATCH);
+
+ _exit(0);
+}
+
+static void drbd_match(libxl__remus_device_ops *self,
+ libxl__remus_device *dev)
+{
+ drbd_async_call(dev, drbd_match_async, match_async_call_done);
+}
+
+static void setup_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ pid_t pid, int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(child, *dev, child);
+ STATE_AO_GC(dev->rds->ao);
+
+ if (WIFEXITED(status)) {
+ dev->callback(egc, dev, 0);
+ } else {
+ dev->callback(egc, dev, ERROR_FAIL);
+ }
+}
+
+static void drbd_setup_async(libxl__remus_device *dev)
+{
+ libxl__remus_drbd_disk *drbd_disk = dev->data;
+
+ if (drbd_disk->ctl_fd < 0)
+ abort();
+
+ _exit(0);
+}
+
+static void drbd_setup(libxl__remus_device *dev)
+{
+ libxl__remus_drbd_disk *drbd_disk;
+ const libxl_device_disk *disk = dev->backend_dev;
+ STATE_AO_GC(dev->rds->ao);
+
+ GCNEW(drbd_disk);
+ dev->data = drbd_disk;
+ drbd_disk->path = disk->pdev_path;
+ drbd_disk->ackwait = 0;
+ drbd_disk->ctl_fd = open(drbd_disk->path, O_RDONLY);
+ drbd_async_call(dev, drbd_setup_async, setup_async_call_done);
+}
+
+static void teardown_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ pid_t pid, int status)
+{
+ libxl__remus_device *dev = CONTAINER_OF(child, *dev, child);
+ STATE_AO_GC(dev->rds->ao);
+
+ dev->callback(egc, dev, 0);
+}
+
+static void drbd_teardown_async(libxl__remus_device *dev)
+{
+ _exit(0);
+}
+
+static void drbd_teardown(libxl__remus_device *dev)
+{
+ libxl__remus_drbd_disk *drbd_disk = dev->data;
+
+ close(drbd_disk->ctl_fd);
+ drbd_async_call(dev, drbd_teardown_async, teardown_async_call_done);
+}
+
+libxl__remus_device_ops remus_device_drbd_disk = {
+ .init = drbd_init,
+ .destroy = drbd_destroy,
+ .postsuspend = drbd_postsuspend,
+ .preresume = drbd_preresume,
+ .match = drbd_match,
+ .setup = drbd_setup,
+ .teardown = drbd_teardown,
+};
--
1.9.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v10 0/5] Remus/Libxl: Network buffering support
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
` (5 preceding siblings ...)
2014-05-26 1:11 ` [PATCH v10 0/5] Remus/Libxl: Network buffering support Hongyang Yang
@ 2014-05-28 10:01 ` Hongyang Yang
6 siblings, 0 replies; 9+ messages in thread
From: Hongyang Yang @ 2014-05-28 10:01 UTC (permalink / raw)
To: xen-devel
Cc: ian.campbell, wency, stefano.stabellini, andrew.cooper3,
yunhong.jiang, ian.jackson, laijs, eddie.dong, rshriram,
roger.pau
the code is hosted on github:
url: https://github.com/laijs/xen
branch: remus-0528
On 05/21/2014 06:14 PM, Yang Hongyang wrote:
> This patch series adds support for network buffering in the Remus
> codebase in libxl.
>
> Changes in V10:
> Restructured the whole patch series.
> Introduce the remus device abstract layer.
> Make remus checkpoint asynchronous.
>
> Changes in V9:
> Use async exec script api to exec scripts.
>
> Changes in V8:
> Applied some comments(by IanJ).
> Merge some struct definitions to it's implementation.
> (2/3/5 in V7 => 3 in V8)
>
> Changes in V7:
> Applied missing comments(by IanJ).
> Applied Shriram comments.
>
> merge netbufering tangled setup/teardown code into one patch.
> (2/6/8 in V6 => 5 in V7. 9/10 in V6 => 7 in V7)
>
> Changes in V6:
> Applied Ian Jackson's comments of V5 series.
> the [PATCH 2/4 V5] is split by small functionalities.
>
> [PATCH 4/4 V5] --> [PATCH 13/13] netbuffer is default enabled.
>
> Changes in V5:
>
> Merge hotplug script patch (2/5) and hotplug script setup/teardown
> patch (3/5) into a single patch.
>
> Changes in V4:
>
> [1/5] Remove check for libnl command line utils in autoconf checks
>
> [2/5] minor nits
>
> [3/5] define LIBXL_HAVE_REMUS_NETBUF in libxl.h
>
> [4/5] clean ups. Make the usleep in checkpoint callback asynchronous
>
> [5/5] minor nits
>
> Changes in V3:
> [1/5] Fix redundant checks in configure scripts
> (based on Ian Campbell's suggestions)
>
> [2/5] Introduce locking in the script, during IFB setup.
> Add xenstore paths used by netbuf scripts
> to xenstore-paths.markdown
>
> [3/5] Hotplug scripts setup/teardown invocations are now asynchronous
> following IanJ's feedback. However, the invocations are still
> sequential.
>
> [5/5] Allow per-domain specification of netbuffer scripts in xl remus
> commmand.
>
> And minor nits throughout the series based on feedback from
> the last version
>
> Changes in V2:
> [1/5] Configure script will automatically enable/disable network
> buffer support depending on the availability of the appropriate
> libnl3 version. [If libnl3 is unavailable, a warning message will be
> printed to let the user know that the feature has been disabled.]
>
> use macros from pkg.m4 instead of pkg-config commands
> removed redundant checks for libnl3 libraries.
>
> [3,4/5] - Minor nits.
>
> Version 1:
>
> [1/5] Changes to autoconf scripts to check for libnl3. Add linker flags
> to libxl Makefile.
>
> [2/5] External script to setup/teardown network buffering using libnl3's
> CLI. This script will be invoked by libxl before starting Remus.
> The script's main job is to bring up an IFB device with plug qdisc
> attached to it. It then re-routes egress traffic from the guest's
> vif to the IFB device.
>
> [3/5] Libxl code to invoke the external setup script, followed by netlink
> related setup to obtain a handle on the output buffers attached
> to each vif.
>
> [4/5] Libxl interaction with network buffer module in the kernel via
> libnl3 API.
>
> [5/5] xl cmdline switch to explicitly enable network buffering when
> starting remus.
>
>
> Few things to note(by shriram):
>
> a) Based on previous email discussions, the setup/teardown task has
> been moved to a hotplug style shell script which can be customized as
> desired, instead of implementing it as C code inside libxl.
>
> b) Libnl3 is not available on NetBSD. Nor is it available on CentOS
> (Linux). So I have made network buffering support an optional feature
> so that it can be disabled if desired.
>
> c) NetBSD does not have libnl3. So I have put the setup script under
> tools/hotplug/Linux folder.
>
> thanks
>
> Shriram Rajagopalan (1):
> libxl: network buffering cmdline switch
>
> Yang Hongyang (4):
> libxl: introduce asynchronous execution API
> remus: add libnl3 dependency for network buffering support
> remus: introduce remus device
> remus: implement remus network buffering for nic devices
>
> README | 4 +
> config/Tools.mk.in | 4 +
> docs/man/xl.conf.pod.5 | 6 +
> docs/man/xl.pod.1 | 11 +-
> docs/misc/xenstore-paths.markdown | 4 +
> tools/configure.ac | 15 +
> tools/hotplug/Linux/Makefile | 1 +
> tools/hotplug/Linux/remus-netbuf-setup | 183 +++++++++++
> tools/libxl/Makefile | 15 +
> tools/libxl/libxl.c | 52 +++-
> tools/libxl/libxl.h | 13 +
> tools/libxl/libxl_aoutils.c | 89 ++++++
> tools/libxl/libxl_device.c | 78 ++---
> tools/libxl/libxl_dom.c | 132 +++++++-
> tools/libxl/libxl_internal.h | 151 ++++++++-
> tools/libxl/libxl_netbuffer.c | 550 +++++++++++++++++++++++++++++++++
> tools/libxl/libxl_nonetbuffer.c | 98 ++++++
> tools/libxl/libxl_remus_device.c | 323 +++++++++++++++++++
> tools/libxl/libxl_save_msgs_gen.pl | 2 +-
> tools/libxl/libxl_types.idl | 3 +
> tools/libxl/xl.c | 4 +
> tools/libxl/xl.h | 1 +
> tools/libxl/xl_cmdimpl.c | 28 +-
> tools/libxl/xl_cmdtable.c | 3 +
> tools/remus/README | 6 +
> 25 files changed, 1694 insertions(+), 82 deletions(-)
> create mode 100644 tools/hotplug/Linux/remus-netbuf-setup
> create mode 100644 tools/libxl/libxl_netbuffer.c
> create mode 100644 tools/libxl/libxl_nonetbuffer.c
> create mode 100644 tools/libxl/libxl_remus_device.c
>
--
Thanks,
Yang.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-05-28 10:01 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-05-21 10:14 [PATCH v10 0/5] Remus/Libxl: Network buffering support Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 1/5] libxl: introduce asynchronous execution API Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 2/5] remus: add libnl3 dependency for network buffering support Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 3/5] remus: introduce remus device Yang Hongyang
2014-05-21 10:14 ` [PATCH v10 4/5] remus: implement remus network buffering for nic devices Yang Hongyang
2014-05-21 10:15 ` [PATCH v10 5/5] libxl: network buffering cmdline switch Yang Hongyang
2014-05-28 9:22 ` [PATCH v10] remus drbd: Implement remus drbd replicated disk Yang Hongyang
2014-05-26 1:11 ` [PATCH v10 0/5] Remus/Libxl: Network buffering support Hongyang Yang
2014-05-28 10:01 ` Hongyang Yang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).