[PATCH RFC v3 RESEND 00/12] Migration with Local Disks Mirroring

[PATCH RFC v3 RESEND 00/12] Migration with Local Disks Mirroring

[Bruno Alvisio]

I have worked on a solution to be able to migrate domains that use QEMU as the
backend disk driver. I have adapted the migration flow and piggybacked on the
"drive-mirroring" capability already provided by QEMU.

Overview

1. The "xl migrate" command has an additional "-q" flag. When provided the local
storage of the domain is mirrored to the destination during the migration
process.

2. Internally, the modification consists on adding a new
libxl__stream_read_state struct to the libxl__domain_create_state structure and
libxl__stream_write_state struct to the libxl__domain_save_state struct.

3. Migration flow can now be divided into three phases:
   a. Phase One: Copies the necessary PFNs/params to start a QEMU process on the
      destination. QEMU is started with the "-incoming defer" option.
   b. Phase Two: Disk is mirrored using the QEMU embedded NBD server.
   c. Phase Three: Once the disk is completely mirrored, virtual RAM of the
      domain is live migrated to the destination. This phase most closely 
      resembles to the current migration flow.

4. If the “-q” option is not provided the migration is equivalent to the current
migration flow.

The new migration flow has follows the following major sequence of steps:
1. 1st stream copies the necessary PFNs and params from source to destination
to start the QEMU process in destination.
2. QEMU process is started on the destination with the option "-incoming defer".
(This creates the QEMU process but it doesn’t start running the main loop until
"migrate incoming" command is executed)
3. "drive mirror" QMP command is executed so that the disk is mirrored to the
destination node.
4. An event listener waits for the QMP BLOCK_JOB_READY event sent by QEMU which
signals that the "disk mirror job" is complete.
5. 2nd stream copies the virtual RAM from source to destination. At this point, 
the domain is suspended on source.
6. "migrate incoming" QMP command is executed in the destination node.
7. Domain is restored in destination.

Notes

1. Note that as of now "xen_platform_pci=0" for this feature to work. 
This is necessary so that the block devices are seen by QEMU. Further 
modification should be made for the case "xen_platform_pci=1" if we still 
want to use NBD mirroring capability provided by QEMU.
2. The current branch has still some hardcoded values but they can be easily
removed (I wanted initial feedback first):
    a. Port used for disks mirroring ("11000"): Changed by opening a socket and
       sending the port number to the source node.
    b. Name of the block devices ("ide0-hd0"): Currently the branch only 
       supports domains with one IDE drive. This constraint can easily be 
       removed by querying QEMU for the block-devices and checking their 
       backends on Xenstore. The name of the block devices to be mirrored 
       would be then sent to the destination node for starting the NBD server.
3.This feature needs a small patch to QEMU-XEN.

Here is a link to the Xen branch in Github:
https://github.com/balvisio/xen/tree/feature/migration_with_local_disks_mirroring

Here is a link to the QEMU-Xen branch in Github:
https://github.com/balvisio/qemu-xen/tree/feature/migration_with_local_disks_mirroring

Any feedback/suggestion is appreciated.

Cheers,

Bruno

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 01/12] Migration with Local Disks Mirroring: Added support in libxl to handle QMP events

[Bruno Alvisio]

Migration with Local Disk Mirroring uses the QEMU embedded NBD server. To
migrate the disk, a 'disk mirror job' is started from the source so that the
block devices emulated by QEMU are mirrored to the destination node. Once the
mirroring job is ready, QEMU sends an asynchronous QMP event. This code adds
support to register handlers when QMP events are received.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl_qmp.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index eab993a..0e993af 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -59,6 +59,13 @@ typedef struct callback_id_pair {
     LIBXL_STAILQ_ENTRY(struct callback_id_pair) next;
 } callback_id_pair;
 
+typedef struct event_handler_pair {
+    const char* event_type;
+    void *opaque;
+    qmp_request_context *context;
+    qmp_callback_t event_handler;
+} event_handler_pair;
+
 struct libxl__qmp_handler {
     struct sockaddr_un addr;
     int qmp_fd;
@@ -66,6 +73,9 @@ struct libxl__qmp_handler {
     time_t timeout;
     /* wait_for_id will be used by the synchronous send function */
     int wait_for_id;
+    /* wait_for_event_type is used to wait on QMP events */
+    const char* wait_for_event_type;
+    event_handler_pair *ehp;
 
     char buffer[QMP_RECEIVE_BUFFER_SIZE + 1];
     libxl__yajl_ctx *yajl_ctx;
@@ -287,6 +297,27 @@ static void qmp_handle_error_response(libxl__gc *gc, libxl__qmp_handler *qmp,
          libxl__json_object_get_string(resp));
 }
 
+static void qmp_handle_event(libxl__gc *gc, libxl__qmp_handler *qmp,
+                             const libxl__json_object *event)
+{
+    const char* event_type = NULL;
+    const libxl__json_object *event_o = NULL;
+    event_o = libxl__json_map_get("event", event, JSON_ANY);
+    event_type = libxl__json_object_get_string(event_o);
+    int rc;
+
+    if(qmp->wait_for_event_type &&
+        !strcmp(event_type, qmp->wait_for_event_type)) {
+        if(qmp->ehp->event_handler) {
+            rc = qmp->ehp->event_handler(qmp,
+                           libxl__json_map_get("data", event, JSON_ANY),
+                           qmp->ehp->opaque);
+        }
+        qmp->ehp->context->rc = rc;
+        qmp->wait_for_event_type = NULL;
+    }
+}
+
 static int qmp_handle_response(libxl__gc *gc, libxl__qmp_handler *qmp,
                                const libxl__json_object *resp)
 {
@@ -325,6 +356,7 @@ static int qmp_handle_response(libxl__gc *gc, libxl__qmp_handler *qmp,
         qmp_handle_error_response(gc, qmp, resp);
         return -1;
     case LIBXL__QMP_MESSAGE_TYPE_EVENT:
+        qmp_handle_event(gc, qmp, resp);
         return 0;
     case LIBXL__QMP_MESSAGE_TYPE_INVALID:
         return -1;
@@ -624,6 +656,32 @@ static void qmp_free_handler(libxl__qmp_handler *qmp)
     free(qmp);
 }
 
+static int __attribute__((unused)) wait_for_event(libxl__qmp_handler *qmp,
+                                                  event_handler_pair *ehp,
+                                                  int timeout)
+{
+    int ret = 0;
+    GC_INIT(qmp->ctx);
+    qmp->timeout = timeout;
+    qmp_request_context context = { .rc = 0 };
+    qmp->ehp = ehp;
+    qmp->wait_for_event_type = ehp->event_type;
+    ehp->context = &context;
+
+    while (qmp->wait_for_event_type) {
+        if ((ret = qmp_next(gc, qmp)) < 0) {
+            break;
+        }
+    }
+
+    if (!qmp->wait_for_event_type && ret == 0)
+        ret = context.rc;
+
+    GC_FREE;
+
+    return ret;
+}
+
 /*
  * QMP Parameters Helpers
  */
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 02/12] Migration with Local Disks Mirroring: Added QMP commands used for mirroring disks

[Bruno Alvisio]

Migration with local disks mirroring uses the embedded NBD server in QEMU. A NBD
server is added in libxl during instance creation time in the destination (QMP
command: nbd_server_add). The drive mirror command (QMP: driver_mirror) is
executed on the source during 'instance save' to replicate the disk in the
destination. When the local disk migration option is used, the QEMU process in
the destination will be started with the "-incoming defer" option instead of
"-incoming fd". Once the disk and memory have been transferred to the
destination the QEMU process will be resumed using the migrate incoming command
(QMP command: migrate_incoming).

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl_internal.h |  11 ++++
 tools/libxl/libxl_qmp.c      | 139 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 150 insertions(+)

diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index bfa95d8..4d7679e 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1830,6 +1830,17 @@ _hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid,
 /* Start replication */
 _hidden int libxl__qmp_start_replication(libxl__gc *gc, int domid,
                                          bool primary);
+/* Mirror drive */
+_hidden int libxl__qmp_drive_mirror(libxl__gc *gc, int domid,
+                                    const char* device, const char* target,
+                                    const char* format);
+/* Query block devices */
+_hidden int libxl__qmp_query_block(libxl__gc *gc, int domid,
+                                   char *device_names);
+/* Resume QEMU process started with -incoming defer option */
+_hidden int libxl__qmp_migrate_incoming(libxl__gc *gc, int domid,
+                                        const char* uri);
+
 /* Get replication error that occurs when the vm is running */
 _hidden int libxl__qmp_query_xen_replication_status(libxl__gc *gc, int domid);
 /* Do checkpoint */
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index 0e993af..1a1a318 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -1159,6 +1159,145 @@ int libxl__qmp_stop_replication(libxl__gc *gc, int domid, bool primary)
     return qmp_run_command(gc, domid, "xen-set-replication", args, NULL, NULL);
 }
 
+static int block_job_ready_handler(libxl__qmp_handler *qmp,
+                                   const libxl__json_object *data,
+                                   void *opaque)
+{
+    GC_INIT(qmp->ctx);
+    int rc = -1;
+
+    const char *type;
+    const char *device;
+    unsigned int len;
+    unsigned int offset;
+    unsigned int speed;
+
+    const libxl__json_object *obj = NULL;
+
+    obj = libxl__json_map_get("type", data, JSON_STRING);
+    if (!obj) {
+        LOGD(ERROR, qmp->domid, "Failed to retrieve job type.");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+    type = libxl__json_object_get_string(obj);
+
+    obj = libxl__json_map_get("device", data, JSON_STRING);
+    if (!obj) {
+        LOGD(ERROR, qmp->domid, "Failed to retrieve device.");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+    device = libxl__json_object_get_string(obj);
+
+    obj = libxl__json_map_get("len", data, JSON_INTEGER);
+    if (!obj) {
+        LOGD(ERROR, qmp->domid, "Failed to retrieve length.");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+    len = libxl__json_object_get_integer(obj);
+
+    obj = libxl__json_map_get("offset", data, JSON_INTEGER);
+    if (!obj) {
+        LOGD(ERROR, qmp->domid, "Failed to retrieve offset.");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+    offset = libxl__json_object_get_integer(obj);
+
+    obj = libxl__json_map_get("speed", data, JSON_INTEGER);
+    if (!obj) {
+        LOGD(ERROR, qmp->domid, "Failed to retrieve speed.");
+        rc = ERROR_FAIL;
+        goto out;
+    }
+    speed = libxl__json_object_get_integer(obj);
+
+    LOGD(INFO, qmp->domid, "Block Job Ready: Details: Device: %s, Type: %s, "
+                           "Len: %u, Offset: %u, Speed %u.", device, type,
+                           len, offset, speed);
+
+    rc = 0;
+out:
+    GC_FREE;
+    return rc;
+}
+
+int libxl__qmp_drive_mirror(libxl__gc *gc, int domid, const char* device,
+                            const char* target, const char* format)
+{
+    libxl__qmp_handler *qmp = NULL;
+    libxl__json_object *args = NULL;
+    int block_job_timeout = 3600;
+    int rc = 0;
+
+    qmp = libxl__qmp_initialize(gc, domid);
+    if (!qmp)
+        return -1;
+
+    qmp_parameters_add_string(gc, &args, "device", device);
+    qmp_parameters_add_string(gc, &args, "target", target);
+    qmp_parameters_add_string(gc, &args, "sync", "full");
+    qmp_parameters_add_string(gc, &args, "format", format);
+    qmp_parameters_add_string(gc, &args, "mode", "existing");
+    qmp_parameters_add_integer(gc, &args, "granularity", 0);
+    qmp_parameters_add_integer(gc, &args, "buf-size", 0);
+
+    rc = qmp_synchronous_send(qmp, "drive-mirror", args,
+                              NULL, NULL, qmp->timeout);
+
+    if ( !rc ) {
+        event_handler_pair hep = {
+            .event_type = "BLOCK_JOB_READY",
+            .event_handler = block_job_ready_handler,
+        };
+
+        rc = wait_for_event(qmp, &hep, block_job_timeout);
+    }
+    libxl__qmp_close(qmp);
+    return rc;
+}
+
+static int query_block_callback(libxl__qmp_handler *qmp,
+                                const libxl__json_object *response,
+                                void *opaque)
+{
+    GC_INIT(qmp->ctx);
+    const libxl__json_object *blockinfo = NULL;
+    int i, rc = -1;
+
+    for (i = 0; (blockinfo = libxl__json_array_get(response, i)); i++) {
+        const libxl__json_object *d;
+        const char* device_name;
+        d = libxl__json_map_get("device", blockinfo, JSON_STRING);
+        if (!d)
+            goto out;
+
+        device_name = libxl__json_object_get_string(d);
+    }
+
+    rc = 0;
+out:
+    GC_FREE;
+    return rc;
+}
+
+int libxl__qmp_query_block(libxl__gc *gc, int domid, char *device_names)
+{
+    return qmp_run_command(gc, domid, "query-block", NULL, query_block_callback,
+                           device_names);
+}
+
+int libxl__qmp_migrate_incoming(libxl__gc *gc, int domid, const char* uri)
+{
+    libxl__json_object *args = NULL;
+
+    qmp_parameters_add_string(gc, &args, "uri", uri);
+
+    return qmp_run_command(gc, domid, "migrate-incoming", args, NULL, NULL);
+}
+
 int libxl__qmp_nbd_server_stop(libxl__gc *gc, int domid)
 {
     return qmp_run_command(gc, domid, "nbd-server-stop", NULL, NULL, NULL);
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 03/12] Migration with Local Disks Mirroring: Refactored migrate_read_fixedmessage

[Bruno Alvisio]

The function migrate_fixed_message is going to be used in the libxl create and
save flow for event synchronization during migration. It needs to be accessible
from libxl_create and libxl_dom_save and thus it is moved to libxl_utils.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl_utils.c | 21 +++++++++++++++++++
 tools/libxl/libxl_utils.h |  3 +++
 tools/xl/xl_migrate.c     | 52 +++++++++++++++--------------------------------
 3 files changed, 40 insertions(+), 36 deletions(-)

diff --git a/tools/libxl/libxl_utils.c b/tools/libxl/libxl_utils.c
index 507ee56..5139320 100644
--- a/tools/libxl/libxl_utils.c
+++ b/tools/libxl/libxl_utils.c
@@ -510,6 +510,27 @@ int libxl__read_sysfs_file_contents(libxl__gc *gc, const char *filename,
 READ_WRITE_EXACTLY(read, 1, /* */)
 READ_WRITE_EXACTLY(write, 0, const)
 
+int libxl_read_fixedmessage(libxl_ctx *ctx, int fd, const void *msg, int msgsz,
+                            const char *what, const char *rune)
+{
+    char buf[msgsz];
+    const char *stream;
+    int rc;
+
+    stream = rune ? "migration receiver stream" : "migration stream";
+    rc = libxl_read_exactly(ctx, fd, buf, msgsz, stream, what);
+    if (rc) return 1;
+
+    if (memcmp(buf, msg, msgsz)) {
+        fprintf(stderr, "%s contained unexpected data instead of %s\n",
+                stream, what);
+        if (rune)
+            fprintf(stderr, "(command run was: %s )\n", rune);
+        return 1;
+    }
+    return 0;
+}
+
 int libxl__remove_file(libxl__gc *gc, const char *path)
 {
     for (;;) {
diff --git a/tools/libxl/libxl_utils.h b/tools/libxl/libxl_utils.h
index 9e743dc..d1e80ef 100644
--- a/tools/libxl/libxl_utils.h
+++ b/tools/libxl/libxl_utils.h
@@ -56,6 +56,9 @@ int libxl_write_exactly(libxl_ctx *ctx, int fd, const void *data,
    * logged using filename (which is only used for logging) and what
    * (which may be 0). */
 
+int libxl_read_fixedmessage(libxl_ctx *ctx, int fd, const void *msg, int msgsz,
+                            const char *what, const char *rune);
+
 int libxl_pipe(libxl_ctx *ctx, int pipes[2]);
   /* Just like pipe(2), but log errors. */
 
diff --git a/tools/xl/xl_migrate.c b/tools/xl/xl_migrate.c
index 1f0e87d..33d39e8 100644
--- a/tools/xl/xl_migrate.c
+++ b/tools/xl/xl_migrate.c
@@ -68,26 +68,6 @@ static pid_t create_migration_child(const char *rune, int *send_fd,
     return child;
 }
 
-static int migrate_read_fixedmessage(int fd, const void *msg, int msgsz,
-                                     const char *what, const char *rune) {
-    char buf[msgsz];
-    const char *stream;
-    int rc;
-
-    stream = rune ? "migration receiver stream" : "migration stream";
-    rc = libxl_read_exactly(ctx, fd, buf, msgsz, stream, what);
-    if (rc) return 1;
-
-    if (memcmp(buf, msg, msgsz)) {
-        fprintf(stderr, "%s contained unexpected data instead of %s\n",
-                stream, what);
-        if (rune)
-            fprintf(stderr, "(command run was: %s )\n", rune);
-        return 1;
-    }
-    return 0;
-}
-
 static void migration_child_report(int recv_fd) {
     pid_t child;
     int status, sr;
@@ -162,9 +142,9 @@ static void migrate_do_preamble(int send_fd, int recv_fd, pid_t child,
         exit(EXIT_FAILURE);
     }
 
-    rc = migrate_read_fixedmessage(recv_fd, migrate_receiver_banner,
-                                   sizeof(migrate_receiver_banner)-1,
-                                   "banner", rune);
+    rc = libxl_read_fixedmessage(ctx, recv_fd, migrate_receiver_banner,
+                                 sizeof(migrate_receiver_banner)-1,
+                                 "banner", rune);
     if (rc) {
         close(send_fd);
         migration_child_report(recv_fd);
@@ -219,9 +199,9 @@ static void migrate_domain(uint32_t domid, const char *rune, int debug,
     // Should only be printed when debugging as it's a bit messy with
     // progress indication.
 
-    rc = migrate_read_fixedmessage(recv_fd, migrate_receiver_ready,
-                                   sizeof(migrate_receiver_ready),
-                                   "ready message", rune);
+    rc = libxl_read_fixedmessage(ctx, recv_fd, migrate_receiver_ready,
+                                 sizeof(migrate_receiver_ready),
+                                 "ready message", rune);
     if (rc) goto failed_resume;
 
     xtl_stdiostream_adjust_flags(logger, 0, XTL_STDIOSTREAM_HIDE_PROGRESS);
@@ -251,9 +231,9 @@ static void migrate_domain(uint32_t domid, const char *rune, int debug,
                              "migration stream", "GO message");
     if (rc) goto failed_badly;
 
-    rc = migrate_read_fixedmessage(recv_fd, migrate_report,
-                                   sizeof(migrate_report),
-                                   "success/failure report message", rune);
+    rc = libxl_read_fixedmessage(ctx, recv_fd, migrate_report,
+                                 sizeof(migrate_report),
+                                 "success/failure report message", rune);
     if (rc) goto failed_badly;
 
     rc = libxl_read_exactly(ctx, recv_fd,
@@ -265,10 +245,10 @@ static void migrate_domain(uint32_t domid, const char *rune, int debug,
         fprintf(stderr, "migration sender: Target reports startup failure"
                 " (status code %d).\n", rc_buf);
 
-        rc = migrate_read_fixedmessage(recv_fd, migrate_permission_to_go,
-                                       sizeof(migrate_permission_to_go),
-                                       "permission for sender to resume",
-                                       rune);
+        rc = libxl_read_fixedmessage(ctx, recv_fd, migrate_permission_to_go,
+                                     sizeof(migrate_permission_to_go),
+                                     "permission for sender to resume",
+                                     rune);
         if (rc) goto failed_badly;
 
         fprintf(stderr, "migration sender: Trying to resume at our end.\n");
@@ -416,9 +396,9 @@ static void migrate_receive(int debug, int daemonize, int monitor,
                              "migration ack stream", "ready message");
     if (rc) exit(EXIT_FAILURE);
 
-    rc = migrate_read_fixedmessage(recv_fd, migrate_permission_to_go,
-                                   sizeof(migrate_permission_to_go),
-                                   "GO message", 0);
+    rc = libxl_read_fixedmessage(ctx, recv_fd, migrate_permission_to_go,
+                                 sizeof(migrate_permission_to_go),
+                                 "GO message", 0);
     if (rc) goto perhaps_destroy_notify_rc;
 
     fprintf(stderr, "migration target: Got permission, starting domain.\n");
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 04/12] Migration with Local Disks Mirroring: Added a new '-q' flag to xl migrate for disk mirorring

[Bruno Alvisio]

When the '-q' flag is provided to the 'xl migrate' command all the block devices
that are local should be mirrored to the destination node. If the flag is not
present migration flow will be equivalent to current migration flow. A new
'mirror_disks' field is added to the xl domain_create struct to indicate that
disks will be mirrored during instance startup. A new LIBXL_SUSPEND_MIRROR_DISK
flag is added to indicate that the block mirror jobs should be performed during
the suspension of the instance. libxl_domain_suspend takes a new 'hostname'
param with the name of the host where the QEMU drives will be mirrored to.
libxl_domain_suspend takes a 'recv_fd' param that is used for receving messages
from destination during migration.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl.h                  |  4 +-
 tools/libxl/libxl_domain.c           |  4 +-
 tools/ocaml/libs/xl/xenlight_stubs.c |  2 +-
 tools/xl/xl.h                        |  1 +
 tools/xl/xl_cmdtable.c               |  3 +-
 tools/xl/xl_migrate.c                | 74 +++++++++++++++++++++++++++++++-----
 tools/xl/xl_saverestore.c            |  2 +-
 7 files changed, 75 insertions(+), 15 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 5e9aed7..7828ba8 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -1444,12 +1444,14 @@ int libxl_retrieve_domain_configuration(libxl_ctx *ctx, uint32_t domid,
                                         libxl_domain_config *d_config)
                                         LIBXL_EXTERNAL_CALLERS_ONLY;
 
-int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd,
+int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int recv_fd,
                          int flags, /* LIBXL_SUSPEND_* */
+                         const char* hostname,
                          const libxl_asyncop_how *ao_how)
                          LIBXL_EXTERNAL_CALLERS_ONLY;
 #define LIBXL_SUSPEND_DEBUG 1
 #define LIBXL_SUSPEND_LIVE 2
+#define LIBXL_SUSPEND_MIRROR_DISKS 4
 
 /* @param suspend_cancel [from xenctrl.h:xc_domain_resume( @param fast )]
  *   If this parameter is true, use co-operative resume. The guest
diff --git a/tools/libxl/libxl_domain.c b/tools/libxl/libxl_domain.c
index 814f812..d1209cd 100644
--- a/tools/libxl/libxl_domain.c
+++ b/tools/libxl/libxl_domain.c
@@ -486,7 +486,8 @@ static void domain_suspend_cb(libxl__egc *egc,
 
 }
 
-int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags,
+int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int recv_fd,
+                         int flags, const char* hostname,
                          const libxl_asyncop_how *ao_how)
 {
     AO_CREATE(ctx, domid, ao_how);
@@ -506,6 +507,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int flags,
 
     dss->domid = domid;
     dss->fd = fd;
+    dss->recv_fd = recv_fd;
     dss->type = type;
     dss->live = flags & LIBXL_SUSPEND_LIVE;
     dss->debug = flags & LIBXL_SUSPEND_DEBUG;
diff --git a/tools/ocaml/libs/xl/xenlight_stubs.c b/tools/ocaml/libs/xl/xenlight_stubs.c
index 0140780..a757782 100644
--- a/tools/ocaml/libs/xl/xenlight_stubs.c
+++ b/tools/ocaml/libs/xl/xenlight_stubs.c
@@ -611,7 +611,7 @@ value stub_libxl_domain_suspend(value ctx, value domid, value fd, value async, v
 	libxl_asyncop_how *ao_how = aohow_val(async);
 
 	caml_enter_blocking_section();
-	ret = libxl_domain_suspend(CTX, c_domid, c_fd, 0, ao_how);
+	ret = libxl_domain_suspend(CTX, c_domid, c_fd, 0, 0, NULL, ao_how);
 	caml_leave_blocking_section();
 
 	free(ao_how);
diff --git a/tools/xl/xl.h b/tools/xl/xl.h
index 6b60d1d..fd8e8e6 100644
--- a/tools/xl/xl.h
+++ b/tools/xl/xl.h
@@ -35,6 +35,7 @@ struct domain_create {
     int daemonize;
     int monitor; /* handle guest reboots etc */
     int paused;
+    int mirror_disks;
     int dryrun;
     int quiet;
     int vnc;
diff --git a/tools/xl/xl_cmdtable.c b/tools/xl/xl_cmdtable.c
index 5546cf6..45a0b1a 100644
--- a/tools/xl/xl_cmdtable.c
+++ b/tools/xl/xl_cmdtable.c
@@ -165,7 +165,8 @@ struct cmd_spec cmd_table[] = {
       "-e              Do not wait in the background (on <host>) for the death\n"
       "                of the domain.\n"
       "--debug         Print huge (!) amount of debug during the migration process.\n"
-      "-p              Do not unpause domain after migrating it."
+      "-p              Do not unpause domain after migrating it.\n"
+      "-q              Mirror local disks to destination - Copy all local storage devices."
     },
     { "restore",
       &main_restore, 0, 1,
diff --git a/tools/xl/xl_migrate.c b/tools/xl/xl_migrate.c
index 33d39e8..48b0179 100644
--- a/tools/xl/xl_migrate.c
+++ b/tools/xl/xl_migrate.c
@@ -157,7 +157,8 @@ static void migrate_do_preamble(int send_fd, int recv_fd, pid_t child,
 }
 
 static void migrate_domain(uint32_t domid, const char *rune, int debug,
-                           const char *override_config_file)
+                           const char *override_config_file,
+                           int mirror_disks, const char* hostname)
 {
     pid_t child = -1;
     int rc;
@@ -185,7 +186,10 @@ static void migrate_domain(uint32_t domid, const char *rune, int debug,
 
     if (debug)
         flags |= LIBXL_SUSPEND_DEBUG;
-    rc = libxl_domain_suspend(ctx, domid, send_fd, flags, NULL);
+    if (mirror_disks)
+        flags |= LIBXL_SUSPEND_MIRROR_DISKS;
+    rc = libxl_domain_suspend(ctx, domid, send_fd, recv_fd, flags, hostname,
+                              NULL);
     if (rc) {
         fprintf(stderr, "migration sender: libxl_domain_suspend failed"
                 " (rc=%d)\n", rc);
@@ -296,7 +300,7 @@ static void migrate_domain(uint32_t domid, const char *rune, int debug,
 }
 
 static void migrate_receive(int debug, int daemonize, int monitor,
-                            int pause_after_migration,
+                            int pause_after_migration, int mirror_disks,
                             int send_fd, int recv_fd,
                             libxl_checkpointed_stream checkpointed,
                             char *colo_proxy_script,
@@ -323,6 +327,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
     dom_info.daemonize = daemonize;
     dom_info.monitor = monitor;
     dom_info.paused = 1;
+    dom_info.mirror_disks = mirror_disks;
     dom_info.migrate_fd = recv_fd;
     dom_info.send_back_fd = send_fd;
     dom_info.migration_domname_r = &migration_domname;
@@ -458,6 +463,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
 int main_migrate_receive(int argc, char **argv)
 {
     int debug = 0, daemonize = 1, monitor = 1, pause_after_migration = 0;
+    int mirror_disks = 0;
     libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
     int opt;
     bool userspace_colo_proxy = false;
@@ -470,7 +476,7 @@ int main_migrate_receive(int argc, char **argv)
         COMMON_LONG_OPTS
     };
 
-    SWITCH_FOREACH_OPT(opt, "Fedrp", opts, "migrate-receive", 0) {
+    SWITCH_FOREACH_OPT(opt, "Fedrpq", opts, "migrate-receive", 0) {
     case 'F':
         daemonize = 0;
         break;
@@ -496,6 +502,9 @@ int main_migrate_receive(int argc, char **argv)
     case 'p':
         pause_after_migration = 1;
         break;
+    case 'q':
+        mirror_disks = 1;
+        break;
     }
 
     if (argc-optind != 0) {
@@ -503,7 +512,7 @@ int main_migrate_receive(int argc, char **argv)
         return EXIT_FAILURE;
     }
     migrate_receive(debug, daemonize, monitor, pause_after_migration,
-                    STDOUT_FILENO, STDIN_FILENO,
+                    mirror_disks, STDOUT_FILENO, STDIN_FILENO,
                     checkpointed, script, userspace_colo_proxy);
 
     return EXIT_SUCCESS;
@@ -512,18 +521,22 @@ int main_migrate_receive(int argc, char **argv)
 int main_migrate(int argc, char **argv)
 {
     uint32_t domid;
+    libxl_domain_config d_config;
     const char *config_filename = NULL;
     const char *ssh_command = "ssh";
     char *rune = NULL;
     char *host;
-    int opt, daemonize = 1, monitor = 1, debug = 0, pause_after_migration = 0;
+    char *hostname;
+    int opt, daemonize = 1, monitor = 1, debug = 0, pause_after_migration = 0,
+    mirror_disks = 0;
+    int rc;
     static struct option opts[] = {
         {"debug", 0, 0, 0x100},
         {"live", 0, 0, 0x200},
         COMMON_LONG_OPTS
     };
 
-    SWITCH_FOREACH_OPT(opt, "FC:s:ep", opts, "migrate", 2) {
+    SWITCH_FOREACH_OPT(opt, "FC:s:epq", opts, "migrate", 2) {
     case 'C':
         config_filename = optarg;
         break;
@@ -540,6 +553,9 @@ int main_migrate(int argc, char **argv)
     case 'p':
         pause_after_migration = 1;
         break;
+    case 'q':
+        mirror_disks = 1;
+        break;
     case 0x100: /* --debug */
         debug = 1;
         break;
@@ -551,6 +567,33 @@ int main_migrate(int argc, char **argv)
     domid = find_domain(argv[optind]);
     host = argv[optind + 1];
 
+    hostname = strchr(host, '@');
+    hostname++;
+
+    if (mirror_disks) {
+        libxl_domain_config_init(&d_config);
+        rc = libxl_retrieve_domain_configuration(ctx, domid, &d_config);
+        if (rc) {
+            fprintf(stderr, "unable to retrieve domain configuration\n");
+            goto failed_config;
+        }
+        libxl_domain_build_info b_info = d_config.b_info;
+        libxl_device_model_version model_version = b_info.device_model_version;
+        libxl_domain_type type = b_info.type;
+
+        if (model_version != LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN)
+            goto failed_mirror;
+
+        if (type != LIBXL_DOMAIN_TYPE_HVM) {
+            goto failed_mirror;
+        } else {
+            libxl_defbool_setdefault(&b_info.u.hvm.xen_platform_pci, true);
+            if (libxl_defbool_val(b_info.u.hvm.xen_platform_pci))
+                goto failed_mirror;
+        }
+        libxl_domain_config_dispose(&d_config);
+    }
+
     bool pass_tty_arg = progress_use_cr || (isatty(2) > 0);
 
     if (!ssh_command[0]) {
@@ -567,17 +610,28 @@ int main_migrate(int argc, char **argv)
         } else {
             verbose_len = (minmsglevel_default - minmsglevel) + 2;
         }
-        xasprintf(&rune, "exec %s %s xl%s%.*s migrate-receive%s%s%s",
+        xasprintf(&rune, "exec %s %s xl%s%.*s migrate-receive%s%s%s%s",
                   ssh_command, host,
                   pass_tty_arg ? " -t" : "",
                   verbose_len, verbose_buf,
                   daemonize ? "" : " -e",
                   debug ? " -d" : "",
-                  pause_after_migration ? " -p" : "");
+                  pause_after_migration ? " -p" : "",
+                  mirror_disks ? " -q" : "");
     }
 
-    migrate_domain(domid, rune, debug, config_filename);
+    migrate_domain(domid, rune, debug, config_filename, mirror_disks,
+                   hostname);
     return EXIT_SUCCESS;
+
+ failed_mirror:
+    fprintf(stderr, "Migration with local disks mirroring is only supported "
+                    "for HVM using QEMU_XEN as device model version and "
+                    "xen_platform_pci = 1\n");
+
+ failed_config:
+    libxl_domain_config_dispose(&d_config);
+    exit(EXIT_FAILURE);
 }
 
 int main_remus(int argc, char **argv)
diff --git a/tools/xl/xl_saverestore.c b/tools/xl/xl_saverestore.c
index 9afeade..9215a45 100644
--- a/tools/xl/xl_saverestore.c
+++ b/tools/xl/xl_saverestore.c
@@ -141,7 +141,7 @@ static int save_domain(uint32_t domid, const char *filename, int checkpoint,
 
     save_domain_core_writeconfig(fd, filename, config_data, config_len);
 
-    int rc = libxl_domain_suspend(ctx, domid, fd, 0, NULL);
+    int rc = libxl_domain_suspend(ctx, domid, fd, 0, 0, NULL, NULL);
     close(fd);
 
     if (rc < 0) {
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 05/12] Migration with Local Disks Mirroring: QEMU process is started with '-incoming defer' option

[Bruno Alvisio]

For the migration with local disks mirroring scenario, the QEMU process is
started before the virtual RAM is transferred to the destination node so that
the QEMU embedded NBD server and disks mirroring jobs can be started. After the
virtual RAM and QEMU state are transferred, the QEMU process will be resumed
using the QMP 'migrate-incoming' command.

When qemu-xen is started with '-incoming defer' option the initial state of the
instance will be 'inmigrate'. Thus, when either 'running' or 'inmigrate' are
read from xenstore the spawn detach is carried out.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl_dm.c       | 20 ++++++++++++++------
 tools/libxl/libxl_internal.h |  1 +
 2 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index a2ea95a..e12e015 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -940,7 +940,7 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
                                         const libxl_domain_config *guest_config,
                                         char ***args, char ***envs,
                                         const libxl__domain_build_state *state,
-                                        int *dm_state_fd)
+                                        int *dm_state_fd, int mirror_disks)
 {
     const libxl_domain_create_info *c_info = &guest_config->c_info;
     const libxl_domain_build_info *b_info = &guest_config->b_info;
@@ -1431,6 +1431,12 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
         flexarray_append(dm_args, "-incoming");
         flexarray_append(dm_args, GCSPRINTF("fd:%d",*dm_state_fd));
     }
+
+    if (mirror_disks) {
+        flexarray_append(dm_args, "-incoming");
+        flexarray_append(dm_args, "defer");
+    }
+
     for (i = 0; b_info->extra && b_info->extra[i] != NULL; i++)
         flexarray_append(dm_args, b_info->extra[i]);
 
@@ -1728,7 +1734,7 @@ static int libxl__build_device_model_args(libxl__gc *gc,
                                         const libxl_domain_config *guest_config,
                                         char ***args, char ***envs,
                                         const libxl__domain_build_state *state,
-                                        int *dm_state_fd)
+                                        int *dm_state_fd, int mirror_disks)
 /* dm_state_fd may be NULL iff caller knows we are using old stubdom
  * and therefore will be passing a filename rather than a fd. */
 {
@@ -1744,7 +1750,8 @@ static int libxl__build_device_model_args(libxl__gc *gc,
         return libxl__build_device_model_args_new(gc, dm,
                                                   guest_domid, guest_config,
                                                   args, envs,
-                                                  state, dm_state_fd);
+                                                  state, dm_state_fd,
+                                                  mirror_disks);
     default:
         LOGED(ERROR, guest_domid, "unknown device model version %d",
               guest_config->b_info.device_model_version);
@@ -1964,7 +1971,7 @@ void libxl__spawn_stub_dm(libxl__egc *egc, libxl__stub_dm_spawn_state *sdss)
 
     ret = libxl__build_device_model_args(gc, "stubdom-dm", guest_domid,
                                          guest_config, &args, NULL,
-                                         d_state, NULL);
+                                         d_state, NULL, 0);
     if (ret) {
         ret = ERROR_FAIL;
         goto out;
@@ -2267,7 +2274,8 @@ void libxl__spawn_local_dm(libxl__egc *egc, libxl__dm_spawn_state *dmss)
     }
     rc = libxl__build_device_model_args(gc, dm, domid, guest_config,
                                           &args, &envs, state,
-                                          &dm_state_fd);
+                                          &dm_state_fd,
+                                          dmss->mirror_disks);
     if (rc)
         goto out;
 
@@ -2397,7 +2405,7 @@ static void device_model_confirm(libxl__egc *egc, libxl__spawn_state *spawn,
     if (!xsdata)
         return;
 
-    if (strcmp(xsdata, "running"))
+    if (strcmp(xsdata, "running") && strcmp(xsdata, "inmigrate"))
         return;
 
     libxl__spawn_initiate_detach(gc, spawn);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 4d7679e..e8a4dc2 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3711,6 +3711,7 @@ struct libxl__dm_spawn_state {
     libxl__spawn_state spawn;
     /* filled in by user, must remain valid: */
     uint32_t guest_domid; /* domain being served */
+    int mirror_disks;
     libxl_domain_config *guest_config;
     libxl__domain_build_state *build_state; /* relates to guest_domid */
     libxl__dm_spawn_cb *callback;
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 06/12] Migration with Local Disks Mirroring: Added 'mirror_disks' field to domain_create_state

[Bruno Alvisio]

A new field 'mirror_disks' is added to the libxl struct
libxl__domain_create_state to record if QEMU drives should be mirrored during
migration. This variable is used to setup the sequence of calls and streams in
libxl and libxc.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl.h                  | 4 ++--
 tools/libxl/libxl_create.c           | 8 +++++---
 tools/libxl/libxl_internal.h         | 1 +
 tools/ocaml/libs/xl/xenlight_stubs.c | 2 +-
 tools/xl/xl_vmcontrol.c              | 5 +++--
 5 files changed, 12 insertions(+), 8 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 7828ba8..25245cc 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -1371,7 +1371,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
                             LIBXL_EXTERNAL_CALLERS_ONLY;
 int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
                                 uint32_t *domid, int restore_fd,
-                                int send_back_fd,
+                                int send_back_fd, int mirror_disks,
                                 const libxl_domain_restore_params *params,
                                 const libxl_asyncop_how *ao_how,
                                 const libxl_asyncprogress_how *aop_console_how)
@@ -1412,7 +1412,7 @@ static inline int libxl_domain_create_restore_0x040400(
     LIBXL_EXTERNAL_CALLERS_ONLY
 {
     return libxl_domain_create_restore(ctx, d_config, domid, restore_fd,
-                                       -1, params, ao_how, aop_console_how);
+                                       -1, 0, params, ao_how, aop_console_how);
 }
 
 #define libxl_domain_create_restore libxl_domain_create_restore_0x040400
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f15fb21..95978a8 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1639,6 +1639,7 @@ static void domain_create_cb(libxl__egc *egc,
 
 static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
                             uint32_t *domid, int restore_fd, int send_back_fd,
+                            int mirror_disks,
                             const libxl_domain_restore_params *params,
                             const libxl_asyncop_how *ao_how,
                             const libxl_asyncprogress_how *aop_console_how)
@@ -1654,6 +1655,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
     libxl_domain_config_copy(ctx, &cdcs->dcs.guest_config_saved, d_config);
     cdcs->dcs.restore_fd = cdcs->dcs.libxc_fd = restore_fd;
     cdcs->dcs.send_back_fd = send_back_fd;
+    cdcs->dcs.mirror_disks = mirror_disks;
     if (restore_fd > -1) {
         cdcs->dcs.restore_params = *params;
         rc = libxl__fd_flags_modify_save(gc, cdcs->dcs.restore_fd,
@@ -1881,13 +1883,13 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
                             const libxl_asyncprogress_how *aop_console_how)
 {
     unset_disk_colo_restore(d_config);
-    return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
+    return do_domain_create(ctx, d_config, domid, -1, -1, 0, NULL,
                             ao_how, aop_console_how);
 }
 
 int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
                                 uint32_t *domid, int restore_fd,
-                                int send_back_fd,
+                                int send_back_fd, int mirror_disks,
                                 const libxl_domain_restore_params *params,
                                 const libxl_asyncop_how *ao_how,
                                 const libxl_asyncprogress_how *aop_console_how)
@@ -1899,7 +1901,7 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
     }
 
     return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
-                            params, ao_how, aop_console_how);
+                            mirror_disks, params, ao_how, aop_console_how);
 }
 
 int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e8a4dc2..ee62bfb 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3757,6 +3757,7 @@ struct libxl__domain_create_state {
     int restore_fd, libxc_fd;
     int restore_fdfl; /* original flags of restore_fd */
     int send_back_fd;
+    int mirror_disks;
     libxl_domain_restore_params restore_params;
     uint32_t domid_soft_reset;
     libxl__domain_create_cb *callback;
diff --git a/tools/ocaml/libs/xl/xenlight_stubs.c b/tools/ocaml/libs/xl/xenlight_stubs.c
index a757782..900d7f1 100644
--- a/tools/ocaml/libs/xl/xenlight_stubs.c
+++ b/tools/ocaml/libs/xl/xenlight_stubs.c
@@ -538,7 +538,7 @@ value stub_libxl_domain_create_restore(value ctx, value domain_config, value par
 
 	caml_enter_blocking_section();
 	ret = libxl_domain_create_restore(CTX, &c_dconfig, &c_domid, restore_fd,
-		-1, &c_params, ao_how, NULL);
+		-1, 0, &c_params, ao_how, NULL);
 	caml_leave_blocking_section();
 
 	free(ao_how);
diff --git a/tools/xl/xl_vmcontrol.c b/tools/xl/xl_vmcontrol.c
index 89c2b25..c22c6fd 100644
--- a/tools/xl/xl_vmcontrol.c
+++ b/tools/xl/xl_vmcontrol.c
@@ -882,8 +882,9 @@ start:
 
         ret = libxl_domain_create_restore(ctx, &d_config,
                                           &domid, restore_fd,
-                                          send_back_fd, &params,
-                                          0, autoconnect_console_how);
+                                          send_back_fd,
+                                          dom_info->mirror_disks,
+                                          &params, 0, autoconnect_console_how);
 
         libxl_domain_restore_params_dispose(&params);
 
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 07/12] Migration with Local Disks Mirroring: Added new libxl_read_stream and callbacks in restore flow

[Bruno Alvisio]

For migration with local disks mirroring a QEMU NBD server is started while
restoring the domain. This server will be responsible for receiving the disks
sent from the source node: The disks transfer will be triggered during the
domain save on the source using the QMP drive-mirror command.

A second libxl_read_stream is created (srs_mirror_disks): During domain restore
the first libxc stream transfers the necessary PFNs and params only so that the
QEMU process and NBD server can be started. After the disk mirroring jobs are
completed the second libxl_read_stream is started and the virtual RAM is
transferred.

In the migration with disks mirroring case:
1. domcreate_devmodel_deferred_started is the callback function for launch_dm

2. domcreate_stream_done is the callback function for the end of 1st
libxl_read_stream

3. domcreate_post_mirror_disks_stream_done is the callback function for the 2nd
libxl_read_stream

The original libxl_create calling sequence:
1. domcreate_bootloader_done
2. libxl/libxc stream
3. domcreate_stream_done
4. domcreate_pre_build
5. domcreate_rebuild_done
6. domcreate_multidev_begin
7. domcreate_launch_dm
8. domcreate_devmodel_started

In the case of migration with local disk mirroring the sequence becomes:

1. domcreate_bootloader_done
2. pre mirror disk libxl/libxc stream: Transfers PFNs/params necessary for
QEMU initilization
3. domcreate_stream_done
4. domcreate_pre_build
5. domcreate_rebuild_done
6. domcreate_multidev_begin
7. domcreate_launch_dm: Starts QEMU with "-incoming defer option"
8. domcreate_devmodel_deferred_started: Starts NBD server for disks mirroring
9. post mirror disks libxl/libxc stream: Transfers all virtual RAM: Similar to
original stream
10. domcreate_post_mirror_disks_steam_done: Resumes QEMU process by
executing QMP "migrate-incoming" command
11. domcreate_devmodel_started

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl.h          |   6 +++
 tools/libxl/libxl_create.c   | 117 +++++++++++++++++++++++++++++++++++++++++--
 tools/libxl/libxl_internal.h |   2 +
 3 files changed, 120 insertions(+), 5 deletions(-)

diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 25245cc..af2aa9a 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -1453,6 +1453,12 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int recv_fd,
 #define LIBXL_SUSPEND_LIVE 2
 #define LIBXL_SUSPEND_MIRROR_DISKS 4
 
+#define DRIVE_MIRROR_PORT "11000"
+#define DRIVE_MIRROR_DEVICE "ide0-hd0"
+
+static const char nbd_server_started_banner[]=
+    "nbd server started on source, start mirror job.\n";
+
 /* @param suspend_cancel [from xenctrl.h:xc_domain_resume( @param fast )]
  *   If this parameter is true, use co-operative resume. The guest
  *   must support this.
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 95978a8..f834282 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -761,6 +761,9 @@ static int store_libxl_entry(libxl__gc *gc, uint32_t domid,
 static void domcreate_devmodel_started(libxl__egc *egc,
                                        libxl__dm_spawn_state *dmss,
                                        int rc);
+static void domcreate_devmodel_deferred_started(libxl__egc *egc,
+                                                libxl__dm_spawn_state *dmss,
+                                                int ret);
 static void domcreate_bootloader_console_available(libxl__egc *egc,
                                                    libxl__bootloader_state *bl);
 static void domcreate_bootloader_done(libxl__egc *egc,
@@ -777,6 +780,10 @@ static void domcreate_stream_done(libxl__egc *egc,
                                   libxl__stream_read_state *srs,
                                   int ret);
 
+static void domcreate_post_mirror_disks_stream_done(libxl__egc *egc,
+                                                    libxl__stream_read_state *srs,
+                                                    int ret);
+
 static void domcreate_rebuild_done(libxl__egc *egc,
                                    libxl__domain_create_state *dcs,
                                    int ret);
@@ -1036,6 +1043,48 @@ static void libxl__colo_restore_setup_done(libxl__egc *egc,
     libxl__stream_read_start(egc, &dcs->srs);
 }
 
+static void domcreate_devmodel_deferred_started(libxl__egc *egc,
+                                                libxl__dm_spawn_state *dmss,
+                                                int ret)
+{
+    libxl__domain_create_state *dcs = CONTAINER_OF(dmss, *dcs, sdss.dm);
+    STATE_AO_GC(dmss->spawn.ao);
+    const uint32_t domid = dcs->guest_domid;
+    dcs->sdss.dm.guest_domid = domid;
+
+    if (ret) {
+        LOGD(ERROR, domid, "device model did not start: %d", ret);
+        goto error_out;
+    }
+
+    ret = libxl__qmp_nbd_server_start(gc, domid, "::", DRIVE_MIRROR_PORT);
+    if (ret) {
+        LOGD(ERROR, domid, "Failed to start NBD Server");
+        goto error_out;
+    }
+
+    ret = libxl__qmp_nbd_server_add(gc, domid, DRIVE_MIRROR_DEVICE);
+    if (ret) {
+        LOGD(ERROR, domid, "Failed to add NBD Server");
+        goto error_out;
+    }
+
+    ret = libxl_write_exactly(CTX, dcs->send_back_fd,
+                              nbd_server_started_banner,
+                              sizeof(nbd_server_started_banner)-1,
+                              "migration stream",
+                              "nbd server success/failure report");
+    if (ret)
+        goto error_out;
+
+    libxl__stream_read_start(egc, &dcs->srs);
+    return;
+
+error_out:
+    assert(ret);
+    domcreate_complete(egc, dcs, ret);
+}
+
 static void domcreate_bootloader_done(libxl__egc *egc,
                                       libxl__bootloader_state *bl,
                                       int rc)
@@ -1053,6 +1102,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
     libxl_domain_build_info *const info = &d_config->b_info;
     libxl__srm_restore_autogen_callbacks *const callbacks =
         &dcs->srs.shs.callbacks.restore.a;
+    libxl__srm_restore_autogen_callbacks *const callbacks_mirror_disks =
+        &dcs->srs_mirror_disks.shs.callbacks.restore.a;
 
     if (rc) {
         domcreate_rebuild_done(egc, dcs, rc);
@@ -1070,8 +1121,14 @@ static void domcreate_bootloader_done(libxl__egc *egc,
     dcs->sdss.dm.spawn.ao = ao;
     dcs->sdss.dm.guest_config = dcs->guest_config;
     dcs->sdss.dm.build_state = &dcs->build_state;
-    dcs->sdss.dm.callback = domcreate_devmodel_started;
-    dcs->sdss.callback = domcreate_devmodel_started;
+    dcs->sdss.dm.mirror_disks = dcs->mirror_disks;
+    if (!dcs->mirror_disks) {
+        dcs->sdss.dm.callback = domcreate_devmodel_started;
+        dcs->sdss.callback = domcreate_devmodel_started;
+    } else {
+        dcs->sdss.dm.callback = domcreate_devmodel_deferred_started;
+        dcs->sdss.callback = domcreate_devmodel_deferred_started;
+    }
 
     if (restore_fd < 0 && dcs->domid_soft_reset == INVALID_DOMID) {
         rc = libxl__domain_build(gc, d_config, domid, state);
@@ -1081,6 +1138,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 
     /* Restore */
     callbacks->restore_results = libxl__srm_callout_callback_restore_results;
+    callbacks_mirror_disks->restore_results = libxl__srm_callout_callback_restore_results;
 
     /* COLO only supports HVM now because it does not work very
      * well with pv drivers:
@@ -1106,7 +1164,21 @@ static void domcreate_bootloader_done(libxl__egc *egc,
     dcs->srs.fd = restore_fd;
     dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
     dcs->srs.back_channel = false;
-    dcs->srs.completion_callback = domcreate_stream_done;
+    dcs->srs.mirror_disks = 0;
+
+    if (dcs->mirror_disks) {
+        dcs->srs_mirror_disks.ao = ao;
+        dcs->srs_mirror_disks.dcs = dcs;
+        dcs->srs_mirror_disks.fd = restore_fd;
+        dcs->srs_mirror_disks.legacy = (dcs->restore_params.stream_version == 1);
+        dcs->srs_mirror_disks.back_channel = false;
+        dcs->srs_mirror_disks.completion_callback = domcreate_stream_done;
+        dcs->srs_mirror_disks.mirror_disks = 1;
+
+        dcs->srs.completion_callback = domcreate_post_mirror_disks_stream_done;
+    } else {
+        dcs->srs.completion_callback = domcreate_stream_done;
+    }
 
     if (restore_fd >= 0) {
         switch (checkpointed_stream) {
@@ -1124,7 +1196,11 @@ static void domcreate_bootloader_done(libxl__egc *egc,
             libxl__remus_restore_setup(egc, dcs);
             /* fall through */
         case LIBXL_CHECKPOINTED_STREAM_NONE:
-            libxl__stream_read_start(egc, &dcs->srs);
+            if (dcs->mirror_disks) {
+                libxl__stream_read_start(egc, &dcs->srs_mirror_disks);
+            } else {
+                libxl__stream_read_start(egc, &dcs->srs);
+            }
         }
         return;
     }
@@ -1146,6 +1222,37 @@ void libxl__srm_callout_callback_restore_results(xen_pfn_t store_mfn,
     shs->need_results =           0;
 }
 
+static void domcreate_post_mirror_disks_stream_done(libxl__egc *egc,
+                                                    libxl__stream_read_state *srs,
+                                                    int ret)
+{
+    libxl__domain_create_state *dcs = srs->dcs;
+    STATE_AO_GC(dcs->ao);
+
+    const uint32_t domid = dcs->guest_domid;
+    const char* uri;
+    const char* state_file = GCSPRINTF(
+                             LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", domid);
+    if (ret)
+        goto error_out;
+
+    ret = libxl__qmp_nbd_server_stop(gc, domid);
+    if (ret){
+        LOGD(ERROR, domid, "Failed to stop NBD server");
+        goto error_out;
+    }
+    uri = GCSPRINTF("exec: /bin/cat %s", state_file);
+    ret = libxl__qmp_migrate_incoming(gc, domid, uri);
+    if (ret)
+        goto error_out;
+    domcreate_devmodel_started(egc, &dcs->sdss.dm, ret);
+    return;
+
+error_out:
+    assert(ret);
+    domcreate_complete(egc, dcs, ret);
+}
+
 static void domcreate_stream_done(libxl__egc *egc,
                                   libxl__stream_read_state *srs,
                                   int ret)
@@ -1212,7 +1319,7 @@ static void domcreate_stream_done(libxl__egc *egc,
     if (ret)
         goto out;
 
-    if (info->type == LIBXL_DOMAIN_TYPE_HVM) {
+    if (info->type == LIBXL_DOMAIN_TYPE_HVM && !dcs->mirror_disks) {
         state->saved_state = GCSPRINTF(
                        LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", domid);
     }
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index ee62bfb..feb9370 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3118,6 +3118,7 @@ struct libxl__stream_read_state {
     int fd;
     bool legacy;
     bool back_channel;
+    int mirror_disks;
     void (*completion_callback)(libxl__egc *egc,
                                 libxl__stream_read_state *srs,
                                 int rc);
@@ -3774,6 +3775,7 @@ struct libxl__domain_create_state {
         /* If we're not doing stubdom, we use only dmss.dm,
          * for the non-stubdom device model. */
     libxl__stream_read_state srs;
+    libxl__stream_read_state srs_mirror_disks;
     /* necessary if the domain creation failed and we have to destroy it */
     libxl__domain_destroy_state dds;
     libxl__multidev multidev;
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 08/12] Migration with Local Disks Mirroring: New stream phase type for libxl streams

[Bruno Alvisio]

The libxl streams are classified by a stream phase parameter (stream_phase):

0. DEFAULT: This is the stream phase when no local disks are being mirrored as
part of the domain save or restore flow. (=0)

1. POST_MIRROR_DISKS: This stream phase happens during the migration flow after
the disks have been completely mirrored. In time order, this is the second libxl
stream of the migration flow. (=1)

2. PRE_MIRROR_DISKS: This stream happens before the disks start to be mirrored
to the destination. In time order, this is the first libxl stream of the
migration flow. (=2)

libxl__xc_mirror_disks_restore_returned is the callback function in the
libxl_read_stream that is registered for the return of libxc_mirror_disks
stream.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxc/include/xenguest.h   |  3 ++-
 tools/libxc/xc_nomigrate.c       |  3 ++-
 tools/libxc/xc_sr_restore.c      |  3 ++-
 tools/libxl/libxl_colo_restore.c |  4 ++--
 tools/libxl/libxl_internal.h     | 12 +++++++++++-
 tools/libxl/libxl_save_callout.c | 13 +++++++++----
 tools/libxl/libxl_save_helper.c  |  5 ++++-
 tools/libxl/libxl_stream_read.c  | 24 ++++++++++++++++++++++--
 tools/libxl/libxl_types.idl      |  6 ++++++
 9 files changed, 60 insertions(+), 13 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index b4b2e19..014dee0 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -199,7 +199,8 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
                       unsigned long *console_mfn, uint32_t console_domid,
                       unsigned int hvm, unsigned int pae,
                       xc_migration_stream_t stream_type,
-                      struct restore_callbacks *callbacks, int send_back_fd);
+                      struct restore_callbacks *callbacks, int send_back_fd,
+                      int stream_phase);
 
 /**
  * This function will create a domain for a paravirtualized Linux
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index 6d6169d..75fe560 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -34,7 +34,8 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
                       unsigned long *console_mfn, uint32_t console_domid,
                       unsigned int hvm, unsigned int pae,
                       xc_migration_stream_t stream_type,
-                      struct restore_callbacks *callbacks, int send_back_fd)
+                      struct restore_callbacks *callbacks, int send_back_fd,
+                      int stream_phase)
 {
     errno = ENOSYS;
     return -1;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index ea7b033..7f74d28 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -829,7 +829,8 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
                       unsigned long *console_gfn, uint32_t console_domid,
                       unsigned int hvm, unsigned int pae,
                       xc_migration_stream_t stream_type,
-                      struct restore_callbacks *callbacks, int send_back_fd)
+                      struct restore_callbacks *callbacks, int send_back_fd,
+                      int stream_phase)
 {
     xen_pfn_t nr_pfns;
     struct xc_sr_context ctx =
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index 0c535bd..a45bf51 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -132,7 +132,7 @@ static void colo_resume_vm(libxl__egc *egc,
         return;
     }
 
-    libxl__xc_domain_restore_done(egc, dcs, 0, 0, 0);
+    libxl__xc_domain_restore_done(egc, dcs, dcs->srs, 0, 0, 0);
 
     return;
 }
@@ -325,7 +325,7 @@ void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
         /* crcs->status is LIBXL_COLO_SETUPED */
         dcs->srs.completion_callback = NULL;
     }
-    libxl__xc_domain_restore_done(egc, dcs, ret, retval, errnoval);
+    libxl__xc_domain_restore_done(egc, dcs, dcs->srs, ret, retval, errnoval);
 
     if (crs->qdisk_setuped) {
         libxl__qmp_stop_replication(gc, crs->domid, false);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index feb9370..76bbe48 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3825,13 +3825,23 @@ _hidden int libxl__restore_emulator_xenstore_data
 _hidden void libxl__xc_domain_restore(libxl__egc *egc,
                                       libxl__domain_create_state *dcs,
                                       libxl__save_helper_state *shs,
-                                      int hvm, int pae);
+                                      int hvm, int pae, int stream_phase);
 /* If rc==0 then retval is the return value from xc_domain_save
  * and errnoval is the errno value it provided.
  * If rc!=0, retval and errnoval are undefined. */
 _hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
+                                           libxl__stream_read_state *stream,
                                            int rc, int retval, int errnoval);
 
+_hidden void libxl__xc_domain_restore_returned(libxl__egc *egc, void *dcs_void,
+                                               int rc, int retval,
+                                               int errnoval);
+
+_hidden void libxl__xc_mirror_disks_restore_returned(libxl__egc *egc,
+                                                     void *dcs_void,
+                                                     int rc, int retval,
+                                                     int errnoval);
+
 _hidden void libxl__save_helper_init(libxl__save_helper_state *shs);
 _hidden void libxl__save_helper_abort(libxl__egc *egc,
                                       libxl__save_helper_state *shs);
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 6452d70..0ff9a01 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -43,7 +43,7 @@ static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs);
 
 void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
                               libxl__save_helper_state *shs,
-                              int hvm, int pae)
+                              int hvm, int pae, int stream_phase)
 {
     STATE_AO_GC(dcs->ao);
 
@@ -63,16 +63,21 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
         state->console_domid,
         hvm, pae,
         cbflags, dcs->restore_params.checkpointed_stream,
+        stream_phase,
     };
 
     shs->ao = ao;
     shs->domid = domid;
     shs->recv_callback = libxl__srm_callout_received_restore;
     if (dcs->restore_params.checkpointed_stream ==
-        LIBXL_CHECKPOINTED_STREAM_COLO)
+        LIBXL_CHECKPOINTED_STREAM_COLO) {
         shs->completion_callback = libxl__colo_restore_teardown;
-    else
-        shs->completion_callback = libxl__xc_domain_restore_done;
+    } else {
+        if ( stream_phase != LIBXL_STREAM_PHASE_PRE_MIRROR_DISKS )
+            shs->completion_callback = libxl__xc_domain_restore_returned;
+        else
+            shs->completion_callback = libxl__xc_mirror_disks_restore_returned;
+    }
     shs->caller_state = dcs;
     shs->need_results = 1;
 
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 38089a0..f1e8dca 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -279,6 +279,7 @@ int main(int argc, char **argv)
         unsigned int pae =                  strtoul(NEXTARG,0,10);
         unsigned cbflags =                  strtoul(NEXTARG,0,10);
         xc_migration_stream_t stream_type = strtoul(NEXTARG,0,10);
+        int stream_phase =                  atoi(NEXTARG);
         assert(!*++argv);
 
         helper_setcallbacks_restore(&helper_restore_callbacks, cbflags);
@@ -293,7 +294,9 @@ int main(int argc, char **argv)
                               store_domid, console_evtchn, &console_mfn,
                               console_domid, hvm, pae,
                               stream_type,
-                              &helper_restore_callbacks, send_back_fd);
+                              &helper_restore_callbacks, send_back_fd,
+                              stream_phase);
+
         helper_stub_restore_results(store_mfn,console_mfn,0);
         complete(r);
 
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index fcb39ee..4c845c5 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -561,6 +561,7 @@ static bool process_record(libxl__egc *egc,
 {
     STATE_AO_GC(stream->ao);
     libxl__domain_create_state *dcs = stream->dcs;
+    int create_mirror_disks = stream->dcs->mirror_disks;
     libxl__sr_record_buf *rec;
     libxl_sr_checkpoint_state *srcs;
     bool further_action_needed = false;
@@ -580,7 +581,9 @@ static bool process_record(libxl__egc *egc,
         break;
 
     case REC_TYPE_LIBXC_CONTEXT:
-        libxl__xc_domain_restore(egc, dcs, &stream->shs, 0, 0);
+        libxl__xc_domain_restore(egc, dcs, &stream->shs, 0, 0,
+                                 stream->mirror_disks +
+                                 create_mirror_disks);
         break;
 
     case REC_TYPE_EMULATOR_XENSTORE_DATA:
@@ -834,10 +837,10 @@ static void stream_done(libxl__egc *egc,
 }
 
 void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
+                                   libxl__stream_read_state *stream,
                                    int rc, int retval, int errnoval)
 {
     libxl__domain_create_state *dcs = dcs_void;
-    libxl__stream_read_state *stream = &dcs->srs;
     STATE_AO_GC(dcs->ao);
 
     /* convenience aliases */
@@ -894,6 +897,23 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
     }
 }
 
+void libxl__xc_domain_restore_returned(libxl__egc *egc, void *dcs_void,
+                                       int rc, int retval, int errnoval)
+{
+    libxl__domain_create_state *dcs = dcs_void;
+    libxl__stream_read_state *stream = &dcs->srs;
+    libxl__xc_domain_restore_done(egc, dcs_void, stream, rc, retval, errnoval);
+}
+
+void libxl__xc_mirror_disks_restore_returned(libxl__egc *egc, void *dcs_void,
+                                             int rc, int retval, int errnoval)
+{
+    libxl__domain_create_state *dcs = dcs_void;
+    libxl__stream_read_state *stream = &dcs->srs_mirror_disks;
+    libxl__xc_domain_restore_done(egc, dcs_void, stream, rc, retval, errnoval);
+
+}
+
 static void conversion_done(libxl__egc *egc,
                             libxl__conversion_helper_state *chs, int rc)
 {
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a239324..3775f25 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -242,6 +242,12 @@ libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
     (2, "COLO"),
     ])
 
+libxl_stream_phase = Enumeration("stream_phase", [
+    (0, "DEFAULT"),
+    (1, "POST_MIRROR_DISKS"),
+    (2, "PRE_MIRROR_DISKS"),
+    ])
+
 libxl_vuart_type = Enumeration("vuart_type", [
     (0, "unknown"),
     (1, "sbsa_uart"),
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 09/12] Migration with Local Disks Mirroring: New stream phase type for libxc streams

[Bruno Alvisio]

Adapted libxc restore stream. Defined libxc stream phase types:

0. XC_STREAM_PHASE_DEFAULT: This is the stream phase when no local disks are
being mirrored as part of the domain save or restore (=0)

1. XC_STREAM_PHASE_POST_MIRROR_DISKS: This stream phase transfers the virtual
RAM from source to destination. It happens after disks mirroring is completed.

2. XC_STREAM_PHASE_PRE_MIRROR_DISKS: This stream transfers pfns and parameters
necessary to start the QEMU process in the destination. It happens before the
disks mirroring.

The PRE_MIRROR_DISKS phase stream type skips the stream_complete ops to restore
the domain. The restore is performed by the POST_MIRROR_DISKS phase stream that
is executed later in the restore flow. If no local disks are mirrored the
restore is executed by the DEFAULT phase stream type.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxc/xc_sr_common.h        |  1 +
 tools/libxc/xc_sr_restore.c       | 51 ++++++++++++++++++++++-----------------
 tools/libxc/xc_sr_stream_format.h |  5 ++++
 3 files changed, 35 insertions(+), 22 deletions(-)

diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index a145a15..8cf393f 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -177,6 +177,7 @@ struct xc_sr_context
     xc_interface *xch;
     uint32_t domid;
     int fd;
+    int stream_phase;
 
     xc_dominfo_t dominfo;
 
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 7f74d28..924386c 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -736,7 +736,10 @@ static int restore(struct xc_sr_context *ctx)
     struct xc_sr_record rec;
     int rc, saved_rc = 0, saved_errno = 0;
 
-    IPRINTF("Restoring domain");
+    if ( ctx->stream_phase != XC_STREAM_PHASE_PRE_MIRROR_DISKS )
+        IPRINTF("Restoring domain");
+    else
+        IPRINTF("Mirroring disks restoring phase");
 
     rc = setup(ctx);
     if ( rc )
@@ -799,11 +802,16 @@ static int restore(struct xc_sr_context *ctx)
      * With Remus, if we reach here, there must be some error on primary,
      * failover from the last checkpoint state.
      */
-    rc = ctx->restore.ops.stream_complete(ctx);
-    if ( rc )
-        goto err;
+    if ( ctx->stream_phase != XC_STREAM_PHASE_PRE_MIRROR_DISKS )
+    {
+        rc = ctx->restore.ops.stream_complete(ctx);
+        if ( rc )
+            goto err;
 
-    IPRINTF("Restore successful");
+        IPRINTF("Restore successful");
+    } else {
+        IPRINTF("Mirroring disks restore phase successful");
+    }
     goto done;
 
  err:
@@ -837,6 +845,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
         {
             .xch = xch,
             .fd = io_fd,
+            .stream_phase = stream_phase
         };
 
     /* GCC 4.4 (of CentOS 6.x vintage) can' t initialise anonymous unions. */
@@ -890,29 +899,27 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
     ctx.restore.p2m_size = nr_pfns;
 
     if ( ctx.dominfo.hvm )
-    {
         ctx.restore.ops = restore_ops_x86_hvm;
-        if ( restore(&ctx) )
-            return -1;
-    }
     else
-    {
         ctx.restore.ops = restore_ops_x86_pv;
-        if ( restore(&ctx) )
-            return -1;
-    }
 
-    IPRINTF("XenStore: mfn %#"PRIpfn", dom %d, evt %u",
-            ctx.restore.xenstore_gfn,
-            ctx.restore.xenstore_domid,
-            ctx.restore.xenstore_evtchn);
+    if ( restore(&ctx) )
+        return -1;
+
+    if ( stream_phase != XC_STREAM_PHASE_PRE_MIRROR_DISKS )
+    {
+        IPRINTF("XenStore: mfn %#"PRIpfn", dom %d, evt %u",
+                ctx.restore.xenstore_gfn,
+                ctx.restore.xenstore_domid,
+                ctx.restore.xenstore_evtchn);
 
-    IPRINTF("Console: mfn %#"PRIpfn", dom %d, evt %u",
-            ctx.restore.console_gfn,
-            ctx.restore.console_domid,
-            ctx.restore.console_evtchn);
+        IPRINTF("Console: mfn %#"PRIpfn", dom %d, evt %u",
+                ctx.restore.console_gfn,
+                ctx.restore.console_domid,
+                ctx.restore.console_evtchn);
 
-    *console_gfn = ctx.restore.console_gfn;
+        *console_gfn = ctx.restore.console_gfn;
+    }
     *store_mfn = ctx.restore.xenstore_gfn;
 
     return 0;
diff --git a/tools/libxc/xc_sr_stream_format.h b/tools/libxc/xc_sr_stream_format.h
index 15ff1c7..c705da4 100644
--- a/tools/libxc/xc_sr_stream_format.h
+++ b/tools/libxc/xc_sr_stream_format.h
@@ -138,6 +138,11 @@ struct xc_sr_rec_hvm_params
     struct xc_sr_rec_hvm_params_entry param[0];
 };
 
+/* LIBXC stream phase types */
+#define XC_STREAM_PHASE_DEFAULT 0
+#define XC_STREAM_PHASE_POST_MIRROR_DISKS 1
+#define XC_STREAM_PHASE_PRE_MIRROR_DISKS 2
+
 #endif
 /*
  * Local variables:
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 10/12] Migration with Local Disks Mirroring: libxl save flow support

[Bruno Alvisio]

The domain save state contains a new 'mirror_disks' field. It determines the
flow of the domain save. If false, the save flow will invoke the DEFAULT phase
libxl/libxc stream type. If true, the save flow invokes the PRE_MIRROR_DISKS
phase stream type first. Upon reception of the signal from the source that the
NBD server has been started it triggers the disks mirroring job. Once it
receives an event from QEMU that the mirroring job is complete, the
POST_MIRROR_DISKS phase stream type is invoked.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxl/libxl_dom_save.c | 53 ++++++++++++++++++++++++++++++++++++++++++--
 tools/libxl/libxl_domain.c   |  2 ++
 tools/libxl/libxl_internal.h |  4 ++++
 3 files changed, 57 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 6487672..78c4e86 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -25,6 +25,9 @@ static void stream_done(libxl__egc *egc,
                         libxl__stream_write_state *sws, int rc);
 static void domain_save_done(libxl__egc *egc,
                              libxl__domain_save_state *dss, int rc);
+static void domain_pre_mirror_disks_done(libxl__egc *egc,
+                                         libxl__stream_write_state *sws,
+                                         int rc);
 
 /*----- complicated callback, called by xc_domain_save -----*/
 
@@ -411,14 +414,60 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
     dss->sws.fd  = dss->fd;
     dss->sws.back_channel = false;
     dss->sws.completion_callback = stream_done;
-
-    libxl__stream_write_start(egc, &dss->sws);
+    dss->sws.mirror_disks = 0;
+
+    if (!dss->mirror_disks) {
+        libxl__stream_write_start(egc, &dss->sws);
+    } else {
+        dss->sws_mirror_disks.ao  = dss->ao;
+        dss->sws_mirror_disks.dss = dss;
+        dss->sws_mirror_disks.fd  = dss->fd;
+        dss->sws_mirror_disks.back_channel = false;
+        dss->sws_mirror_disks.mirror_disks = 1;
+        dss->sws_mirror_disks.completion_callback = domain_pre_mirror_disks_done;
+
+        libxl__stream_write_start(egc, &dss->sws_mirror_disks);
+    }
     return;
 
  out:
     domain_save_done(egc, dss, rc);
 }
 
+static void domain_pre_mirror_disks_done(libxl__egc *egc,
+                                         libxl__stream_write_state *sws,
+                                         int rc)
+{
+    libxl__domain_save_state *dss = sws->dss;
+    STATE_AO_GC(dss->ao);
+    const uint32_t domid = dss->domid;
+    char* target;
+
+    if (rc)
+        goto err;
+
+    rc = libxl_read_fixedmessage(CTX, dss->recv_fd, nbd_server_started_banner,
+                                 sizeof(nbd_server_started_banner)-1,
+                                 "migration stream", 0);
+    if (rc)
+        goto err;
+
+    target = GCSPRINTF("nbd:%s:%s:exportname=%s", dss->hostname,
+                       DRIVE_MIRROR_PORT, DRIVE_MIRROR_DEVICE);
+    rc = libxl__qmp_drive_mirror(gc, dss->domid, DRIVE_MIRROR_DEVICE, target,
+                                 "raw");
+    if (rc) {
+        LOGD(ERROR, domid, "Sending QMP drive mirror command failed");
+        goto err;
+    }
+
+    libxl__stream_write_start(egc, &sws->dss->sws);
+    return;
+
+ err:
+   dss->callback(egc, dss, rc);
+}
+
 static void stream_done(libxl__egc *egc,
                         libxl__stream_write_state *sws, int rc)
 {
diff --git a/tools/libxl/libxl_domain.c b/tools/libxl/libxl_domain.c
index d1209cd..341ed52 100644
--- a/tools/libxl/libxl_domain.c
+++ b/tools/libxl/libxl_domain.c
@@ -511,6 +511,8 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, int fd, int recv_fd,
     dss->type = type;
     dss->live = flags & LIBXL_SUSPEND_LIVE;
     dss->debug = flags & LIBXL_SUSPEND_DEBUG;
+    dss->mirror_disks = (flags & LIBXL_SUSPEND_MIRROR_DISKS) ? 1 : 0;
+    dss->hostname = hostname;
     dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
 
     rc = libxl__fd_flags_modify_save(gc, dss->fd,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 76bbe48..0a3215f 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3197,6 +3197,7 @@ struct libxl__stream_write_state {
     libxl__domain_save_state *dss;
     int fd;
     bool back_channel;
+    int mirror_disks;
     void (*completion_callback)(libxl__egc *egc,
                                 libxl__stream_write_state *sws,
                                 int rc);
@@ -3296,6 +3297,8 @@ struct libxl__domain_save_state {
     libxl_domain_type type;
     int live;
     int debug;
+    int mirror_disks;
+    const char* hostname;
     int checkpointed_stream;
     const libxl_domain_remus_info *remus;
     /* private */
@@ -3311,6 +3314,7 @@ struct libxl__domain_save_state {
     };
     libxl__checkpoint_devices_state cds;
     libxl__stream_write_state sws;
+    libxl__stream_write_state sws_mirror_disks;
     libxl__logdirty_switch logdirty;
 };
 
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 11/12] Migration with Local Disks Mirroring: libxl write stream support for stream phase type

[Bruno Alvisio]

The libxl stream write receives stream phase type (DEFAULT, PRE_MIRROR_DISKS or
POST_MIRROR_DISKS) and registers the appropiate callback to the
libxl_save_helper. If the stream phase == PRE_MIRROR_DISKS the stream skips
writing the context record since it is written at a later time by the
POST_MIRROR_DISKS libxl stream.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxc/include/xenguest.h   |  3 +-
 tools/libxc/xc_nomigrate.c       |  3 +-
 tools/libxc/xc_sr_save.c         |  3 +-
 tools/libxl/libxl_internal.h     | 12 +++++--
 tools/libxl/libxl_save_callout.c | 10 ++++--
 tools/libxl/libxl_save_helper.c  |  3 +-
 tools/libxl/libxl_stream_write.c | 67 +++++++++++++++++++++++++++++-----------
 7 files changed, 73 insertions(+), 28 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 014dee0..743294b 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -134,7 +134,8 @@ typedef enum {
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
                    uint32_t flags /* XCFLAGS_xxx */,
                    struct save_callbacks* callbacks, int hvm,
-                   xc_migration_stream_t stream_type, int recv_fd);
+                   xc_migration_stream_t stream_type, int recv_fd,
+                   int stream_phase);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index 75fe560..ab2d744 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -22,7 +22,8 @@
 
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t flags,
                    struct save_callbacks* callbacks, int hvm,
-                   xc_migration_stream_t stream_type, int recv_fd)
+                   xc_migration_stream_t stream_type, int recv_fd,
+                   int stream_phase)
 {
     errno = ENOSYS;
     return -1;
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index 5a40e58..b7498e3 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -967,7 +967,8 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
 
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
                    uint32_t flags, struct save_callbacks* callbacks,
-                   int hvm, xc_migration_stream_t stream_type, int recv_fd)
+                   int hvm, xc_migration_stream_t stream_type, int recv_fd,
+                   int stream_phase)
 {
     struct xc_sr_context ctx =
         {
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 0a3215f..320c97d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3799,12 +3799,18 @@ _hidden void libxl__domain_save(libxl__egc *egc,
 /* calls libxl__xc_domain_suspend_done when done */
 _hidden void libxl__xc_domain_save(libxl__egc *egc,
                                    libxl__domain_save_state *dss,
-                                   libxl__save_helper_state *shs);
+                                   libxl__save_helper_state *shs,
+                                   int stream_phase);
 /* If rc==0 then retval is the return value from xc_domain_save
  * and errnoval is the errno value it provided.
  * If rc!=0, retval and errnoval are undefined. */
-_hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
-                                        int rc, int retval, int errnoval);
+_hidden void libxl__xc_domain_save_returned(libxl__egc*, void *dss_void,
+                                            int rc, int retval, int errnoval);
+
+_hidden void libxl__xc_mirror_disks_save_returned(libxl__egc *egc,
+                                                  void *dss_void,
+                                                  int rc, int retval,
+                                                  int errnoval);
 
 /* Used by asynchronous callbacks: ie ones which xc regards as
  * returning a value, but which we want to handle asynchronously.
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 0ff9a01..090d50b 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -86,7 +86,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
 }
 
 void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_save_state *dss,
-                           libxl__save_helper_state *shs)
+                           libxl__save_helper_state *shs, int stream_phase)
 {
     STATE_AO_GC(dss->ao);
 
@@ -95,13 +95,17 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_save_state *dss,
 
     const unsigned long argnums[] = {
         dss->domid, dss->xcflags, dss->hvm, cbflags,
-        dss->checkpointed_stream,
+        dss->checkpointed_stream, stream_phase,
     };
 
     shs->ao = ao;
     shs->domid = dss->domid;
     shs->recv_callback = libxl__srm_callout_received_save;
-    shs->completion_callback = libxl__xc_domain_save_done;
+    if ( stream_phase != LIBXL_STREAM_PHASE_PRE_MIRROR_DISKS )
+        shs->completion_callback = libxl__xc_domain_save_returned;
+    else
+        shs->completion_callback = libxl__xc_mirror_disks_save_returned;
+
     shs->caller_state = dss;
     shs->need_results = 0;
 
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index f1e8dca..53b4d78 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -255,6 +255,7 @@ int main(int argc, char **argv)
         int hvm =                           atoi(NEXTARG);
         unsigned cbflags =                  strtoul(NEXTARG,0,10);
         xc_migration_stream_t stream_type = strtoul(NEXTARG,0,10);
+        int stream_phase =                  atoi(NEXTARG);
         assert(!*++argv);
 
         helper_setcallbacks_save(&helper_save_callbacks, cbflags);
@@ -263,7 +264,7 @@ int main(int argc, char **argv)
         setup_signals(save_signal_handler);
 
         r = xc_domain_save(xch, io_fd, dom, flags, &helper_save_callbacks,
-                           hvm, stream_type, recv_fd);
+                           hvm, stream_type, recv_fd, stream_phase);
         complete(r);
 
     } else if (!strcmp(mode,"--restore-domain")) {
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index 634f324..0ff096a 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -330,31 +330,43 @@ static void stream_header_done(libxl__egc *egc,
 static void libxc_header_done(libxl__egc *egc,
                               libxl__stream_write_state *stream)
 {
-    libxl__xc_domain_save(egc, stream->dss, &stream->shs);
+    int save_mirror_disks = stream->dss->mirror_disks;
+    libxl__xc_domain_save(egc, stream->dss, &stream->shs,
+                          save_mirror_disks + stream->mirror_disks);
 }
 
-void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
-                                int rc, int retval, int errnoval)
+static void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
+                                       libxl__stream_write_state *stream,
+                                       int rc, int retval, int errnoval)
 {
     libxl__domain_save_state *dss = dss_void;
-    libxl__stream_write_state *stream = &dss->sws;
     STATE_AO_GC(dss->ao);
 
     if (rc)
         goto err;
 
     if (retval) {
-        LOGEVD(ERROR, errnoval, dss->domid, "saving domain: %s",
-              dss->dsps.guest_responded ?
-              "domain responded to suspend request" :
-              "domain did not respond to suspend request");
-        if (!dss->dsps.guest_responded)
-            rc = ERROR_GUEST_TIMEDOUT;
-        else if (dss->rc)
-            rc = dss->rc;
-        else
-            rc = ERROR_FAIL;
-        goto err;
+        if (!stream->mirror_disks) {
+            LOGEVD(ERROR, errnoval, dss->domid, "saving domain: %s",
+                   dss->dsps.guest_responded ?
+                   "domain responded to suspend request" :
+                   "domain did not respond to suspend request");
+            if (!dss->dsps.guest_responded)
+                rc = ERROR_GUEST_TIMEDOUT;
+            else if (dss->rc)
+                rc = dss->rc;
+            else
+                rc = ERROR_FAIL;
+            goto err;
+        } else {
+            LOGEVD(ERROR, errnoval, dss->domid, "saving domain: Error occurred "
+                   "in libxc pre mirror phase stream");
+            if (dss->rc)
+                rc = dss->rc;
+            else
+                rc = ERROR_FAIL;
+            goto err;
+        }
     }
 
  err:
@@ -380,6 +392,22 @@ void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
     }
 }
 
+void libxl__xc_mirror_disks_save_returned(libxl__egc *egc, void *dss_void,
+                                          int rc, int retval, int errnoval)
+{
+    libxl__domain_save_state *dss = dss_void;
+    libxl__stream_write_state *stream = &dss->sws_mirror_disks;
+    libxl__xc_domain_save_done(egc, dss_void, stream, rc, retval, errnoval);
+}
+
+void libxl__xc_domain_save_returned(libxl__egc *egc, void *dss_void,
+                                    int rc, int retval, int errnoval)
+{
+    libxl__domain_save_state *dss = dss_void;
+    libxl__stream_write_state *stream = &dss->sws;
+    libxl__xc_domain_save_done(egc, dss_void, stream, rc, retval, errnoval);
+}
+
 static void write_emulator_xenstore_record(libxl__egc *egc,
                                            libxl__stream_write_state *stream)
 {
@@ -424,9 +452,12 @@ static void emulator_xenstore_record_done(libxl__egc *egc,
 {
     libxl__domain_save_state *dss = stream->dss;
 
-    if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
-        write_emulator_context_record(egc, stream);
-    else {
+    if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
+        if (!stream->mirror_disks)
+            write_emulator_context_record(egc, stream);
+        else
+            write_end_record(egc, stream);
+    } else {
         if (stream->in_checkpoint)
             write_checkpoint_end_record(egc, stream);
         else
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH RFC v3 RESEND 12/12] Migration with Local Disks Mirroring: Introduce pre_mirror_disks_stream_phase op to xc_sr_save_ops

[Bruno Alvisio]

A new op pre_mirror_stream_phase is introduced as part of the xc_sr_save_ops.
This op sends all pfns and params that need to be transferred before the disks
mirroring jobs can be started. Note that no new libxc record type is created.

The save flow is modified such that: if the stream_phase ==
XC_PRE_MIRROR_DISKS_STREAM_PHASE only the pre_mirror_disks op is executed as
part of the save(). In all other libxc phase types, the original flow is
executed.

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>
---
 tools/libxc/xc_sr_common.h       |  11 ++++
 tools/libxc/xc_sr_save.c         |  24 +++++++--
 tools/libxc/xc_sr_save_x86_hvm.c | 109 +++++++++++++++++++++++++++------------
 3 files changed, 105 insertions(+), 39 deletions(-)

diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 8cf393f..44f4103 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -96,6 +96,13 @@ struct xc_sr_save_ops
      * after a successful save, or upon encountering an error.
      */
     int (*cleanup)(struct xc_sr_context *ctx);
+
+    /**
+     * Send the necessary records/params to allow the start of the local
+     * disks mirroring job in the destination node. It will be called exactly
+     * once only if the stream phase type == XC_STREAM_PHASE_PRE_MIRROR_DISKS
+     */
+    int (*pre_mirror_disks_stream_phase)(struct xc_sr_context *ctx);
 };
 
 
@@ -398,6 +405,10 @@ int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec);
 int populate_pfns(struct xc_sr_context *ctx, unsigned count,
                   const xen_pfn_t *original_pfns, const uint32_t *types);
 
+int add_to_batch(struct xc_sr_context *ctx, xen_pfn_t pfn);
+
+int flush_batch(struct xc_sr_context *ctx);
+
 #endif
 /*
  * Local variables:
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index b7498e3..557dafe 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -279,7 +279,7 @@ static int write_batch(struct xc_sr_context *ctx)
 /*
  * Flush a batch of pfns into the stream.
  */
-static int flush_batch(struct xc_sr_context *ctx)
+int flush_batch(struct xc_sr_context *ctx)
 {
     int rc = 0;
 
@@ -301,7 +301,7 @@ static int flush_batch(struct xc_sr_context *ctx)
 /*
  * Add a single pfn to the batch, flushing the batch if full.
  */
-static int add_to_batch(struct xc_sr_context *ctx, xen_pfn_t pfn)
+int add_to_batch(struct xc_sr_context *ctx, xen_pfn_t pfn)
 {
     int rc = 0;
 
@@ -842,8 +842,12 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
     xc_interface *xch = ctx->xch;
     int rc, saved_rc = 0, saved_errno = 0;
 
-    IPRINTF("Saving domain %d, type %s",
-            ctx->domid, dhdr_type_to_str(guest_type));
+    if ( ctx->stream_phase == XC_STREAM_PHASE_PRE_MIRROR_DISKS )
+        IPRINTF("Pre-mirroring disks save phase for domain %d, type %s",
+                ctx->domid, dhdr_type_to_str(guest_type));
+    else
+        IPRINTF("Saving domain %d, type %s",
+                ctx->domid, dhdr_type_to_str(guest_type));
 
     rc = setup(ctx);
     if ( rc )
@@ -855,6 +859,13 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
     if ( rc )
         goto err;
 
+    if ( ctx->stream_phase == XC_STREAM_PHASE_PRE_MIRROR_DISKS ) {
+        rc = ctx->save.ops.pre_mirror_disks_stream_phase(ctx);
+        if ( rc )
+            goto err;
+        goto end;
+    }
+
     rc = ctx->save.ops.start_of_stream(ctx);
     if ( rc )
         goto err;
@@ -939,6 +950,7 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
         }
     } while ( ctx->save.checkpointed != XC_MIG_STREAM_NONE );
 
+ end:
     xc_report_progress_single(xch, "End of stream");
 
     rc = write_end_record(ctx);
@@ -974,6 +986,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
         {
             .xch = xch,
             .fd = io_fd,
+            .stream_phase = stream_phase
         };
 
     /* GCC 4.4 (of CentOS 6.x vintage) can' t initialise anonymous unions. */
@@ -989,7 +1002,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
            stream_type == XC_MIG_STREAM_COLO);
 
     /* Sanity checks for callbacks. */
-    if ( hvm )
+    /* The pre mirror disks phase stream doesn't enable/disable qemu log */
+    if ( hvm && ctx.stream_phase != XC_STREAM_PHASE_PRE_MIRROR_DISKS )
         assert(callbacks->switch_qemu_logdirty);
     if ( ctx.save.checkpointed )
         assert(callbacks->checkpoint && callbacks->postcopy);
diff --git a/tools/libxc/xc_sr_save_x86_hvm.c b/tools/libxc/xc_sr_save_x86_hvm.c
index 97a8c49..423edd7 100644
--- a/tools/libxc/xc_sr_save_x86_hvm.c
+++ b/tools/libxc/xc_sr_save_x86_hvm.c
@@ -4,6 +4,32 @@
 
 #include <xen/hvm/params.h>
 
+static const unsigned int params[] = {
+    HVM_PARAM_STORE_PFN,
+    HVM_PARAM_IOREQ_PFN,
+    HVM_PARAM_BUFIOREQ_PFN,
+    HVM_PARAM_PAGING_RING_PFN,
+    HVM_PARAM_MONITOR_RING_PFN,
+    HVM_PARAM_SHARING_RING_PFN,
+    HVM_PARAM_VM86_TSS_SIZED,
+    HVM_PARAM_CONSOLE_PFN,
+    HVM_PARAM_ACPI_IOPORTS_LOCATION,
+    HVM_PARAM_VIRIDIAN,
+    HVM_PARAM_IDENT_PT,
+    HVM_PARAM_PAE_ENABLED,
+    HVM_PARAM_VM_GENERATION_ID_ADDR,
+    HVM_PARAM_IOREQ_SERVER_PFN,
+    HVM_PARAM_NR_IOREQ_SERVER_PAGES,
+    HVM_PARAM_X87_FIP_WIDTH,
+    HVM_PARAM_MCA_CAP,
+};
+
+static const unsigned int params_mirroring[] = {
+    HVM_PARAM_STORE_PFN,
+    HVM_PARAM_IOREQ_PFN,
+    HVM_PARAM_BUFIOREQ_PFN,
+};
+
 /*
  * Query for the HVM context and write an HVM_CONTEXT record into the stream.
  */
@@ -58,30 +84,11 @@ static int write_hvm_context(struct xc_sr_context *ctx)
  * Query for a range of HVM parameters and write an HVM_PARAMS record into the
  * stream.
  */
-static int write_hvm_params(struct xc_sr_context *ctx)
+static int write_hvm_params(struct xc_sr_context *ctx,
+                            const unsigned int *params, unsigned int nr_params)
 {
-    static const unsigned int params[] = {
-        HVM_PARAM_STORE_PFN,
-        HVM_PARAM_IOREQ_PFN,
-        HVM_PARAM_BUFIOREQ_PFN,
-        HVM_PARAM_PAGING_RING_PFN,
-        HVM_PARAM_MONITOR_RING_PFN,
-        HVM_PARAM_SHARING_RING_PFN,
-        HVM_PARAM_VM86_TSS_SIZED,
-        HVM_PARAM_CONSOLE_PFN,
-        HVM_PARAM_ACPI_IOPORTS_LOCATION,
-        HVM_PARAM_VIRIDIAN,
-        HVM_PARAM_IDENT_PT,
-        HVM_PARAM_PAE_ENABLED,
-        HVM_PARAM_VM_GENERATION_ID_ADDR,
-        HVM_PARAM_IOREQ_SERVER_PFN,
-        HVM_PARAM_NR_IOREQ_SERVER_PAGES,
-        HVM_PARAM_X87_FIP_WIDTH,
-        HVM_PARAM_MCA_CAP,
-    };
-
     xc_interface *xch = ctx->xch;
-    struct xc_sr_rec_hvm_params_entry entries[ARRAY_SIZE(params)];
+    struct xc_sr_rec_hvm_params_entry entries[nr_params];
     struct xc_sr_rec_hvm_params hdr = {
         .count = 0,
     };
@@ -93,7 +100,7 @@ static int write_hvm_params(struct xc_sr_context *ctx)
     unsigned int i;
     int rc;
 
-    for ( i = 0; i < ARRAY_SIZE(params); i++ )
+    for ( i = 0; i < nr_params; i++ )
     {
         uint32_t index = params[i];
         uint64_t value;
@@ -160,7 +167,8 @@ static int x86_hvm_setup(struct xc_sr_context *ctx)
 
     ctx->save.p2m_size = nr_pfns;
 
-    if ( ctx->save.callbacks->switch_qemu_logdirty(
+    if ( ctx->stream_phase != XC_STREAM_PHASE_PRE_MIRROR_DISKS &&
+         ctx->save.callbacks->switch_qemu_logdirty(
              ctx->domid, 1, ctx->save.callbacks->data) )
     {
         PERROR("Couldn't enable qemu log-dirty mode");
@@ -205,19 +213,51 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx)
         return rc;
 
     /* Write HVM_PARAMS record contains applicable HVM params. */
-    rc = write_hvm_params(ctx);
+    rc = write_hvm_params(ctx, params, ARRAY_SIZE(params));
     if ( rc )
         return rc;
 
     return 0;
 }
 
+static int x86_hvm_send_pre_mirror_disks_pages(struct xc_sr_context *ctx)
+{
+    xc_interface *xch = ctx->xch;
+    uint64_t value;
+    unsigned int i;
+    int rc;
+
+    xc_set_progress_prefix(xch, "Pre-mirroring local disks phase");
+
+    for (i = 0; i < ARRAY_SIZE(params_mirroring); i++)
+    {
+        rc = xc_hvm_param_get(xch, ctx->domid, params_mirroring[i], &value);
+        if ( rc )
+            goto out;
+        rc = add_to_batch(ctx, value);
+        if ( rc )
+            goto out;
+    }
+
+    rc = flush_batch(ctx);
+    if ( rc )
+        goto out;
+
+    rc = write_hvm_params(ctx, params_mirroring, ARRAY_SIZE(params_mirroring));
+    if ( rc )
+        goto out;
+
+ out:
+    return rc;
+}
+
 static int x86_hvm_cleanup(struct xc_sr_context *ctx)
 {
     xc_interface *xch = ctx->xch;
 
     /* If qemu successfully enabled logdirty mode, attempt to disable. */
-    if ( ctx->x86_hvm.save.qemu_enabled_logdirty &&
+    if ( ctx->stream_phase != XC_STREAM_PHASE_PRE_MIRROR_DISKS &&
+         ctx->x86_hvm.save.qemu_enabled_logdirty &&
          ctx->save.callbacks->switch_qemu_logdirty(
              ctx->domid, 0, ctx->save.callbacks->data) )
     {
@@ -230,14 +270,15 @@ static int x86_hvm_cleanup(struct xc_sr_context *ctx)
 
 struct xc_sr_save_ops save_ops_x86_hvm =
 {
-    .pfn_to_gfn          = x86_hvm_pfn_to_gfn,
-    .normalise_page      = x86_hvm_normalise_page,
-    .setup               = x86_hvm_setup,
-    .start_of_stream     = x86_hvm_start_of_stream,
-    .start_of_checkpoint = x86_hvm_start_of_checkpoint,
-    .end_of_checkpoint   = x86_hvm_end_of_checkpoint,
-    .check_vm_state      = x86_hvm_check_vm_state,
-    .cleanup             = x86_hvm_cleanup,
+    .pfn_to_gfn                    = x86_hvm_pfn_to_gfn,
+    .normalise_page                = x86_hvm_normalise_page,
+    .setup                         = x86_hvm_setup,
+    .start_of_stream               = x86_hvm_start_of_stream,
+    .start_of_checkpoint           = x86_hvm_start_of_checkpoint,
+    .end_of_checkpoint             = x86_hvm_end_of_checkpoint,
+    .check_vm_state                = x86_hvm_check_vm_state,
+    .cleanup                       = x86_hvm_cleanup,
+    .pre_mirror_disks_stream_phase = x86_hvm_send_pre_mirror_disks_pages,
 };
 
 /*
-- 
2.3.2 (Apple Git-55)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel