[Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li]

This RFC patch series tries to introduce a mechanism using side
channel pipe for RAM via SCM_RIGHTS with unix domain socket
protocol migration.

This side channel will be used for the page flipping by vmsplice,
which will be the internal mechanism for localhost migration that
we are trying to add. The previous patch series for localhost migration
as link,

http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html

After this series, will adjust the process of current migration for
the localhost migration and involve the vmsplice based on the previous
patch set as link above.

Please let me know if it is the proper way for it or there is anything
need to be improved. Your suggestions and comments are very welcome, and
thanks for Paolo for his review and useful suggestions.


Lei Li (8):
  migration-local: add pipe protocol for QEMUFileOps
  migration-local: add qemu_fopen_pipe()
  migration-local: add send_pipefd()
  migration-local: add recv_pipefd()
  QAPI: introduce magration capability unix_page_flipping
  migration: add migrate_unix_page_flipping()
  migration-unix: side channel support on unix outgoing
  migration-unix: side channel support on unix incoming

 Makefile.target               |    1 +
 include/migration/migration.h |    3 +
 include/migration/qemu-file.h |    4 +
 migration-local.c             |  247 +++++++++++++++++++++++++++++++++++++++++
 migration-unix.c              |   48 +++++++-
 migration.c                   |    9 ++
 qapi-schema.json              |    8 +-
 7 files changed, 315 insertions(+), 5 deletions(-)
 create mode 100644 migration-local.c

[Qemu-devel] [PATCH 1/8] migration-local: add pipe protocol for QEMUFileOps

[Lei Li]

This patch adds QEMUFileOps with pipe protocol, will be used to
transmit RAM for localhost migration. The source process would put
buffer, correspondingly the destination process would get buffer.
The read side of the pipe file descriptor would be passed to destination
process via SCM_RIGHTS.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 Makefile.target   |    1 +
 migration-local.c |  114 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 115 insertions(+), 0 deletions(-)
 create mode 100644 migration-local.c

diff --git a/Makefile.target b/Makefile.target
index 9a49852..ae66fef 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -121,6 +121,7 @@ obj-$(CONFIG_KVM) += kvm-all.o
 obj-y += memory.o savevm.o cputlb.o
 obj-y += memory_mapping.o
 obj-y += dump.o
+obj-y += migration-local.o
 LIBS+=$(libs_softmmu)
 
 # xen support
diff --git a/migration-local.c b/migration-local.c
new file mode 100644
index 0000000..1fb62b6
--- /dev/null
+++ b/migration-local.c
@@ -0,0 +1,114 @@
+/*
+ * QEMU localhost live migration
+ *
+ * Copyright IBM, Corp. 2013
+ *
+ * Authors:
+ *   Lei Li   <lilei@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#include "config-host.h"
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "exec/cpu-common.h"
+#include "config.h"
+#include "exec/cpu-all.h"
+#include "monitor/monitor.h"
+#include "migration/qemu-file.h"
+#include "qemu/iov.h"
+#include "sysemu/arch_init.h"
+#include "sysemu/sysemu.h"
+#include "block/block.h"
+#include "qemu/sockets.h"
+#include "migration/block.h"
+#include "qemu/thread.h"
+#include "qmp-commands.h"
+#include "trace.h"
+#include "qemu/osdep.h"
+
+//#define DEBUG_MIGRATION_LOCAL
+
+#ifdef DEBUG_MIGRATION_LOCAL
+#define DPRINTF(fmt, ...) \
+    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+    do { } while (0)
+#endif
+
+typedef struct QEMUFilePipe {
+    QEMUFile *file;
+    int pipefd;
+} QEMUFilePipe;
+
+
+static int qemu_local_get_fd(void *opaque)
+{
+    QEMUFilePipe *s = opaque;
+
+    return s->pipefd;
+}
+
+static int qemu_local_get_buffer(void *opaque, uint8_t *buf,
+                                 int64_t pos, int size)
+{
+    QEMUFilePipe *s = opaque;
+    ssize_t len;
+
+    for (;;) {
+        errno = 0;
+
+        len = read(s->pipefd, buf, size);
+        if (len != -1) {
+            break;
+        }
+
+        if (errno == EAGAIN) {
+            yield_until_fd_readable(s->pipefd);
+        } else if (errno != EINTR) {
+            break;
+        }
+    }
+
+    return len;
+}
+
+static int qemu_local_put_buffer(void *opaque, const uint8_t *buf,
+                                 int64_t pos, int size)
+{
+    QEMUFilePipe *s = opaque;
+    ssize_t len;
+
+    len = write(s->pipefd, buf, size);
+    if (len < size) {
+        len = -1;
+    }
+
+    return len;
+}
+
+static int qemu_local_close(void *opaque)
+{
+    QEMUFilePipe *s = opaque;
+
+    close(s->pipefd);
+    g_free(s);
+
+    return 0;
+}
+
+static const QEMUFileOps pipe_read_ops = {
+    .get_fd     = qemu_local_get_fd,
+    .get_buffer = qemu_local_get_buffer,
+    .close      = qemu_local_close
+};
+
+static const QEMUFileOps pipe_write_ops = {
+    .put_buffer = qemu_local_put_buffer,
+    //.save_page  = qemu_local_save_ram,
+    .close      = qemu_local_close
+};
-- 
1.7.7.6

[Qemu-devel] [PATCH 2/8] migration-loca: add qemu_fopen_pipe()

[Lei Li]

Add qemu_fopen_pipe() to open QEMUFileOps pipe_write_ops and
pipe_read_ops introduced for pipe protocol.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/qemu-file.h |    2 ++
 migration-local.c             |   20 ++++++++++++++++++++
 2 files changed, 22 insertions(+), 0 deletions(-)

diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 0f757fb..39ad0bd 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -99,6 +99,8 @@ QEMUFile *qemu_fopen(const char *filename, const char *mode);
 QEMUFile *qemu_fdopen(int fd, const char *mode);
 QEMUFile *qemu_fopen_socket(int fd, const char *mode);
 QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
+QEMUFile *qemu_fopen_pipe(int sockfd, const char *mode);
+
 int qemu_get_fd(QEMUFile *f);
 int qemu_fclose(QEMUFile *f);
 int64_t qemu_ftell(QEMUFile *f);
diff --git a/migration-local.c b/migration-local.c
index 1fb62b6..d90b2ff 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -112,3 +112,23 @@ static const QEMUFileOps pipe_write_ops = {
     //.save_page  = qemu_local_save_ram,
     .close      = qemu_local_close
 };
+
+QEMUFile *qemu_fopen_pipe(int pipefd, const char *mode)
+{
+    QEMUFilePipe *s;
+
+    if (qemu_file_mode_is_not_valid(mode)) {
+        return NULL;
+    }
+
+    s = g_malloc0(sizeof(QEMUFilePipe));
+    s->pipefd = pipefd;
+
+    if (mode[0] == 'w') {
+        s->file = qemu_fopen_ops(s, &pipe_write_ops);
+    } else {
+        s->file = qemu_fopen_ops(s, &pipe_read_ops);
+    }
+
+    return s->file;
+}
-- 
1.7.7.6

[Qemu-devel] [PATCH 3/8] migration-local: add send_pipefd()

[Lei Li]

This patch adds send_pipefd() to pass the pipe file descriptor
to destination process.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/qemu-file.h |    1 +
 migration-local.c             |   57 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+), 0 deletions(-)

diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 39ad0bd..5fad65f 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -101,6 +101,7 @@ QEMUFile *qemu_fopen_socket(int fd, const char *mode);
 QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
 QEMUFile *qemu_fopen_pipe(int sockfd, const char *mode);
 
+int send_pipefd(int sockfd, int pipefd);
 int qemu_get_fd(QEMUFile *f);
 int qemu_fclose(QEMUFile *f);
 int64_t qemu_ftell(QEMUFile *f);
diff --git a/migration-local.c b/migration-local.c
index d90b2ff..3875b27 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -132,3 +132,60 @@ QEMUFile *qemu_fopen_pipe(int pipefd, const char *mode)
 
     return s->file;
 }
+
+union MsgControl {
+    struct cmsghdr cmsg;
+    char control[CMSG_SPACE(sizeof(int))];
+};
+
+/*
+ * Pass a pipe file descriptor to another process.
+ *
+ * Return negative value If pipefd < 0. Return 0 on
+ * success.
+ *
+ */
+
+int send_pipefd(int sockfd, int pipefd)
+{
+    struct cmsghdr *cmsg;
+    struct iovec iov[1];
+    struct msghdr msg;
+    union MsgControl msg_control;
+
+    char msbuf[1] = { 0x01 };
+    int ret;
+
+    iov[0].iov_base = msbuf;
+    iov[0].iov_len = sizeof(msbuf);
+
+    msg.msg_iov = iov;
+    msg.msg_iovlen = 1;
+
+    if (pipefd < 0) {
+        msg.msg_control = NULL;
+        msg.msg_controllen = 0;
+        /* Negative status means error */
+        msbuf[0] = pipefd;
+    } else {
+        msg.msg_control = &msg_control;
+        msg.msg_controllen = sizeof(msg_control);
+
+        cmsg = &msg_control.cmsg;
+        cmsg->cmsg_level = SOL_SOCKET;
+        cmsg->cmsg_type = SCM_RIGHTS;
+        cmsg->cmsg_len = CMSG_LEN(sizeof(pipefd));
+
+        /* The pipefd to be passed */
+        memcpy(CMSG_DATA(cmsg), &pipefd, sizeof(pipefd));
+    }
+
+    do {
+        ret = sendmsg(sockfd, &msg, 0);
+    } while (ret < 0 && errno == EINTR);
+
+    if (ret < 0) {
+        return ret;
+    }
+    return 0;
+}
-- 
1.7.7.6

[Qemu-devel] [PATCH 4/8] migration-local: add recv_pipefd()

[Lei Li]

This patch adds recv_pipefd() to receive the pipe file descriptor
that passed by source process.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/qemu-file.h |    1 +
 migration-local.c             |   56 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+), 0 deletions(-)

diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 5fad65f..e4df258 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -102,6 +102,7 @@ QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
 QEMUFile *qemu_fopen_pipe(int sockfd, const char *mode);
 
 int send_pipefd(int sockfd, int pipefd);
+int recv_pipefd(int sockfd);
 int qemu_get_fd(QEMUFile *f);
 int qemu_fclose(QEMUFile *f);
 int64_t qemu_ftell(QEMUFile *f);
diff --git a/migration-local.c b/migration-local.c
index 3875b27..40b8f34 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -189,3 +189,59 @@ int send_pipefd(int sockfd, int pipefd)
     }
     return 0;
 }
+
+/*
+ * Receive a pipe file descriptor from a source process
+ * via unix socket.
+ *
+ * Return negative value if there has been an recvmsg error or
+ * no fd to be reveived. Return 0 if the connection closed by
+ * source. Return file descriptor on success.
+ *
+ */
+int recv_pipefd(int sockfd)
+{
+    int pipefd;
+    struct cmsghdr *cmsg;
+    struct msghdr msg;
+    struct iovec iov[1];
+    char req[1];
+    int ret;
+    union MsgControl msg_control;
+
+    iov[0].iov_base = req;
+    iov[0].iov_len = sizeof(req);
+
+    memset(&msg, 0, sizeof(msg));
+    msg.msg_iov = iov;
+    msg.msg_iovlen = 1;
+    msg.msg_control = &msg_control;
+    msg.msg_controllen = sizeof(msg_control);
+
+    do {
+        ret = recvmsg(sockfd, &msg, 0);
+    } while (ret < 0 && errno == EINTR);
+    if (ret <= 0) {
+        return ret;
+    }
+
+    /* req 0x01 means there is a file descriptor to receive */
+    if (req[0] != 0x01) {
+        return -1;
+    }
+
+    cmsg = CMSG_FIRSTHDR(&msg);
+    while (cmsg) {
+        if (cmsg->cmsg_level == SOL_SOCKET &&
+            cmsg->cmsg_type == SCM_RIGHTS &&
+            cmsg->cmsg_len == CMSG_LEN(sizeof(int))) {
+
+            /* The file descriptor to be received */
+            memcpy(&pipefd, CMSG_DATA(cmsg), sizeof(pipefd));
+            return pipefd;
+        }
+        cmsg = CMSG_NXTHDR(&msg, cmsg);
+    }
+
+    return -1;
+}
-- 
1.7.7.6

[Qemu-devel] [PATCH 5/8] QAPI: introduce magration capability unix_page_flipping

[Lei Li]

Introduce unix_page_flipping to MigrationCapability for localhost
migration.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 qapi-schema.json |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index 145eca8..f63dab4 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -629,10 +629,16 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @unix-page-flipping: If enabled, QEMU will support localhost migration. This
+#          feature allows live upgrade of a running QEMU instance by doing localhost
+#          migration with page flipping. It requires the source and destination
+#          are both on localhost. Disabled by default. (since 1.7)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'x-rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'x-rdma-pin-all', 'auto-converge', 'zero-blocks',
+           'unix-page-flipping'] }
 
 ##
 # @MigrationCapabilityStatus
-- 
1.7.7.6

[Qemu-devel] [PATCH 6/8] migration: add migrate_unix_page_flipping()

[Lei Li]

Add migrate_unix_page_flipping() to check if MIGRATION_CAPABILITY_UNIX_PAGE_FLIPPING
is enabled.
 
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/migration.h |    3 +++
 migration.c                   |    9 +++++++++
 2 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 140e6b4..7e5d01a 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -131,10 +131,13 @@ void migrate_add_blocker(Error *reason);
 void migrate_del_blocker(Error *reason);
 
 bool migrate_rdma_pin_all(void);
+
 bool migrate_zero_blocks(void);
 
 bool migrate_auto_converge(void);
 
+bool migrate_unix_page_flipping(void);
+
 int xbzrle_encode_buffer(uint8_t *old_buf, uint8_t *new_buf, int slen,
                          uint8_t *dst, int dlen);
 int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen);
diff --git a/migration.c b/migration.c
index 200d404..825a455 100644
--- a/migration.c
+++ b/migration.c
@@ -540,6 +540,15 @@ int64_t migrate_xbzrle_cache_size(void)
     return s->xbzrle_cache_size;
 }
 
+bool migrate_unix_page_flipping(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_UNIX_PAGE_FLIPPING];
+}
+
 /* migration thread support */
 
 static void *migration_thread(void *opaque)
-- 
1.7.7.6

[Qemu-devel] [PATCH 7/8] migration-unix: side channel support on unix outgoing

[Lei Li]

This patch adds side channel support on the outgoing of unix
migration. It will create a pipe and pass the read pipe fd to
destination process by send_pipefd(). If the pipe fd was passed
successfully, the qemu_fopen_pipe will be called with write mode
to send RAM to the write pipe fd.
 
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-unix.c |   34 +++++++++++++++++++++++++++++++---
 1 files changed, 31 insertions(+), 3 deletions(-)

diff --git a/migration-unix.c b/migration-unix.c
index 651fc5b..0bfc1c7 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -33,16 +33,44 @@
 static void unix_wait_for_connect(int fd, void *opaque)
 {
     MigrationState *s = opaque;
+    int pipefd[2];
 
     if (fd < 0) {
         DPRINTF("migrate connect error\n");
-        s->file = NULL;
-        migrate_fd_error(s);
+        goto fail;
     } else {
         DPRINTF("migrate connect success\n");
-        s->file = qemu_fopen_socket(fd, "wb");
+
+        if (s->enabled_capabilities[MIGRATION_CAPABILITY_UNIX_PAGE_FLIPPING]) {
+            if (pipe(pipefd) < 0) {
+                DPRINTF("qemu_fopen_pipe: %s", strerror(errno));
+                goto fail;
+            }
+
+            /* Send pipefd[0] to destination QEMU process */
+            if (send_pipefd(fd, pipefd[0]) < 0) {
+                DPRINTF("failed to pass pipe\n");
+                goto fail;
+            }
+
+            s->file = qemu_fopen_pipe(pipefd[1], "w");
+        } else {
+            s->file = qemu_fopen_socket(fd, "wb");
+        }
+
+        if (s->file == NULL) {
+            DPRINTF("failed to open migration target");
+            migrate_fd_error(s);
+            return;
+        }
+
         migrate_fd_connect(s);
+        return;
     }
+
+fail:
+    s->file = NULL;
+    migrate_fd_error(s);
 }
 
 void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp)
-- 
1.7.7.6

[Qemu-devel] [PATCH 8/8] migration-unix: side channel support on unix incoming

[Lei Li]

Add side channel support on incoming part of unix migration to enable
localhost migration with page flipping approach. If there is a pipe
available by checking recv_pipefd(), it will fetch the pipe file
descriptor and open it by qemu_fopen_pipe() with read mode for the
pipe protocol introduced.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-unix.c |   14 +++++++++++++-
 1 files changed, 13 insertions(+), 1 deletions(-)

diff --git a/migration-unix.c b/migration-unix.c
index 0bfc1c7..878e063 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -85,6 +85,7 @@ static void unix_accept_incoming_migration(void *opaque)
     int s = (intptr_t)opaque;
     QEMUFile *f;
     int c;
+    int pipefd;
 
     do {
         c = qemu_accept(s, (struct sockaddr *)&addr, &addrlen);
@@ -99,7 +100,18 @@ static void unix_accept_incoming_migration(void *opaque)
         goto out;
     }
 
-    f = qemu_fopen_socket(c, "rb");
+    /* Check if there is a pipe fd to be recevied for a page-flipping
+     * migration */
+
+    pipefd = recv_pipefd(c);
+    if (pipefd > 0) {
+        DPRINTF("receive pipe file descriptor successfully\n");
+
+        f = qemu_fopen_pipe(pipefd, "r");
+    } else {
+        f = qemu_fopen_socket(c, "rb");
+    }
+
     if (f == NULL) {
         fprintf(stderr, "could not qemu_fopen socket\n");
         goto out;
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Paolo Bonzini]

Il 25/09/2013 16:32, Lei Li ha scritto:
> This RFC patch series tries to introduce a mechanism using side
> channel pipe for RAM via SCM_RIGHTS with unix domain socket
> protocol migration.
> 
> This side channel will be used for the page flipping by vmsplice,
> which will be the internal mechanism for localhost migration that
> we are trying to add. The previous patch series for localhost migration
> as link,
> 
> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html
> 
> After this series, will adjust the process of current migration for
> the localhost migration and involve the vmsplice based on the previous
> patch set as link above.
> 
> Please let me know if it is the proper way for it or there is anything
> need to be improved. Your suggestions and comments are very welcome, and
> thanks for Paolo for his review and useful suggestions.
> 
> 
> Lei Li (8):
>   migration-local: add pipe protocol for QEMUFileOps
>   migration-local: add qemu_fopen_pipe()
>   migration-local: add send_pipefd()
>   migration-local: add recv_pipefd()
>   QAPI: introduce magration capability unix_page_flipping
>   migration: add migrate_unix_page_flipping()
>   migration-unix: side channel support on unix outgoing
>   migration-unix: side channel support on unix incoming
> 
>  Makefile.target               |    1 +
>  include/migration/migration.h |    3 +
>  include/migration/qemu-file.h |    4 +
>  migration-local.c             |  247 +++++++++++++++++++++++++++++++++++++++++
>  migration-unix.c              |   48 +++++++-
>  migration.c                   |    9 ++
>  qapi-schema.json              |    8 +-
>  7 files changed, 315 insertions(+), 5 deletions(-)
>  create mode 100644 migration-local.c
> 

Yes, this is much closer!

There are two problems to be fixed, but it is getting there.

First, it breaks migration from old QEMU to new QEMU, and also migration
where the source uses "unix:" and the destination uses "fd:" migration
(this should work as long as page flipping is disabled).  The problem is
that recv_pipefd() "eats" one byte, and old versions of QEMU do not send
that byte.

The second problem is that you are not really using a side channel; you
are still using the QEMUFile and relying on the normal migration code to
send pages on the pipe.  This will not be possible when you use vmsplice.

Both problems can be addressed with a single change in your approach:
always use the Unix socket QEMUFile but, if page flipping is enabled,
only transmit page addresses on the socket; page data will be on the
pipe.  You can use hooks such as before_ram_iterate, save_page and
hook_ram_load to do all your customizations: send the pipe file
descriptor, read the pipe file descriptor, and use the pipe as a side
channel.

To fix the first problem, you can use the before_ram_iterate callback to
send the fd, and the hook_ram_load callback to receive it.  The
before_ram_iterate callback can write a special 8-byte record (with the
RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by
send_pipefd().  The load_hook callback is called after the first 8-byte
record is sent, and can just do recv_pipefd().

To fix the second problem, and really use the pipe as a side channel,
you can use the save_page QEMUFile callback on the send side.  This
callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is
disabled.  If it is enabled, it should write another 8-byte record with
the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on
the Unix socket; then write the page data on the pipe, and return 0.  On
the receive side, the 8-byte page address will once more cause the
load_hook callback to be called.  This time you already have a file
descriptor, so you do not need to call recv_pipefd(): you just extract
the page address from the 8-byte record and read the page data from the
pipe.

The basis of your code will still be the socket-based QEMUFile, but
you'll need your own QEMUFile since you're adding Unix-specific
functionality.  For this it is not a problem to have two copies the
QEMUFile code for sockets, one in savevm.c and one in migration-unix.c.
 It's a very small amount of code.

Paolo

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li]

On 09/25/2013 11:02 PM, Paolo Bonzini wrote:
> Il 25/09/2013 16:32, Lei Li ha scritto:
>> This RFC patch series tries to introduce a mechanism using side
>> channel pipe for RAM via SCM_RIGHTS with unix domain socket
>> protocol migration.
>>
>> This side channel will be used for the page flipping by vmsplice,
>> which will be the internal mechanism for localhost migration that
>> we are trying to add. The previous patch series for localhost migration
>> as link,
>>
>> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html
>>
>> After this series, will adjust the process of current migration for
>> the localhost migration and involve the vmsplice based on the previous
>> patch set as link above.
>>
>> Please let me know if it is the proper way for it or there is anything
>> need to be improved. Your suggestions and comments are very welcome, and
>> thanks for Paolo for his review and useful suggestions.
>>
>>
>> Lei Li (8):
>>    migration-local: add pipe protocol for QEMUFileOps
>>    migration-local: add qemu_fopen_pipe()
>>    migration-local: add send_pipefd()
>>    migration-local: add recv_pipefd()
>>    QAPI: introduce magration capability unix_page_flipping
>>    migration: add migrate_unix_page_flipping()
>>    migration-unix: side channel support on unix outgoing
>>    migration-unix: side channel support on unix incoming
>>
>>   Makefile.target               |    1 +
>>   include/migration/migration.h |    3 +
>>   include/migration/qemu-file.h |    4 +
>>   migration-local.c             |  247 +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   48 +++++++-
>>   migration.c                   |    9 ++
>>   qapi-schema.json              |    8 +-
>>   7 files changed, 315 insertions(+), 5 deletions(-)
>>   create mode 100644 migration-local.c
>>
> Yes, this is much closer!
>
> There are two problems to be fixed, but it is getting there.
>
> First, it breaks migration from old QEMU to new QEMU, and also migration
> where the source uses "unix:" and the destination uses "fd:" migration
> (this should work as long as page flipping is disabled).  The problem is
> that recv_pipefd() "eats" one byte, and old versions of QEMU do not send
> that byte.

Hi Paolo,

I didn't consider this, thanks for pointing it out!

> The second problem is that you are not really using a side channel; you
> are still using the QEMUFile and relying on the normal migration code to
> send pages on the pipe.  This will not be possible when you use vmsplice.

Yes, you are right, and I am trying to involve the vmsplice.

>
> Both problems can be addressed with a single change in your approach:
> always use the Unix socket QEMUFile but, if page flipping is enabled,
> only transmit page addresses on the socket; page data will be on the
> pipe.  You can use hooks such as before_ram_iterate, save_page and
> hook_ram_load to do all your customizations: send the pipe file
> descriptor, read the pipe file descriptor, and use the pipe as a side
> channel.
>
> To fix the first problem, you can use the before_ram_iterate callback to
> send the fd, and the hook_ram_load callback to receive it.  The
> before_ram_iterate callback can write a special 8-byte record (with the
> RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by
> send_pipefd().  The load_hook callback is called after the first 8-byte
> record is sent, and can just do recv_pipefd().
>
> To fix the second problem, and really use the pipe as a side channel,
> you can use the save_page QEMUFile callback on the send side.  This
> callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is
> disabled.  If it is enabled, it should write another 8-byte record with
> the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on
> the Unix socket; then write the page data on the pipe, and return 0.  On
> the receive side, the 8-byte page address will once more cause the
> load_hook callback to be called.  This time you already have a file
> descriptor, so you do not need to call recv_pipefd(): you just extract
> the page address from the 8-byte record and read the page data from the
> pipe.

Thanks for your comprehensive suggestions, really nice ideas!

>
> The basis of your code will still be the socket-based QEMUFile, but
> you'll need your own QEMUFile since you're adding Unix-specific
> functionality.  For this it is not a problem to have two copies the
> QEMUFile code for sockets, one in savevm.c and one in migration-unix.c.

Have two copies of the QEMUFile code for sockets, do you mean in my own
QEMUFile, say QEMUFilePipe, includes both the copy of QEMUFileSocket
code (like get_fd, get_buffer, writev_buffer..) and the Unix-specific
functionality code that override these three hooks like your suggestions
above?

I guess 'migration-unix.c' you typed is 'migration-local.c', right?

>   It's a very small amount of code.
>
> Paolo
>


-- 
Lei

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Paolo Bonzini]

Il 26/09/2013 14:44, Lei Li ha scritto:
>>
>> The basis of your code will still be the socket-based QEMUFile, but
>> you'll need your own QEMUFile since you're adding Unix-specific
>> functionality.  For this it is not a problem to have two copies the
>> QEMUFile code for sockets, one in savevm.c and one in migration-unix.c.
> 
> Have two copies of the QEMUFile code for sockets, do you mean in my own
> QEMUFile, say QEMUFilePipe, includes both the copy of QEMUFileSocket
> code (like get_fd, get_buffer, writev_buffer..) and the Unix-specific
> functionality code that override these three hooks like your suggestions
> above?

Yes (the name could be either QEMUFilePipe or QEMUFileUnix, I guess).

> I guess 'migration-unix.c' you typed is 'migration-local.c', right?

I wasn't sure of the reason why 'migration-unix.c' and
'migration-local.c' were split, since now the choice is done with a
capability rather than a different protocol.

Thanks,

Paolo

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li]

On 09/26/2013 08:44 PM, Lei Li wrote:
> On 09/25/2013 11:02 PM, Paolo Bonzini wrote:
>> Il 25/09/2013 16:32, Lei Li ha scritto:
>>> This RFC patch series tries to introduce a mechanism using side
>>> channel pipe for RAM via SCM_RIGHTS with unix domain socket
>>> protocol migration.
>>>
>>> This side channel will be used for the page flipping by vmsplice,
>>> which will be the internal mechanism for localhost migration that
>>> we are trying to add. The previous patch series for localhost migration
>>> as link,
>>>
>>> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html
>>>
>>> After this series, will adjust the process of current migration for
>>> the localhost migration and involve the vmsplice based on the previous
>>> patch set as link above.
>>>
>>> Please let me know if it is the proper way for it or there is anything
>>> need to be improved. Your suggestions and comments are very welcome, 
>>> and
>>> thanks for Paolo for his review and useful suggestions.
>>>
>>>
>>> Lei Li (8):
>>>    migration-local: add pipe protocol for QEMUFileOps
>>>    migration-local: add qemu_fopen_pipe()
>>>    migration-local: add send_pipefd()
>>>    migration-local: add recv_pipefd()
>>>    QAPI: introduce magration capability unix_page_flipping
>>>    migration: add migrate_unix_page_flipping()
>>>    migration-unix: side channel support on unix outgoing
>>>    migration-unix: side channel support on unix incoming
>>>
>>>   Makefile.target               |    1 +
>>>   include/migration/migration.h |    3 +
>>>   include/migration/qemu-file.h |    4 +
>>>   migration-local.c             |  247 
>>> +++++++++++++++++++++++++++++++++++++++++
>>>   migration-unix.c              |   48 +++++++-
>>>   migration.c                   |    9 ++
>>>   qapi-schema.json              |    8 +-
>>>   7 files changed, 315 insertions(+), 5 deletions(-)
>>>   create mode 100644 migration-local.c
>>>
>> Yes, this is much closer!
>>
>> There are two problems to be fixed, but it is getting there.
>>
>> First, it breaks migration from old QEMU to new QEMU, and also migration
>> where the source uses "unix:" and the destination uses "fd:" migration
>> (this should work as long as page flipping is disabled).  The problem is
>> that recv_pipefd() "eats" one byte, and old versions of QEMU do not send
>> that byte.
>
> Hi Paolo,
>
> I didn't consider this, thanks for pointing it out!
>
>> The second problem is that you are not really using a side channel; you
>> are still using the QEMUFile and relying on the normal migration code to
>> send pages on the pipe.  This will not be possible when you use 
>> vmsplice.
>
> Yes, you are right, and I am trying to involve the vmsplice.
>
>>
>> Both problems can be addressed with a single change in your approach:
>> always use the Unix socket QEMUFile but, if page flipping is enabled,
>> only transmit page addresses on the socket; page data will be on the
>> pipe.  You can use hooks such as before_ram_iterate, save_page and
>> hook_ram_load to do all your customizations: send the pipe file
>> descriptor, read the pipe file descriptor, and use the pipe as a side
>> channel.
>>
>> To fix the first problem, you can use the before_ram_iterate callback to
>> send the fd, and the hook_ram_load callback to receive it.  The
>> before_ram_iterate callback can write a special 8-byte record (with the
>> RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by
>> send_pipefd().  The load_hook callback is called after the first 8-byte
>> record is sent, and can just do recv_pipefd().

Hi Paolo,

When debugging the code, I realized that this problem might still exist. In the
incoming part, it will qemu_fopen_pipe() in unix_accept_incoming_migration
first to enable the load_hook callback, the check action of this RAM_SAVE_FLAG_HOOK
flags would lead to 8 bytes taken. Turns out, it will break normal unix migration
(without unix-page-flipping), because no matter normal unix migration or
unix-page-flipping migration, the incoming side has to check this 8-byes flags
first to decide whether the load_hook is called, and normal unix migration
did not send this 8-byte flags.

I wonder if I didn't understand your suggestion correctly?

>>
>> To fix the second problem, and really use the pipe as a side channel,
>> you can use the save_page QEMUFile callback on the send side. This
>> callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is
>> disabled.  If it is enabled, it should write another 8-byte record with
>> the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on
>> the Unix socket; then write the page data on the pipe, and return 0.  On
>> the receive side, the 8-byte page address will once more cause the
>> load_hook callback to be called.  This time you already have a file
>> descriptor, so you do not need to call recv_pipefd(): you just extract
>> the page address from the 8-byte record and read the page data from the
>> pipe.
>
> Thanks for your comprehensive suggestions, really nice ideas!
>
>>
>> The basis of your code will still be the socket-based QEMUFile, but
>> you'll need your own QEMUFile since you're adding Unix-specific
>> functionality.  For this it is not a problem to have two copies the
>> QEMUFile code for sockets, one in savevm.c and one in migration-unix.c.
>
> Have two copies of the QEMUFile code for sockets, do you mean in my own
> QEMUFile, say QEMUFilePipe, includes both the copy of QEMUFileSocket
> code (like get_fd, get_buffer, writev_buffer..) and the Unix-specific
> functionality code that override these three hooks like your suggestions
> above?
>
> I guess 'migration-unix.c' you typed is 'migration-local.c', right?
>
>>   It's a very small amount of code.
>>
>> Paolo
>>
>
>


-- 
Lei

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Paolo Bonzini]

Il 03/10/2013 06:03, Lei Li ha scritto:
>>>
> 
> Hi Paolo,
> 
> When debugging the code, I realized that this problem might still
> exist. In the incoming part, it will qemu_fopen_pipe() in
> unix_accept_incoming_migration first to enable the load_hook
> callback, the check action of this RAM_SAVE_FLAG_HOOK flags would
> lead to 8 bytes taken. Turns out, it will break normal unix 
> migration (without unix-page-flipping), because no matter normal unix
> migration or unix-page-flipping migration, the incoming side has to
> check this 8-byes flags first to decide whether the load_hook is
> called, and normal unix migration did not send this 8-byte flags.

Why is the load_hook callback being called at all without page flipping?
 Without page flipping, the before_iterate and save_page hook will
return immediately (or depending on your code they may never be called),
so the RAM_SAVE_FLAG_HOOK will never be written to the Unix socket.

> I wonder if I didn't understand your suggestion correctly?

Perhaps you want to discuss this tomorrow morning on #qemu?

Paolo

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li]

On 10/03/2013 04:23 PM, Paolo Bonzini wrote:
> Il 03/10/2013 06:03, Lei Li ha scritto:
>> Hi Paolo,
>>
>> When debugging the code, I realized that this problem might still
>> exist. In the incoming part, it will qemu_fopen_pipe() in
>> unix_accept_incoming_migration first to enable the load_hook
>> callback, the check action of this RAM_SAVE_FLAG_HOOK flags would
>> lead to 8 bytes taken. Turns out, it will break normal unix
>> migration (without unix-page-flipping), because no matter normal unix
>> migration or unix-page-flipping migration, the incoming side has to
>> check this 8-byes flags first to decide whether the load_hook is
>> called, and normal unix migration did not send this 8-byte flags.
> Why is the load_hook callback being called at all without page flipping?
>   Without page flipping, the before_iterate and save_page hook will
> return immediately (or depending on your code they may never be called),
> so the RAM_SAVE_FLAG_HOOK will never be written to the Unix socket.

The load_hook callback is only be called if the RAM_SAVE_FLAG_HOOK is received.
To check this flags, it means there would be a check action first in
unix_accept_incoming_migration(), like:

f = qemu_fopen_pipe(c, "rb");
flags = qemu_get_be64(f);
if (flags == RAM_SAVE_FLAG_HOOK) {
     load_hook();
     ...
}

Otherwise, the incoming side has no idea whether the special 8-bytes record
(RAM_SAVE_FLAG_HOOK) is sent.

In unix-page-flipping migration, it is OK. Without page flipping, since the
RAM_SAVE_FLAG_HOOK is not be written to the Unix socket, but the incoming
side will still check it, that will lead to the unexpected 8-bytes taken.

If the logic and the way to deal with it above is correct according to your
suggestion, how about:

1) Use another Unix socket to deal with this flags and pipe fd passing.
or 2) Use a new prefix URI for the incoming.

>
>> I wonder if I didn't understand your suggestion correctly?
> Perhaps you want to discuss this tomorrow morning on #qemu?

I joined the #qemu channel just now, seems you were not there.
I guess it's your lunch time right now. :)

>
> Paolo
>


-- 
Lei

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Paolo Bonzini]

Il 03/10/2013 12:28, Lei Li ha scritto:
> 
> The load_hook callback is only be called if the RAM_SAVE_FLAG_HOOK is
> received.
> To check this flags, it means there would be a check action first in
> unix_accept_incoming_migration(), like:
> 
> f = qemu_fopen_pipe(c, "rb");
> flags = qemu_get_be64(f);
> if (flags == RAM_SAVE_FLAG_HOOK) {
>     load_hook();
>     ...
> }
> 
> Otherwise, the incoming side has no idea whether the special 8-bytes record
> (RAM_SAVE_FLAG_HOOK) is sent.

No, ram_load is taking care of checking for RAM_SAVE_FLAG_HOOK.  If
before_iterate writes the 8 bytes (followed by passing the fd for the
pipe's read-side via SCM_RIGHTS), ram_load will call load_hook before it
loads any page and load_hook will fetch the fd.

Subsequent calls to load_hook will match data written by the sender's
save_page hook (so they contain a RAM address, with the 4k page data
sent on the pipe).

Paolo

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li]

On 10/03/2013 06:34 PM, Paolo Bonzini wrote:
> Il 03/10/2013 12:28, Lei Li ha scritto:
>> The load_hook callback is only be called if the RAM_SAVE_FLAG_HOOK is
>> received.
>> To check this flags, it means there would be a check action first in
>> unix_accept_incoming_migration(), like:
>>
>> f = qemu_fopen_pipe(c, "rb");
>> flags = qemu_get_be64(f);
>> if (flags == RAM_SAVE_FLAG_HOOK) {
>>      load_hook();
>>      ...
>> }
>>
>> Otherwise, the incoming side has no idea whether the special 8-bytes record
>> (RAM_SAVE_FLAG_HOOK) is sent.
> No, ram_load is taking care of checking for RAM_SAVE_FLAG_HOOK.  If
> before_iterate writes the 8 bytes (followed by passing the fd for the
> pipe's read-side via SCM_RIGHTS), ram_load will call load_hook before it
> loads any page and load_hook will fetch the fd.

If let ram_load take care of checking for RAM_SAVE_FLAG_HOOK, then in
unix_accept_incoming_migration(), how to decide which QEMUFile should
be opened? Since there would be two types of QEMUFile, one is the original
QEMUFile opened by qemu_fopen_socket() for normal Unix migration, the
other is opened by qemu_fopen_pipe() for unix-page-flipping migration.

Or, were you suggesting replace this qemu_fopen_socket() with the
qemu_fopen_pipe(), which also contain the copy of the QEMUFile code for
Unix sockets?

>
> Subsequent calls to load_hook will match data written by the sender's
> save_page hook (so they contain a RAM address, with the 4k page data
> sent on the pipe).
>
> Paolo
>


-- 
Lei

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Paolo Bonzini]

Il 03/10/2013 15:29, Lei Li ha scritto:
> On 10/03/2013 06:34 PM, Paolo Bonzini wrote:
>> Il 03/10/2013 12:28, Lei Li ha scritto:
>>> The load_hook callback is only be called if the RAM_SAVE_FLAG_HOOK is
>>> received.
>>> To check this flags, it means there would be a check action first in
>>> unix_accept_incoming_migration(), like:
>>>
>>> f = qemu_fopen_pipe(c, "rb");
>>> flags = qemu_get_be64(f);
>>> if (flags == RAM_SAVE_FLAG_HOOK) {
>>>      load_hook();
>>>      ...
>>> }
>>>
>>> Otherwise, the incoming side has no idea whether the special 8-bytes
>>> record
>>> (RAM_SAVE_FLAG_HOOK) is sent.
>> No, ram_load is taking care of checking for RAM_SAVE_FLAG_HOOK.  If
>> before_iterate writes the 8 bytes (followed by passing the fd for the
>> pipe's read-side via SCM_RIGHTS), ram_load will call load_hook before it
>> loads any page and load_hook will fetch the fd.
> 
> If let ram_load take care of checking for RAM_SAVE_FLAG_HOOK, then in
> unix_accept_incoming_migration(), how to decide which QEMUFile should
> be opened? Since there would be two types of QEMUFile, one is the original
> QEMUFile opened by qemu_fopen_socket() for normal Unix migration, the
> other is opened by qemu_fopen_pipe() for unix-page-flipping migration.
> 
> Or, were you suggesting replace this qemu_fopen_socket() with the
> qemu_fopen_pipe(), which also contain the copy of the QEMUFile code for
> Unix sockets?

Yes (though I'd call it qemu_fopen_socket_local() or something like that).

On the incoming side, if non-page-flipping was enabled you will use the
normal RAM loading code, if page-flipping was enabled you will get
load_hook calls.

Paolo

>>
>> Subsequent calls to load_hook will match data written by the sender's
>> save_page hook (so they contain a RAM address, with the 4k page data
>> sent on the pipe).
>>
>> Paolo
>>
> 
> 

Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li]

On 10/03/2013 09:34 PM, Paolo Bonzini wrote:
> Il 03/10/2013 15:29, Lei Li ha scritto:
>> On 10/03/2013 06:34 PM, Paolo Bonzini wrote:
>>> Il 03/10/2013 12:28, Lei Li ha scritto:
>>>> The load_hook callback is only be called if the RAM_SAVE_FLAG_HOOK is
>>>> received.
>>>> To check this flags, it means there would be a check action first in
>>>> unix_accept_incoming_migration(), like:
>>>>
>>>> f = qemu_fopen_pipe(c, "rb");
>>>> flags = qemu_get_be64(f);
>>>> if (flags == RAM_SAVE_FLAG_HOOK) {
>>>>       load_hook();
>>>>       ...
>>>> }
>>>>
>>>> Otherwise, the incoming side has no idea whether the special 8-bytes
>>>> record
>>>> (RAM_SAVE_FLAG_HOOK) is sent.
>>> No, ram_load is taking care of checking for RAM_SAVE_FLAG_HOOK.  If
>>> before_iterate writes the 8 bytes (followed by passing the fd for the
>>> pipe's read-side via SCM_RIGHTS), ram_load will call load_hook before it
>>> loads any page and load_hook will fetch the fd.
>> If let ram_load take care of checking for RAM_SAVE_FLAG_HOOK, then in
>> unix_accept_incoming_migration(), how to decide which QEMUFile should
>> be opened? Since there would be two types of QEMUFile, one is the original
>> QEMUFile opened by qemu_fopen_socket() for normal Unix migration, the
>> other is opened by qemu_fopen_pipe() for unix-page-flipping migration.
>>
>> Or, were you suggesting replace this qemu_fopen_socket() with the
>> qemu_fopen_pipe(), which also contain the copy of the QEMUFile code for
>> Unix sockets?
> Yes (though I'd call it qemu_fopen_socket_local() or something like that).
>
> On the incoming side, if non-page-flipping was enabled you will use the
> normal RAM loading code, if page-flipping was enabled you will get
> load_hook calls.

Ah, I see. :)
qemu_fopen_socket_local() sounds good, thanks!

> Paolo
>
>>> Subsequent calls to load_hook will match data written by the sender's
>>> save_page hook (so they contain a RAM address, with the 4k page data
>>> sent on the pipe).
>>>
>>> Paolo
>>>
>>


-- 
Lei