[Qemu-devel] [PATCH 11/17] add argument ram_addr_t to hook_ram_load

[Qemu-devel] [PATCH 11/17] add argument ram_addr_t to hook_ram_load

[Lei Li]

Adds argument ram_addr_t to hook_ram_load, and replaces
QEMURamHookFunc with QEMURamLoadHookFunc for it. With this
new argument, it will allows cut almost half of the data
transferred on the Unix socket using by page flipping
migraton.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 arch_init.c                   |    2 +-
 include/migration/migration.h |    2 +-
 include/migration/qemu-file.h |   11 ++++++++++-
 migration-rdma.c              |    2 +-
 savevm.c                      |    4 ++--
 5 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index daaa519..0621893 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -945,7 +945,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
                 goto done;
             }
         } else if (flags & RAM_SAVE_FLAG_HOOK) {
-            ram_control_load_hook(f, flags);
+            ram_control_load_hook(f, addr, flags);
         }
         error = qemu_file_get_error(f);
         if (error) {
diff --git a/include/migration/migration.h b/include/migration/migration.h
index ca852a8..300e52c 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -149,7 +149,7 @@ int64_t xbzrle_cache_resize(int64_t new_size);
 
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
-void ram_control_load_hook(QEMUFile *f, uint64_t flags);
+void ram_control_load_hook(QEMUFile *f, ram_addr_t addr, uint64_t flags);
 
 /* Whenever this is found in the data stream, the flags
  * will be passed to ram_control_load_hook in the incoming-migration
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 6646e89..176c2d9 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -65,6 +65,15 @@ typedef ssize_t (QEMUFileWritevBufferFunc)(void *opaque, struct iovec *iov,
 typedef int (QEMURamHookFunc)(QEMUFile *f, void *opaque, uint64_t flags);
 
 /*
+ * This function provides load hook for RAM migration, allows
+ * override of where the RAM page is loaded (such as page
+ * flipping for example).
+ */
+typedef int (QEMURamLoadHookFunc)(QEMUFile *f, void *opaque,
+                                  ram_addr_t addr,
+                                  uint64_t flags);
+
+/*
  * Constants used by ram_control_* hooks
  */
 #define RAM_CONTROL_SETUP    0
@@ -90,7 +99,7 @@ typedef struct QEMUFileOps {
     QEMUFileWritevBufferFunc *writev_buffer;
     QEMURamHookFunc *before_ram_iterate;
     QEMURamHookFunc *after_ram_iterate;
-    QEMURamHookFunc *hook_ram_load;
+    QEMURamLoadHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
 } QEMUFileOps;
 
diff --git a/migration-rdma.c b/migration-rdma.c
index ae04de4..732ec1a 100644
--- a/migration-rdma.c
+++ b/migration-rdma.c
@@ -2938,7 +2938,7 @@ err_rdma_dest_wait:
  * Keep doing this until the source tells us to stop.
  */
 static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque,
-                                         uint64_t flags)
+                                         ram_addr_t offset, uint64_t flags)
 {
     RDMAControlHeader reg_resp = { .len = sizeof(RDMARegisterResult),
                                .type = RDMA_CONTROL_REGISTER_RESULT,
diff --git a/savevm.c b/savevm.c
index 137e74f..75e397c 100644
--- a/savevm.c
+++ b/savevm.c
@@ -647,12 +647,12 @@ void ram_control_after_iterate(QEMUFile *f, uint64_t flags)
     }
 }
 
-void ram_control_load_hook(QEMUFile *f, uint64_t flags)
+void ram_control_load_hook(QEMUFile *f, ram_addr_t offset, uint64_t flags)
 {
     int ret = -EINVAL;
 
     if (f->ops->hook_ram_load) {
-        ret = f->ops->hook_ram_load(f, f->opaque, flags);
+        ret = f->ops->hook_ram_load(f, f->opaque, offset, flags);
         if (ret < 0) {
             qemu_file_set_error(f, ret);
         }
-- 
1.7.7.6

[Qemu-devel] [PATCH 0/17 v5] Localhost migration with side channel for ram

[Lei Li]

This patch series tries to introduce a mechanism using side
channel pipe for RAM via SCM_RIGHTS with unix domain socket
protocol migration.

This side channel is used for the page flipping by vmsplice,
which is the internal mechanism for localhost migration that
we are trying to add to QEMU. The backgroud info and previous
patch series for reference,

Localhost migration
http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html

migration: Introduce side channel for RAM
http://lists.gnu.org/archive/html/qemu-devel/2013-09/msg04043.html

I have picked patches from the localhost migration series and rebased 
it on the series of side channel, now it is a complete series that
passed the basic test.

Please let me know if there is anything needs to be fixed or improved.
Your suggestions and comments are very welcome, and thanks to Paolo
for his continued review and useful suggestions.


Changes since V4:
  Rename the capability to x-unix-page-flipping for now. (Paolo)

Changes since V3:
  Address comments from Paolo including:

    - Get rid of useless check in send_pipefd() and the override
      of before_ram_iterate, send pipefd in the first save_page
      call, qemu_get_byte() in the first ram_load correspondingly.
    - Add new argument ram_addr_t to hook_ram_load to cut half of
      the data transferred on the socket.
    - Add transition from 'debug' to 'memory-stale'.
    - Other minor fixes.

Changes since V2:
  Address comments from Paolo including:

    - Doc improvement for QAPI.
    - Use callback get_buffer as the only one receiver.
    - Rename the new RunState flipping-migrate to memory-stale, and
      add transition from 'prelaunch' to 'memory-stale'.
    - Other minor fixes.

Changes since V1:
  Address suggestions from Paolo Bonzini including:

    - Use Unix socket QEMUFile as basis of code and adjust the way
      of overriding RDMA hooks.
    - Involve the vmsplice for page flipping.
    - Add new RunState RUN_STATE_FLIPPING_MIGRATE and add it to
      runstate_needs_reset() for the adjustment of the current
      migration process with page flipping.



Lei Li (17):
  QAPI: introduce magration capability unix_page_flipping
  migration: add migrate_unix_page_flipping()
  qmp-command.hx: add missing docs for migration capabilites
  migration-local: add QEMUFileLocal with socket based QEMUFile
  migration-local: introduce qemu_fopen_socket_local()
  migration-local: add send_pipefd()
  save_page: replace block_offset with a MemoryRegion
  migration-local: override save_page for page transmit
  savevm: adjust ram_control_save_page with page flipping
  add unix_msgfd_lookup() to callback get_buffer
  add argument ram_addr_t to hook_ram_load
  migration-local: override hook_ram_load 
  migration-unix: replace qemu_fopen_socket with qemu_fopen_socket_local
  add new RanState RAN_STATE_MEMORY_STALE
  migration-unix: page flipping support on unix outgoing
  migration: adjust migration_thread() process for unix_page_flipping
  hmp: better format for info migrate_capabilities

 Makefile.target               |   1 +
 arch_init.c                   |   4 +-
 migration-local.c             | 512 ++++++++++++++++++++++++++++++++++++++++++
 hmp.c                         |   5 +-
 include/migration/migration.h |   3 +
 include/migration/qemu-file.h |   2 +
 migration-unix.c              |  27 ++-
 migration-rdma.c              |   4 +-
 migration.c                   |  18 +-
 qapi-schema.json              |  18 +-
 qmp-commands.hx               |   8 +
 savevm.c                      |  21 +-
 vl.c                          |  12 +-
 13 files changed, 617 insertions(+), 27 deletions(-)
 create mode 100644 migration-local.c

[Qemu-devel] [PATCH 01/17] QAPI: introduce migration capability x_unix_page_flipping

[Lei Li]

Introduce x_unix_page_flipping to MigrationCapability for
localhost migration.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 qapi-schema.json |   12 +++++++++++-
 1 files changed, 11 insertions(+), 1 deletions(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index 83fa485..ea910ef 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -685,10 +685,20 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @x-unix-page-flipping: If enabled, QEMU can optimize migration when the
+#          destination is a QEMU process that runs on the same host as
+#          the source (as is the case for live upgrade).  If the migration
+#          transport is a Unix socket, QEMU will flip RAM pages directly to
+#          the destination, so that memory is only allocated twice for the
+#          source and destination processes. Disabled by default.
+#          Experimental: will get rid of the x tag after further testing with
+#          the new vmsplice. (since 2.0)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle', 'x-rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+  'data': ['xbzrle', 'x-rdma-pin-all', 'auto-converge', 'zero-blocks',
+           'x-unix-page-flipping'] }
 
 ##
 # @MigrationCapabilityStatus
-- 
1.7.7.6

[Qemu-devel] [PATCH 02/17] migration: add migrate_unix_page_flipping()

[Lei Li]

Add migrate_unix_page_flipping() to check if
MIGRATION_CAPABILITY_X_UNIX_PAGE_FLIPPING is enabled.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/migration.h |    3 +++
 migration.c                   |    9 +++++++++
 2 files changed, 12 insertions(+), 0 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 140e6b4..7e5d01a 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -131,10 +131,13 @@ void migrate_add_blocker(Error *reason);
 void migrate_del_blocker(Error *reason);
 
 bool migrate_rdma_pin_all(void);
+
 bool migrate_zero_blocks(void);
 
 bool migrate_auto_converge(void);
 
+bool migrate_unix_page_flipping(void);
+
 int xbzrle_encode_buffer(uint8_t *old_buf, uint8_t *new_buf, int slen,
                          uint8_t *dst, int dlen);
 int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen);
diff --git a/migration.c b/migration.c
index 2b1ab20..e012cd4 100644
--- a/migration.c
+++ b/migration.c
@@ -541,6 +541,15 @@ int64_t migrate_xbzrle_cache_size(void)
     return s->xbzrle_cache_size;
 }
 
+bool migrate_unix_page_flipping(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_X_UNIX_PAGE_FLIPPING];
+}
+
 /* migration thread support */
 
 static void *migration_thread(void *opaque)
-- 
1.7.7.6

[Qemu-devel] [PATCH 03/17] qmp-command.hx: add missing docs for migration capabilites

[Lei Li]

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 qmp-commands.hx |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/qmp-commands.hx b/qmp-commands.hx
index fba15cd..0df08c0 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -2898,6 +2898,10 @@ migrate-set-capabilities
 Enable/Disable migration capabilities
 
 - "xbzrle": XBZRLE support
+- "x-rdma-pin-all": Pin all pages during RDMA support
+- "zero-blocks": Compress zero blocks during block migration
+- "auto-converge": Block VCPU to help convergence of migration
+- "x-unix-page-flipping": Page flipping for live QEMU upgrade
 
 Arguments:
 
@@ -2922,6 +2926,10 @@ Query current migration capabilities
 
 - "capabilities": migration capabilities state
          - "xbzrle" : XBZRLE state (json-bool)
+         - "x-rdma-pin-all": RDMA state (json-bool)
+         - "zero-blocks": zero-blocks state (json-bool)
+         - "auto-converge": Auto converge state (json-bool)
+         - "x-unix-page-flipping": Page flipping state (json-bool)
 
 Arguments:
 
-- 
1.7.7.6

[Qemu-devel] [PATCH 04/17] migration-local: add QEMUFileLocal with socket based QEMUFile

[Lei Li]

This patch adds QEMUFileLocal with copy of socket based QEMUFile, will
be used as the basis code for Unix socket protocol migration and page
flipping migration.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 Makefile.target   |    1 +
 migration-local.c |  123 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+), 0 deletions(-)
 create mode 100644 migration-local.c

diff --git a/Makefile.target b/Makefile.target
index af6ac7e..aa09960 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -117,6 +117,7 @@ obj-$(CONFIG_KVM) += kvm-all.o
 obj-y += memory.o savevm.o cputlb.o
 obj-y += memory_mapping.o
 obj-y += dump.o
+obj-y += migration-local.o
 LIBS+=$(libs_softmmu)
 
 # xen support
diff --git a/migration-local.c b/migration-local.c
new file mode 100644
index 0000000..ca01a20
--- /dev/null
+++ b/migration-local.c
@@ -0,0 +1,123 @@
+/*
+ * QEMU localhost migration with page flipping
+ *
+ * Copyright IBM, Corp. 2013
+ *
+ * Authors:
+ *   Lei Li   <lilei@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#include "config-host.h"
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "exec/cpu-common.h"
+#include "config.h"
+#include "exec/cpu-all.h"
+#include "exec/memory.h"
+#include "exec/memory-internal.h"
+#include "monitor/monitor.h"
+#include "migration/qemu-file.h"
+#include "qemu/iov.h"
+#include "sysemu/arch_init.h"
+#include "sysemu/sysemu.h"
+#include "block/block.h"
+#include "qemu/sockets.h"
+#include "migration/block.h"
+#include "qemu/thread.h"
+#include "qmp-commands.h"
+#include "trace.h"
+#include "qemu/osdep.h"
+
+//#define DEBUG_MIGRATION_LOCAL
+
+#ifdef DEBUG_MIGRATION_LOCAL
+#define DPRINTF(fmt, ...) \
+    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+    do { } while (0)
+#endif
+
+
+typedef struct QEMUFileLocal {
+    QEMUFile *file;
+    int sockfd;
+    int pipefd[2];
+    int pipefd_passed;
+    int pipefd_received;
+    bool unix_page_flipping;
+} QEMUFileLocal;
+
+static int qemu_local_get_sockfd(void *opaque)
+{
+    QEMUFileLocal *s = opaque;
+
+    return s->sockfd;
+}
+
+static int qemu_local_get_buffer(void *opaque, uint8_t *buf,
+                                 int64_t pos, int size)
+{
+    QEMUFileLocal *s = opaque;
+    ssize_t len;
+
+    for (;;) {
+        len = qemu_recv(s->sockfd, buf, size, 0);
+        if (len != -1) {
+            break;
+        }
+
+        if (socket_error() == EAGAIN) {
+            yield_until_fd_readable(s->sockfd);
+        } else if (socket_error() != EINTR) {
+            break;
+        }
+    }
+
+    if (len == -1) {
+        len = -socket_error();
+    }
+
+    return len;
+}
+
+static ssize_t qemu_local_writev_buffer(void *opaque, struct iovec *iov,
+                                        int iovcnt, int64_t pos)
+{
+    QEMUFileLocal *s = opaque;
+    ssize_t len;
+    ssize_t size = iov_size(iov, iovcnt);
+
+    len = iov_send(s->sockfd, iov, iovcnt, 0, size);
+    if (len < size) {
+        len = -socket_error();
+    }
+
+    return len;
+}
+
+static int qemu_local_close(void *opaque)
+{
+    QEMUFileLocal *s = opaque;
+
+    closesocket(s->sockfd);
+    g_free(s);
+
+    return 0;
+}
+
+static const QEMUFileOps pipe_read_ops = {
+    .get_fd        = qemu_local_get_sockfd,
+    .get_buffer    = qemu_local_get_buffer,
+    .close         = qemu_local_close,
+};
+
+static const QEMUFileOps pipe_write_ops = {
+    .get_fd             = qemu_local_get_sockfd,
+    .writev_buffer      = qemu_local_writev_buffer,
+    .close              = qemu_local_close,
+};
-- 
1.7.7.6

[Qemu-devel] [PATCH 05/17] migration-local: introduce qemu_fopen_socket_local()

[Lei Li]

Add qemu_fopen_socket_local() to open QEMUFileLocal introduced
earlier. It will create a pipe in write mode if unix_page_flipping
is enabled, adjust qemu_local_close() to close pipe as well.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/qemu-file.h |    2 +
 migration-local.c             |   46 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 48 insertions(+), 0 deletions(-)

diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 0f757fb..f9b104a 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -99,6 +99,8 @@ QEMUFile *qemu_fopen(const char *filename, const char *mode);
 QEMUFile *qemu_fdopen(int fd, const char *mode);
 QEMUFile *qemu_fopen_socket(int fd, const char *mode);
 QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
+QEMUFile *qemu_fopen_socket_local(int sockfd, const char *mode);
+
 int qemu_get_fd(QEMUFile *f);
 int qemu_fclose(QEMUFile *f);
 int64_t qemu_ftell(QEMUFile *f);
diff --git a/migration-local.c b/migration-local.c
index ca01a20..929ed60 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -105,6 +105,12 @@ static int qemu_local_close(void *opaque)
     QEMUFileLocal *s = opaque;
 
     closesocket(s->sockfd);
+
+    if (s->unix_page_flipping) {
+        close(s->pipefd[0]);
+        close(s->pipefd[1]);
+    }
+
     g_free(s);
 
     return 0;
@@ -121,3 +127,43 @@ static const QEMUFileOps pipe_write_ops = {
     .writev_buffer      = qemu_local_writev_buffer,
     .close              = qemu_local_close,
 };
+
+QEMUFile *qemu_fopen_socket_local(int sockfd, const char *mode)
+{
+    QEMUFileLocal *s;
+    int pipefd[2];
+
+    if (qemu_file_mode_is_not_valid(mode)) {
+        return NULL;
+    }
+
+    s = g_malloc0(sizeof(QEMUFileLocal));
+    s->sockfd = sockfd;
+
+    if (migrate_unix_page_flipping()) {
+        s->unix_page_flipping = 1;
+    }
+
+    if (mode[0] == 'w') {
+        if (s->unix_page_flipping) {
+            if (pipe(pipefd) < 0) {
+                fprintf(stderr, "failed to create PIPE\n");
+                goto fail;
+            }
+
+            s->pipefd[0] = pipefd[0];
+            s->pipefd[1] = pipefd[1];
+        }
+
+        qemu_set_block(s->sockfd);
+        s->file = qemu_fopen_ops(s, &pipe_write_ops);
+    } else {
+        s->file = qemu_fopen_ops(s, &pipe_read_ops);
+    }
+
+    return s->file;
+
+fail:
+    g_free(s);
+    return NULL;
+}
-- 
1.7.7.6

[Qemu-devel] [PATCH 06/17] migration-local: add send_pipefd()

[Lei Li]

This patch adds send_pipefd() to pass the pipe file descriptor
to destination process.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-local.c |   46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 46 insertions(+), 0 deletions(-)

diff --git a/migration-local.c b/migration-local.c
index 929ed60..f479530 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -167,3 +167,49 @@ fail:
     g_free(s);
     return NULL;
 }
+
+
+/*
+ * Pass a pipe file descriptor to another process.
+ *
+ * Return negative value If pipefd < 0. Return 0 on
+ * success.
+ *
+ */
+static int send_pipefd(int sockfd, int pipefd)
+{
+    struct msghdr msg;
+    struct iovec iov[1];
+    ssize_t ret;
+    char req[1] = { 0x01 };
+
+    union {
+      struct cmsghdr cm;
+      char control[CMSG_SPACE(sizeof(int))];
+    } control_un;
+    struct cmsghdr *cmptr;
+
+    msg.msg_control = control_un.control;
+    msg.msg_controllen = sizeof(control_un.control);
+
+    cmptr = CMSG_FIRSTHDR(&msg);
+    cmptr->cmsg_len = CMSG_LEN(sizeof(int));
+    cmptr->cmsg_level = SOL_SOCKET;
+    cmptr->cmsg_type = SCM_RIGHTS;
+    *((int *) CMSG_DATA(cmptr)) = pipefd;
+
+    msg.msg_name = NULL;
+    msg.msg_namelen = 0;
+
+    iov[0].iov_base = req;
+    iov[0].iov_len = sizeof(req);
+    msg.msg_iov = iov;
+    msg.msg_iovlen = 1;
+
+    ret = sendmsg(sockfd, &msg, 0);
+    if (ret <= 0) {
+        DPRINTF("sendmsg error: %s\n", strerror(errno));
+    }
+
+    return ret;
+}
-- 
1.7.7.6

[Qemu-devel] [PATCH 07/17] save_page: replace block_offset with a MemoryRegion

[Lei Li]

This patch exports MemoryRegion to save_page hook, replacing
argument ram_addr_t block_offset with a MemoryRegion suggested
by Paolo Bonzini.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 arch_init.c                   |    4 ++--
 include/migration/migration.h |    2 +-
 include/migration/qemu-file.h |    8 ++++----
 migration-rdma.c              |    4 ++--
 savevm.c                      |    8 ++++----
 5 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index e0acbc5..daaa519 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -485,8 +485,8 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
 
             /* In doubt sent page as normal */
             bytes_sent = -1;
-            ret = ram_control_save_page(f, block->offset,
-                               offset, TARGET_PAGE_SIZE, &bytes_sent);
+            ret = ram_control_save_page(f, mr, offset, TARGET_PAGE_SIZE,
+                                        &bytes_sent);
 
             if (ret != RAM_SAVE_CONTROL_NOT_SUPP) {
                 if (ret != RAM_SAVE_CONTROL_DELAYED) {
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 7e5d01a..ca852a8 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -161,7 +161,7 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags);
 #define RAM_SAVE_CONTROL_NOT_SUPP -1000
 #define RAM_SAVE_CONTROL_DELAYED  -2000
 
-size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
+size_t ram_control_save_page(QEMUFile *f, MemoryRegion *mr,
                              ram_addr_t offset, size_t size,
                              int *bytes_sent);
 
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index f9b104a..6646e89 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -77,10 +77,10 @@ typedef int (QEMURamHookFunc)(QEMUFile *f, void *opaque, uint64_t flags);
  * is saved (such as RDMA, for example.)
  */
 typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
-                               ram_addr_t block_offset,
-                               ram_addr_t offset,
-                               size_t size,
-                               int *bytes_sent);
+                                 MemoryRegion *mr,
+                                 ram_addr_t offset,
+                                 size_t size,
+                                 int *bytes_sent);
 
 typedef struct QEMUFileOps {
     QEMUFilePutBufferFunc *put_buffer;
diff --git a/migration-rdma.c b/migration-rdma.c
index f94f3b4..ae04de4 100644
--- a/migration-rdma.c
+++ b/migration-rdma.c
@@ -2699,7 +2699,7 @@ static int qemu_rdma_close(void *opaque)
  *                  the protocol because most transfers are sent asynchronously.
  */
 static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque,
-                                  ram_addr_t block_offset, ram_addr_t offset,
+                                  MemoryRegion *mr, ram_addr_t offset,
                                   size_t size, int *bytes_sent)
 {
     QEMUFileRDMA *rfile = opaque;
@@ -2716,7 +2716,7 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque,
          * is full, or the page doen't belong to the current chunk,
          * an actual RDMA write will occur and a new chunk will be formed.
          */
-        ret = qemu_rdma_write(f, rdma, block_offset, offset, size);
+        ret = qemu_rdma_write(f, rdma, mr->ram_addr, offset, size);
         if (ret < 0) {
             fprintf(stderr, "rdma migration: write error! %d\n", ret);
             goto err;
diff --git a/savevm.c b/savevm.c
index 3f912dd..06c1f29 100644
--- a/savevm.c
+++ b/savevm.c
@@ -661,12 +661,12 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags)
     }
 }
 
-size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
-                         ram_addr_t offset, size_t size, int *bytes_sent)
+size_t ram_control_save_page(QEMUFile *f, MemoryRegion *mr, ram_addr_t offset,
+                             size_t size, int *bytes_sent)
 {
     if (f->ops->save_page) {
-        int ret = f->ops->save_page(f, f->opaque, block_offset,
-                                    offset, size, bytes_sent);
+        int ret = f->ops->save_page(f, f->opaque, mr, offset,
+                                    size, bytes_sent);
 
         if (ret != RAM_SAVE_CONTROL_DELAYED) {
             if (bytes_sent && *bytes_sent > 0) {
-- 
1.7.7.6

[Qemu-devel] [PATCH 08/17] migration-local: override save_page for page transmit

[Lei Li]

This patch implements save_page callback for the outside
of page flipping. It will write the address of the page
on the Unix socket and flip the page data on pipe by
vmsplice(). Every page address would have a header flag
RAM_SAVE_FLAG_HOOK, which will trigger the load hook to
receive it in incoming side as well.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-local.c |   63 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 63 insertions(+), 0 deletions(-)

diff --git a/migration-local.c b/migration-local.c
index f479530..9453ec8 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -116,6 +116,68 @@ static int qemu_local_close(void *opaque)
     return 0;
 }
 
+static int send_pipefd(int sockfd, int pipefd);
+
+static size_t qemu_local_save_ram(QEMUFile *f, void *opaque,
+                                  MemoryRegion *mr, ram_addr_t offset,
+                                  size_t size, int *bytes_sent)
+{
+    QEMUFileLocal *s = opaque;
+    ram_addr_t current_addr = mr->ram_addr + offset;
+    void *ram_addr;
+    ssize_t ret;
+
+    if (s->unix_page_flipping) {
+        qemu_put_be64(s->file, current_addr | RAM_SAVE_FLAG_HOOK);
+        qemu_fflush(s->file);
+
+        if (!s->pipefd_passed) {
+            ret = send_pipefd(s->sockfd, s->pipefd[0]);
+            if (ret < 0) {
+                fprintf(stderr, "failed to pass PIPE\n");
+                return ret;
+            }
+            s->pipefd_passed = true;
+        }
+
+        ram_addr = memory_region_get_ram_ptr(mr) + offset;
+
+        /* vmsplice page data to pipe */
+        struct iovec iov = {
+            .iov_base = ram_addr,
+            .iov_len  = size,
+        };
+
+        /*
+         * The flag SPLICE_F_MOVE is introduced in kernel for the page
+         * flipping feature in QEMU, which will move pages rather than
+         * copying, previously unused.
+         *
+         * If a move is not possible the kernel will transparently fall
+         * back to copying data.
+         *
+         * For older kernels the SPLICE_F_MOVE would be ignored and a copy
+         * would occur.
+         */
+
+        ret = vmsplice(s->pipefd[1], &iov, 1, SPLICE_F_GIFT | SPLICE_F_MOVE);
+        if (ret == -1) {
+            if (errno != EAGAIN && errno != EINTR) {
+                fprintf(stderr, "vmsplice save error: %s\n", strerror(errno));
+                return ret;
+            }
+        } else {
+            if (bytes_sent) {
+                *bytes_sent = size;
+            }
+            DPRINTF("block_offset: %lu, offset: %lu\n", mr->ram_addr, offset);
+            return 0;
+        }
+    }
+
+    return RAM_SAVE_CONTROL_NOT_SUPP;
+}
+
 static const QEMUFileOps pipe_read_ops = {
     .get_fd        = qemu_local_get_sockfd,
     .get_buffer    = qemu_local_get_buffer,
@@ -126,6 +188,7 @@ static const QEMUFileOps pipe_write_ops = {
     .get_fd             = qemu_local_get_sockfd,
     .writev_buffer      = qemu_local_writev_buffer,
     .close              = qemu_local_close,
+    .save_page          = qemu_local_save_ram
 };
 
 QEMUFile *qemu_fopen_socket_local(int sockfd, const char *mode)
-- 
1.7.7.6

[Qemu-devel] [PATCH 09/17] savevm: adjust ram_control_save_page for page flipping

[Lei Li]

As callback save_page will always be opened by
qemu_fopen_socket_local(), and without unix_page_flipping
it will return RAM_SAVE_CONTROL_NOT_SUPP, it leads to a
wrong qemu_file_set_error() based on the current logic.
So this patch adds RAM_SAVE_CONTROL_NOT_SUPP to the check.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 savevm.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/savevm.c b/savevm.c
index 06c1f29..137e74f 100644
--- a/savevm.c
+++ b/savevm.c
@@ -668,7 +668,8 @@ size_t ram_control_save_page(QEMUFile *f, MemoryRegion *mr, ram_addr_t offset,
         int ret = f->ops->save_page(f, f->opaque, mr, offset,
                                     size, bytes_sent);
 
-        if (ret != RAM_SAVE_CONTROL_DELAYED) {
+        if (ret != RAM_SAVE_CONTROL_DELAYED &&
+            ret != RAM_SAVE_CONTROL_NOT_SUPP) {
             if (bytes_sent && *bytes_sent > 0) {
                 qemu_update_position(f, *bytes_sent);
             } else if (ret < 0) {
-- 
1.7.7.6

[Qemu-devel] [PATCH 10/17] add unix_msgfd_lookup() to callback get_buffer

[Lei Li]

The control message for exchange of pipe file descriptor should
be received by recvmsg, and it might be eaten to stream file by
qemu_recv() when receiving by two callbacks. So this patch adds
unix_msgfd_lookup() to callback get_buffer as the only one receiver,
where the pipe file descriptor would be caughted.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-local.c |   59 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 56 insertions(+), 3 deletions(-)

diff --git a/migration-local.c b/migration-local.c
index 9453ec8..5f98a01 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -59,16 +59,69 @@ static int qemu_local_get_sockfd(void *opaque)
     return s->sockfd;
 }
 
+static int unix_msgfd_lookup(void *opaque, struct msghdr *msg)
+{
+    QEMUFileLocal *s = opaque;
+    struct cmsghdr *cmsg;
+    bool found = false;
+
+    for (cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg, cmsg)) {
+        if (cmsg->cmsg_len != CMSG_LEN(sizeof(int)) ||
+            cmsg->cmsg_level != SOL_SOCKET ||
+            cmsg->cmsg_type != SCM_RIGHTS)
+            continue;
+
+        /* PIPE file descriptor to be received */
+        s->pipefd[0] = *((int *)CMSG_DATA(cmsg));
+    }
+
+    if (s->pipefd[0] < 0) {
+        fprintf(stderr, "no pipe fd can be received\n");
+        return found;
+    }
+
+    DPRINTF("pipefd successfully received\n");
+    return s->pipefd[0];
+}
+
 static int qemu_local_get_buffer(void *opaque, uint8_t *buf,
                                  int64_t pos, int size)
 {
     QEMUFileLocal *s = opaque;
     ssize_t len;
+    struct msghdr msg = { NULL, };
+    struct iovec iov[1];
+    union {
+        struct cmsghdr cmsg;
+        char control[CMSG_SPACE(sizeof(int))];
+    } msg_control;
+
+    iov[0].iov_base = buf;
+    iov[0].iov_len = size;
+
+    msg.msg_iov = iov;
+    msg.msg_iovlen = 1;
+    msg.msg_control = &msg_control;
+    msg.msg_controllen = sizeof(msg_control);
 
     for (;;) {
-        len = qemu_recv(s->sockfd, buf, size, 0);
-        if (len != -1) {
-            break;
+        if (!s->pipefd_passed) {
+            /*
+             * recvmsg is called here to catch the control message for
+             * the exchange of PIPE file descriptor until it is received.
+             */
+            len = recvmsg(s->sockfd, &msg, 0);
+            if (len != -1) {
+                if (unix_msgfd_lookup(s, &msg) > 0) {
+                    s->pipefd_passed = 1;
+                }
+                break;
+            }
+        } else {
+            len = qemu_recv(s->sockfd, buf, size, 0);
+            if (len != -1) {
+                break;
+            }
         }
 
         if (socket_error() == EAGAIN) {
-- 
1.7.7.6

[Qemu-devel] [PATCH 11/17] add argument ram_addr_t to hook_ram_load

[Lei Li]

Adds argument ram_addr_t to hook_ram_load, and replaces
QEMURamHookFunc with QEMURamLoadHookFunc for it. With this
new argument, it will allow cut almost half of the data
transferred on the Unix socket using by page flipping
migraton.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 arch_init.c                   |    2 +-
 include/migration/migration.h |    2 +-
 include/migration/qemu-file.h |   11 ++++++++++-
 migration-rdma.c              |    2 +-
 savevm.c                      |    4 ++--
 5 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index daaa519..0621893 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -945,7 +945,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
                 goto done;
             }
         } else if (flags & RAM_SAVE_FLAG_HOOK) {
-            ram_control_load_hook(f, flags);
+            ram_control_load_hook(f, addr, flags);
         }
         error = qemu_file_get_error(f);
         if (error) {
diff --git a/include/migration/migration.h b/include/migration/migration.h
index ca852a8..300e52c 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -149,7 +149,7 @@ int64_t xbzrle_cache_resize(int64_t new_size);
 
 void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
-void ram_control_load_hook(QEMUFile *f, uint64_t flags);
+void ram_control_load_hook(QEMUFile *f, ram_addr_t addr, uint64_t flags);
 
 /* Whenever this is found in the data stream, the flags
  * will be passed to ram_control_load_hook in the incoming-migration
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index 6646e89..176c2d9 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -65,6 +65,15 @@ typedef ssize_t (QEMUFileWritevBufferFunc)(void *opaque, struct iovec *iov,
 typedef int (QEMURamHookFunc)(QEMUFile *f, void *opaque, uint64_t flags);
 
 /*
+ * This function provides load hook for RAM migration, allows
+ * override of where the RAM page is loaded (such as page
+ * flipping for example).
+ */
+typedef int (QEMURamLoadHookFunc)(QEMUFile *f, void *opaque,
+                                  ram_addr_t addr,
+                                  uint64_t flags);
+
+/*
  * Constants used by ram_control_* hooks
  */
 #define RAM_CONTROL_SETUP    0
@@ -90,7 +99,7 @@ typedef struct QEMUFileOps {
     QEMUFileWritevBufferFunc *writev_buffer;
     QEMURamHookFunc *before_ram_iterate;
     QEMURamHookFunc *after_ram_iterate;
-    QEMURamHookFunc *hook_ram_load;
+    QEMURamLoadHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
 } QEMUFileOps;
 
diff --git a/migration-rdma.c b/migration-rdma.c
index ae04de4..732ec1a 100644
--- a/migration-rdma.c
+++ b/migration-rdma.c
@@ -2938,7 +2938,7 @@ err_rdma_dest_wait:
  * Keep doing this until the source tells us to stop.
  */
 static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque,
-                                         uint64_t flags)
+                                         ram_addr_t offset, uint64_t flags)
 {
     RDMAControlHeader reg_resp = { .len = sizeof(RDMARegisterResult),
                                .type = RDMA_CONTROL_REGISTER_RESULT,
diff --git a/savevm.c b/savevm.c
index 137e74f..75e397c 100644
--- a/savevm.c
+++ b/savevm.c
@@ -647,12 +647,12 @@ void ram_control_after_iterate(QEMUFile *f, uint64_t flags)
     }
 }
 
-void ram_control_load_hook(QEMUFile *f, uint64_t flags)
+void ram_control_load_hook(QEMUFile *f, ram_addr_t offset, uint64_t flags)
 {
     int ret = -EINVAL;
 
     if (f->ops->hook_ram_load) {
-        ret = f->ops->hook_ram_load(f, f->opaque, flags);
+        ret = f->ops->hook_ram_load(f, f->opaque, offset, flags);
         if (ret < 0) {
             qemu_file_set_error(f, ret);
         }
-- 
1.7.7.6

[Qemu-devel] [PATCH 12/17] migration-local: override hook_ram_load

[Lei Li]

Override hook_ram_load to receive the pipe file descriptor
passed by source process and page address which will be
extracted to vmsplice the page data from pipe.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-local.c |   59 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 59 insertions(+), 0 deletions(-)

diff --git a/migration-local.c b/migration-local.c
index 5f98a01..ce4c070 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -231,10 +231,69 @@ static size_t qemu_local_save_ram(QEMUFile *f, void *opaque,
     return RAM_SAVE_CONTROL_NOT_SUPP;
 }
 
+static int qemu_local_ram_load(QEMUFile *f, void *opaque,
+                               ram_addr_t addr, uint64_t flags)
+{
+    QEMUFileLocal *s = opaque;
+    struct iovec iov;
+    ssize_t ret = -EINVAL;
+
+    if (!s->pipefd_received) {
+        /*
+         * send_pipefd was called at this point, and it wrote one
+         * byte to the stream.
+         */
+        qemu_get_byte(s->file);
+        s->pipefd_received = true;
+    }
+
+    if (s->pipefd_passed) {
+        void *host;
+        /*
+         * Extract the page address from the 8-byte record and
+         * read the page data from the pipe.
+         */
+        host = qemu_get_ram_ptr(addr);
+
+        iov.iov_base = host;
+        iov.iov_len = TARGET_PAGE_SIZE;
+
+        /*
+         * The flag SPLICE_F_MOVE is introduced in kernel for the page
+         * flipping feature in QEMU, which will move pages rather than
+         * copying, previously unused.
+         *
+         * If a move is not possible the kernel will transparently fall
+         * back to copying data.
+         *
+         * For older kernels the SPLICE_F_MOVE would be ignored and a copy
+         * would occur.
+         */
+
+        ret = vmsplice(s->pipefd[0], &iov, 1, SPLICE_F_MOVE);
+        if (ret == -1) {
+            if (errno != EAGAIN && errno != EINTR) {
+                fprintf(stderr, "vmsplice() load error: %s", strerror(errno));
+                return ret;
+            }
+            DPRINTF("vmsplice load error\n");
+        } else if (ret == 0) {
+            DPRINTF(stderr, "load_page: zero read\n");
+        }
+
+        DPRINTF("vmsplice (read): %zu\n", ret);
+        return ret;
+    }
+
+    return -EINVAL;
+}
+
+
 static const QEMUFileOps pipe_read_ops = {
     .get_fd        = qemu_local_get_sockfd,
     .get_buffer    = qemu_local_get_buffer,
     .close         = qemu_local_close,
+    .hook_ram_load = qemu_local_ram_load
 };
 
 static const QEMUFileOps pipe_write_ops = {
-- 
1.7.7.6

[Qemu-devel] [PATCH 13/17] migration-unix: replace qemu_fopen_socket with qemu_fopen_socket_local

[Lei Li]

Replace qemu_fopen_socket with qemu_fopen_socket_local in Unix
protocol migration.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-unix.c |   18 ++++++++++++++----
 1 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/migration-unix.c b/migration-unix.c
index 651fc5b..9beeafe 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -37,12 +37,22 @@ static void unix_wait_for_connect(int fd, void *opaque)
     if (fd < 0) {
         DPRINTF("migrate connect error\n");
         s->file = NULL;
-        migrate_fd_error(s);
+        goto fail;
     } else {
         DPRINTF("migrate connect success\n");
-        s->file = qemu_fopen_socket(fd, "wb");
+
+        s->file = qemu_fopen_socket_local(fd, "wb");
+        if (s->file == NULL) {
+            fprintf(stderr, "failed to open Unix socket\n");
+            goto fail;
+        }
+
         migrate_fd_connect(s);
+        return;
     }
+
+fail:
+    migrate_fd_error(s);
 }
 
 void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp)
@@ -71,9 +81,9 @@ static void unix_accept_incoming_migration(void *opaque)
         goto out;
     }
 
-    f = qemu_fopen_socket(c, "rb");
+    f = qemu_fopen_socket_local(c, "rb");
     if (f == NULL) {
-        fprintf(stderr, "could not qemu_fopen socket\n");
+        fprintf(stderr, "failed to open Unix socket\n");
         goto out;
     }
 
-- 
1.7.7.6

[Qemu-devel] [PATCH 14/17] add new RunState RUN_STATE_MEMORY_STALE

[Lei Li]

Introduce new RunState RUN_STATE_MEMORY_STALE and
add it to runstate_needs_reset().

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 qapi-schema.json |    7 +++++--
 vl.c             |   13 ++++++++++++-
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index ea910ef..6ff46ff 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -176,12 +176,15 @@
 # @watchdog: the watchdog action is configured to pause and has been triggered
 #
 # @guest-panicked: guest has been panicked as a result of guest OS panic
+#
+# @memory-stale: guest is paused to start unix_page_flipping migration
+# process, the destination QEMU will has the newer contents of the memory
 ##
 { 'enum': 'RunState',
   'data': [ 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
             'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
-            'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
-            'guest-panicked' ] }
+            'running', 'save-vm', 'shutdown', 'suspended', 'memory-stale',
+            'watchdog', 'guest-panicked' ] }
 
 ##
 # @SnapshotInfo
diff --git a/vl.c b/vl.c
index 8d5d874..3ea96b2 100644
--- a/vl.c
+++ b/vl.c
@@ -601,6 +601,7 @@ static const RunStateTransition runstate_transitions_def[] = {
 
     { RUN_STATE_PAUSED, RUN_STATE_RUNNING },
     { RUN_STATE_PAUSED, RUN_STATE_FINISH_MIGRATE },
+    { RUN_STATE_PAUSED, RUN_STATE_MEMORY_STALE },
 
     { RUN_STATE_POSTMIGRATE, RUN_STATE_RUNNING },
     { RUN_STATE_POSTMIGRATE, RUN_STATE_FINISH_MIGRATE },
@@ -608,6 +609,7 @@ static const RunStateTransition runstate_transitions_def[] = {
     { RUN_STATE_PRELAUNCH, RUN_STATE_RUNNING },
     { RUN_STATE_PRELAUNCH, RUN_STATE_FINISH_MIGRATE },
     { RUN_STATE_PRELAUNCH, RUN_STATE_INMIGRATE },
+    { RUN_STATE_PRELAUNCH, RUN_STATE_MEMORY_STALE },
 
     { RUN_STATE_FINISH_MIGRATE, RUN_STATE_RUNNING },
     { RUN_STATE_FINISH_MIGRATE, RUN_STATE_POSTMIGRATE },
@@ -624,23 +626,31 @@ static const RunStateTransition runstate_transitions_def[] = {
     { RUN_STATE_RUNNING, RUN_STATE_SHUTDOWN },
     { RUN_STATE_RUNNING, RUN_STATE_WATCHDOG },
     { RUN_STATE_RUNNING, RUN_STATE_GUEST_PANICKED },
+    { RUN_STATE_RUNNING, RUN_STATE_MEMORY_STALE },
 
     { RUN_STATE_SAVE_VM, RUN_STATE_RUNNING },
 
     { RUN_STATE_SHUTDOWN, RUN_STATE_PAUSED },
     { RUN_STATE_SHUTDOWN, RUN_STATE_FINISH_MIGRATE },
+    { RUN_STATE_SHUTDOWN, RUN_STATE_MEMORY_STALE },
 
     { RUN_STATE_DEBUG, RUN_STATE_SUSPENDED },
+    { RUN_STATE_DEBUG, RUN_STATE_MEMORY_STALE },
     { RUN_STATE_RUNNING, RUN_STATE_SUSPENDED },
     { RUN_STATE_SUSPENDED, RUN_STATE_RUNNING },
     { RUN_STATE_SUSPENDED, RUN_STATE_FINISH_MIGRATE },
+    { RUN_STATE_SUSPENDED, RUN_STATE_MEMORY_STALE },
 
     { RUN_STATE_WATCHDOG, RUN_STATE_RUNNING },
     { RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE },
+    { RUN_STATE_WATCHDOG, RUN_STATE_MEMORY_STALE },
 
     { RUN_STATE_GUEST_PANICKED, RUN_STATE_RUNNING },
     { RUN_STATE_GUEST_PANICKED, RUN_STATE_FINISH_MIGRATE },
+    { RUN_STATE_GUEST_PANICKED, RUN_STATE_MEMORY_STALE },
 
+    { RUN_STATE_MEMORY_STALE, RUN_STATE_RUNNING },
+    { RUN_STATE_MEMORY_STALE, RUN_STATE_POSTMIGRATE },
     { RUN_STATE_MAX, RUN_STATE_MAX },
 };
 
@@ -685,7 +695,8 @@ int runstate_is_running(void)
 bool runstate_needs_reset(void)
 {
     return runstate_check(RUN_STATE_INTERNAL_ERROR) ||
-        runstate_check(RUN_STATE_SHUTDOWN);
+        runstate_check(RUN_STATE_SHUTDOWN) ||
+        runstate_check(RUN_STATE_MEMORY_STALE);
 }
 
 StatusInfo *qmp_query_status(Error **errp)
-- 
1.7.7.6

[Qemu-devel] [PATCH 15/17] migration-unix: page flipping support on unix outgoing

[Lei Li]

Add page flipping support on unix outgoing part by stopping
VM with the new RunState RUN_STATE_MEMORY_STALE before
invoking migration if unix_page_flipping enabled.

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration-unix.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/migration-unix.c b/migration-unix.c
index 9beeafe..cbf2087 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -19,6 +19,7 @@
 #include "migration/migration.h"
 #include "migration/qemu-file.h"
 #include "block/block.h"
+#include "sysemu/sysemu.h"
 
 //#define DEBUG_MIGRATION_UNIX
 
@@ -33,6 +34,7 @@
 static void unix_wait_for_connect(int fd, void *opaque)
 {
     MigrationState *s = opaque;
+    int ret;
 
     if (fd < 0) {
         DPRINTF("migrate connect error\n");
@@ -47,6 +49,15 @@ static void unix_wait_for_connect(int fd, void *opaque)
             goto fail;
         }
 
+        /* Stop VM before invoking migration if unix_page_flipping enabled */
+        if (migrate_unix_page_flipping()) {
+            ret = vm_stop_force_state(RUN_STATE_MEMORY_STALE);
+            if (ret < 0) {
+                DPRINTF("failed to stop VM\n");
+                goto fail;
+            }
+        }
+
         migrate_fd_connect(s);
         return;
     }
-- 
1.7.7.6

[Qemu-devel] [PATCH 16/17] migration: adjust migration_thread() process for page flipping

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 migration.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/migration.c b/migration.c
index e012cd4..7e0ec33 100644
--- a/migration.c
+++ b/migration.c
@@ -582,7 +582,7 @@ static void *migration_thread(void *opaque)
             if (pending_size && pending_size >= max_size) {
                 qemu_savevm_state_iterate(s->file);
             } else {
-                int ret;
+                int ret = 0;
 
                 DPRINTF("done iterating\n");
                 qemu_mutex_lock_iothread();
@@ -590,7 +590,10 @@ static void *migration_thread(void *opaque)
                 qemu_system_wakeup_request(QEMU_WAKEUP_REASON_OTHER);
                 old_vm_running = runstate_is_running();
 
-                ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+                if (!runstate_needs_reset()) {
+                    ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
+                }
+
                 if (ret >= 0) {
                     qemu_file_set_rate_limit(s->file, INT_MAX);
                     qemu_savevm_state_complete(s->file);
-- 
1.7.7.6

[Qemu-devel] [PATCH 17/17] hmp: better format for info migrate_capabilities

[Lei Li]

As there might be more capabilities introduced, better to display
it in lines.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 hmp.c |    5 ++---
 1 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/hmp.c b/hmp.c
index 32ee285..dcfa2f9 100644
--- a/hmp.c
+++ b/hmp.c
@@ -226,13 +226,12 @@ void hmp_info_migrate_capabilities(Monitor *mon, const QDict *qdict)
     caps = qmp_query_migrate_capabilities(NULL);
 
     if (caps) {
-        monitor_printf(mon, "capabilities: ");
+        monitor_printf(mon, "Capabilities:\n");
         for (cap = caps; cap; cap = cap->next) {
-            monitor_printf(mon, "%s: %s ",
+            monitor_printf(mon, "%s: %s\n",
                            MigrationCapability_lookup[cap->value->capability],
                            cap->value->state ? "on" : "off");
         }
-        monitor_printf(mon, "\n");
     }
 
     qapi_free_MigrationCapabilityStatusList(caps);
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 06/17] migration-local: add send_pipefd()

[Daniel P. Berrange]

On Mon, Dec 02, 2013 at 05:19:06PM +0800, Lei Li wrote:
> This patch adds send_pipefd() to pass the pipe file descriptor
> to destination process.
> 
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>  migration-local.c |   46 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 files changed, 46 insertions(+), 0 deletions(-)
> 
> diff --git a/migration-local.c b/migration-local.c
> index 929ed60..f479530 100644
> --- a/migration-local.c
> +++ b/migration-local.c
> @@ -167,3 +167,49 @@ fail:
>      g_free(s);
>      return NULL;
>  }
> +
> +
> +/*
> + * Pass a pipe file descriptor to another process.
> + *
> + * Return negative value If pipefd < 0. Return 0 on
> + * success.
> + *
> + */
> +static int send_pipefd(int sockfd, int pipefd)
> +{
> +    struct msghdr msg;
> +    struct iovec iov[1];
> +    ssize_t ret;
> +    char req[1] = { 0x01 };
> +
> +    union {
> +      struct cmsghdr cm;
> +      char control[CMSG_SPACE(sizeof(int))];
> +    } control_un;
> +    struct cmsghdr *cmptr;
> +
> +    msg.msg_control = control_un.control;
> +    msg.msg_controllen = sizeof(control_un.control);
> +
> +    cmptr = CMSG_FIRSTHDR(&msg);
> +    cmptr->cmsg_len = CMSG_LEN(sizeof(int));
> +    cmptr->cmsg_level = SOL_SOCKET;
> +    cmptr->cmsg_type = SCM_RIGHTS;
> +    *((int *) CMSG_DATA(cmptr)) = pipefd;
> +
> +    msg.msg_name = NULL;
> +    msg.msg_namelen = 0;
> +
> +    iov[0].iov_base = req;
> +    iov[0].iov_len = sizeof(req);
> +    msg.msg_iov = iov;
> +    msg.msg_iovlen = 1;
> +
> +    ret = sendmsg(sockfd, &msg, 0);
> +    if (ret <= 0) {
> +        DPRINTF("sendmsg error: %s\n", strerror(errno));
> +    }
> +
> +    return ret;
> +}

Just a reminder about my comments from previous posting. This is
introducing a 3rd private function for sending FDs. The existing
code should be refactored into qemu-socket.{c,h} and shared.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 06/17] migration-local: add send_pipefd()

[Lei Li]

On 12/02/2013 05:33 PM, Daniel P. Berrange wrote:
> On Mon, Dec 02, 2013 at 05:19:06PM +0800, Lei Li wrote:
>> This patch adds send_pipefd() to pass the pipe file descriptor
>> to destination process.
>>
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   migration-local.c |   46 ++++++++++++++++++++++++++++++++++++++++++++++
>>   1 files changed, 46 insertions(+), 0 deletions(-)
>>
>> diff --git a/migration-local.c b/migration-local.c
>> index 929ed60..f479530 100644
>> --- a/migration-local.c
>> +++ b/migration-local.c
>> @@ -167,3 +167,49 @@ fail:
>>       g_free(s);
>>       return NULL;
>>   }
>> +
>> +
>> +/*
>> + * Pass a pipe file descriptor to another process.
>> + *
>> + * Return negative value If pipefd < 0. Return 0 on
>> + * success.
>> + *
>> + */
>> +static int send_pipefd(int sockfd, int pipefd)
>> +{
>> +    struct msghdr msg;
>> +    struct iovec iov[1];
>> +    ssize_t ret;
>> +    char req[1] = { 0x01 };
>> +
>> +    union {
>> +      struct cmsghdr cm;
>> +      char control[CMSG_SPACE(sizeof(int))];
>> +    } control_un;
>> +    struct cmsghdr *cmptr;
>> +
>> +    msg.msg_control = control_un.control;
>> +    msg.msg_controllen = sizeof(control_un.control);
>> +
>> +    cmptr = CMSG_FIRSTHDR(&msg);
>> +    cmptr->cmsg_len = CMSG_LEN(sizeof(int));
>> +    cmptr->cmsg_level = SOL_SOCKET;
>> +    cmptr->cmsg_type = SCM_RIGHTS;
>> +    *((int *) CMSG_DATA(cmptr)) = pipefd;
>> +
>> +    msg.msg_name = NULL;
>> +    msg.msg_namelen = 0;
>> +
>> +    iov[0].iov_base = req;
>> +    iov[0].iov_len = sizeof(req);
>> +    msg.msg_iov = iov;
>> +    msg.msg_iovlen = 1;
>> +
>> +    ret = sendmsg(sockfd, &msg, 0);
>> +    if (ret <= 0) {
>> +        DPRINTF("sendmsg error: %s\n", strerror(errno));
>> +    }
>> +
>> +    return ret;
>> +}
> Just a reminder about my comments from previous posting. This is
> introducing a 3rd private function for sending FDs. The existing
> code should be refactored into qemu-socket.{c,h} and shared.

Hi Daniel,

Yes, I remembered your suggestion. As my reply in the previous version,
I'll make this refactoring in a separate thread. There are some differences
between these private functions (like data type and length of bytes
transmitted), may need a little time to get the common method settle down,
and would be better to do some test to make sure there is no impact on them.
And now this is a complete series as an experimental version, do you mind if
the refactoring would be posted after this series?

>
> Daniel


-- 
Lei

Re: [Qemu-devel] [PATCH 06/17] migration-local: add send_pipefd()

[Daniel P. Berrange]

On Tue, Dec 03, 2013 at 07:19:40PM +0800, Lei Li wrote:
> On 12/02/2013 05:33 PM, Daniel P. Berrange wrote:
> >On Mon, Dec 02, 2013 at 05:19:06PM +0800, Lei Li wrote:
> >>This patch adds send_pipefd() to pass the pipe file descriptor
> >>to destination process.
> >>
> >>Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> >>---
> >>  migration-local.c |   46 ++++++++++++++++++++++++++++++++++++++++++++++
> >>  1 files changed, 46 insertions(+), 0 deletions(-)
> >>
> >>diff --git a/migration-local.c b/migration-local.c
> >>index 929ed60..f479530 100644
> >>--- a/migration-local.c
> >>+++ b/migration-local.c
> >>@@ -167,3 +167,49 @@ fail:
> >>      g_free(s);
> >>      return NULL;
> >>  }
> >>+
> >>+
> >>+/*
> >>+ * Pass a pipe file descriptor to another process.
> >>+ *
> >>+ * Return negative value If pipefd < 0. Return 0 on
> >>+ * success.
> >>+ *
> >>+ */
> >>+static int send_pipefd(int sockfd, int pipefd)
> >>+{
> >>+    struct msghdr msg;
> >>+    struct iovec iov[1];
> >>+    ssize_t ret;
> >>+    char req[1] = { 0x01 };
> >>+
> >>+    union {
> >>+      struct cmsghdr cm;
> >>+      char control[CMSG_SPACE(sizeof(int))];
> >>+    } control_un;
> >>+    struct cmsghdr *cmptr;
> >>+
> >>+    msg.msg_control = control_un.control;
> >>+    msg.msg_controllen = sizeof(control_un.control);
> >>+
> >>+    cmptr = CMSG_FIRSTHDR(&msg);
> >>+    cmptr->cmsg_len = CMSG_LEN(sizeof(int));
> >>+    cmptr->cmsg_level = SOL_SOCKET;
> >>+    cmptr->cmsg_type = SCM_RIGHTS;
> >>+    *((int *) CMSG_DATA(cmptr)) = pipefd;
> >>+
> >>+    msg.msg_name = NULL;
> >>+    msg.msg_namelen = 0;
> >>+
> >>+    iov[0].iov_base = req;
> >>+    iov[0].iov_len = sizeof(req);
> >>+    msg.msg_iov = iov;
> >>+    msg.msg_iovlen = 1;
> >>+
> >>+    ret = sendmsg(sockfd, &msg, 0);
> >>+    if (ret <= 0) {
> >>+        DPRINTF("sendmsg error: %s\n", strerror(errno));
> >>+    }
> >>+
> >>+    return ret;
> >>+}
> >Just a reminder about my comments from previous posting. This is
> >introducing a 3rd private function for sending FDs. The existing
> >code should be refactored into qemu-socket.{c,h} and shared.
> 
> Hi Daniel,
> 
> Yes, I remembered your suggestion. As my reply in the previous version,
> I'll make this refactoring in a separate thread. There are some differences
> between these private functions (like data type and length of bytes
> transmitted), may need a little time to get the common method settle down,
> and would be better to do some test to make sure there is no impact on them.
> And now this is a complete series as an experimental version, do you mind if
> the refactoring would be posted after this series?

IMHO the refactoring should be a pre-requisite of this series. I've seen
too many times where future refactoring was promised but never arrived
because the motivation to fix it is gone once the main series is committed.
It is up to QEMU maintainers though - this is just my personal opinion.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 06/17] migration-local: add send_pipefd()

[Paolo Bonzini]

Il 03/12/2013 12:19, Lei Li ha scritto:
> On 12/02/2013 05:33 PM, Daniel P. Berrange wrote:
>> On Mon, Dec 02, 2013 at 05:19:06PM +0800, Lei Li wrote:
>>> This patch adds send_pipefd() to pass the pipe file descriptor
>>> to destination process.
>>>
>>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>>> ---
>>>   migration-local.c |   46
>>> ++++++++++++++++++++++++++++++++++++++++++++++
>>>   1 files changed, 46 insertions(+), 0 deletions(-)
>>>
>>> diff --git a/migration-local.c b/migration-local.c
>>> index 929ed60..f479530 100644
>>> --- a/migration-local.c
>>> +++ b/migration-local.c
>>> @@ -167,3 +167,49 @@ fail:
>>>       g_free(s);
>>>       return NULL;
>>>   }
>>> +
>>> +
>>> +/*
>>> + * Pass a pipe file descriptor to another process.
>>> + *
>>> + * Return negative value If pipefd < 0. Return 0 on
>>> + * success.
>>> + *
>>> + */
>>> +static int send_pipefd(int sockfd, int pipefd)
>>> +{
>>> +    struct msghdr msg;
>>> +    struct iovec iov[1];
>>> +    ssize_t ret;
>>> +    char req[1] = { 0x01 };
>>> +
>>> +    union {
>>> +      struct cmsghdr cm;
>>> +      char control[CMSG_SPACE(sizeof(int))];
>>> +    } control_un;
>>> +    struct cmsghdr *cmptr;
>>> +
>>> +    msg.msg_control = control_un.control;
>>> +    msg.msg_controllen = sizeof(control_un.control);
>>> +
>>> +    cmptr = CMSG_FIRSTHDR(&msg);
>>> +    cmptr->cmsg_len = CMSG_LEN(sizeof(int));
>>> +    cmptr->cmsg_level = SOL_SOCKET;
>>> +    cmptr->cmsg_type = SCM_RIGHTS;
>>> +    *((int *) CMSG_DATA(cmptr)) = pipefd;
>>> +
>>> +    msg.msg_name = NULL;
>>> +    msg.msg_namelen = 0;
>>> +
>>> +    iov[0].iov_base = req;
>>> +    iov[0].iov_len = sizeof(req);
>>> +    msg.msg_iov = iov;
>>> +    msg.msg_iovlen = 1;
>>> +
>>> +    ret = sendmsg(sockfd, &msg, 0);
>>> +    if (ret <= 0) {
>>> +        DPRINTF("sendmsg error: %s\n", strerror(errno));
>>> +    }
>>> +
>>> +    return ret;
>>> +}
>> Just a reminder about my comments from previous posting. This is
>> introducing a 3rd private function for sending FDs. The existing
>> code should be refactored into qemu-socket.{c,h} and shared.
> 
> Hi Daniel,
> 
> Yes, I remembered your suggestion. As my reply in the previous version,
> I'll make this refactoring in a separate thread. There are some differences
> between these private functions (like data type and length of bytes
> transmitted), may need a little time to get the common method settle down,
> and would be better to do some test to make sure there is no impact on
> them.

You would have to implement it in such a way that the buffer is
specified in the function, for example:

ssize_t qemu_send_with_fd(int sockfd, int passed_fd, const void *buf,
                          size_t len);
ssize_t qemu_recv_with_fd(int sockfd, int *passed_fd, void *buf,
                          size_t len);

The functions can go in util/ (I think not in qemu-socket.c, a new file
is preferrable).

I don't think it's particularly important, but it's definitely welcome.

Paolo

Re: [Qemu-devel] [PATCH 0/17 v5] Localhost migration with side channel for ram

[Paolo Bonzini]

Il 02/12/2013 10:19, Lei Li ha scritto:
> This patch series tries to introduce a mechanism using side
> channel pipe for RAM via SCM_RIGHTS with unix domain socket
> protocol migration.
> 
> This side channel is used for the page flipping by vmsplice,
> which is the internal mechanism for localhost migration that
> we are trying to add to QEMU. The backgroud info and previous
> patch series for reference,
> 
> Localhost migration
> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html
> 
> migration: Introduce side channel for RAM
> http://lists.gnu.org/archive/html/qemu-devel/2013-09/msg04043.html
> 
> I have picked patches from the localhost migration series and rebased 
> it on the series of side channel, now it is a complete series that
> passed the basic test.
> 
> Please let me know if there is anything needs to be fixed or improved.
> Your suggestions and comments are very welcome, and thanks to Paolo
> for his continued review and useful suggestions.
> 
> 
> Changes since V4:
>   Rename the capability to x-unix-page-flipping for now. (Paolo)
> 
> Changes since V3:
>   Address comments from Paolo including:
> 
>     - Get rid of useless check in send_pipefd() and the override
>       of before_ram_iterate, send pipefd in the first save_page
>       call, qemu_get_byte() in the first ram_load correspondingly.
>     - Add new argument ram_addr_t to hook_ram_load to cut half of
>       the data transferred on the socket.
>     - Add transition from 'debug' to 'memory-stale'.
>     - Other minor fixes.
> 
> Changes since V2:
>   Address comments from Paolo including:
> 
>     - Doc improvement for QAPI.
>     - Use callback get_buffer as the only one receiver.
>     - Rename the new RunState flipping-migrate to memory-stale, and
>       add transition from 'prelaunch' to 'memory-stale'.
>     - Other minor fixes.
> 
> Changes since V1:
>   Address suggestions from Paolo Bonzini including:
> 
>     - Use Unix socket QEMUFile as basis of code and adjust the way
>       of overriding RDMA hooks.
>     - Involve the vmsplice for page flipping.
>     - Add new RunState RUN_STATE_FLIPPING_MIGRATE and add it to
>       runstate_needs_reset() for the adjustment of the current
>       migration process with page flipping.
> 
> 
> 
> Lei Li (17):
>   QAPI: introduce magration capability unix_page_flipping
>   migration: add migrate_unix_page_flipping()
>   qmp-command.hx: add missing docs for migration capabilites
>   migration-local: add QEMUFileLocal with socket based QEMUFile
>   migration-local: introduce qemu_fopen_socket_local()
>   migration-local: add send_pipefd()
>   save_page: replace block_offset with a MemoryRegion
>   migration-local: override save_page for page transmit
>   savevm: adjust ram_control_save_page with page flipping
>   add unix_msgfd_lookup() to callback get_buffer
>   add argument ram_addr_t to hook_ram_load
>   migration-local: override hook_ram_load 
>   migration-unix: replace qemu_fopen_socket with qemu_fopen_socket_local
>   add new RanState RAN_STATE_MEMORY_STALE
>   migration-unix: page flipping support on unix outgoing
>   migration: adjust migration_thread() process for unix_page_flipping
>   hmp: better format for info migrate_capabilities
> 
>  Makefile.target               |   1 +
>  arch_init.c                   |   4 +-
>  migration-local.c             | 512 ++++++++++++++++++++++++++++++++++++++++++
>  hmp.c                         |   5 +-
>  include/migration/migration.h |   3 +
>  include/migration/qemu-file.h |   2 +
>  migration-unix.c              |  27 ++-
>  migration-rdma.c              |   4 +-
>  migration.c                   |  18 +-
>  qapi-schema.json              |  18 +-
>  qmp-commands.hx               |   8 +
>  savevm.c                      |  21 +-
>  vl.c                          |  12 +-
>  13 files changed, 617 insertions(+), 27 deletions(-)
>  create mode 100644 migration-local.c
> 
> 
> 

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Re: [Qemu-devel] [PATCH 06/17] migration-local: add send_pipefd()

[Lei Li]

On 12/03/2013 07:35 PM, Daniel P. Berrange wrote:
> On Tue, Dec 03, 2013 at 07:19:40PM +0800, Lei Li wrote:
>> On 12/02/2013 05:33 PM, Daniel P. Berrange wrote:
>>> On Mon, Dec 02, 2013 at 05:19:06PM +0800, Lei Li wrote:
>>>> This patch adds send_pipefd() to pass the pipe file descriptor
>>>> to destination process.
>>>>
>>>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>>>> ---
>>>>   migration-local.c |   46 ++++++++++++++++++++++++++++++++++++++++++++++
>>>>   1 files changed, 46 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/migration-local.c b/migration-local.c
>>>> index 929ed60..f479530 100644
>>>> --- a/migration-local.c
>>>> +++ b/migration-local.c
>>>> @@ -167,3 +167,49 @@ fail:
>>>>       g_free(s);
>>>>       return NULL;
>>>>   }
>>>> +
>>>> +
>>>> +/*
>>>> + * Pass a pipe file descriptor to another process.
>>>> + *
>>>> + * Return negative value If pipefd < 0. Return 0 on
>>>> + * success.
>>>> + *
>>>> + */
>>>> +static int send_pipefd(int sockfd, int pipefd)
>>>> +{
>>>> +    struct msghdr msg;
>>>> +    struct iovec iov[1];
>>>> +    ssize_t ret;
>>>> +    char req[1] = { 0x01 };
>>>> +
>>>> +    union {
>>>> +      struct cmsghdr cm;
>>>> +      char control[CMSG_SPACE(sizeof(int))];
>>>> +    } control_un;
>>>> +    struct cmsghdr *cmptr;
>>>> +
>>>> +    msg.msg_control = control_un.control;
>>>> +    msg.msg_controllen = sizeof(control_un.control);
>>>> +
>>>> +    cmptr = CMSG_FIRSTHDR(&msg);
>>>> +    cmptr->cmsg_len = CMSG_LEN(sizeof(int));
>>>> +    cmptr->cmsg_level = SOL_SOCKET;
>>>> +    cmptr->cmsg_type = SCM_RIGHTS;
>>>> +    *((int *) CMSG_DATA(cmptr)) = pipefd;
>>>> +
>>>> +    msg.msg_name = NULL;
>>>> +    msg.msg_namelen = 0;
>>>> +
>>>> +    iov[0].iov_base = req;
>>>> +    iov[0].iov_len = sizeof(req);
>>>> +    msg.msg_iov = iov;
>>>> +    msg.msg_iovlen = 1;
>>>> +
>>>> +    ret = sendmsg(sockfd, &msg, 0);
>>>> +    if (ret <= 0) {
>>>> +        DPRINTF("sendmsg error: %s\n", strerror(errno));
>>>> +    }
>>>> +
>>>> +    return ret;
>>>> +}
>>> Just a reminder about my comments from previous posting. This is
>>> introducing a 3rd private function for sending FDs. The existing
>>> code should be refactored into qemu-socket.{c,h} and shared.
>> Hi Daniel,
>>
>> Yes, I remembered your suggestion. As my reply in the previous version,
>> I'll make this refactoring in a separate thread. There are some differences
>> between these private functions (like data type and length of bytes
>> transmitted), may need a little time to get the common method settle down,
>> and would be better to do some test to make sure there is no impact on them.
>> And now this is a complete series as an experimental version, do you mind if
>> the refactoring would be posted after this series?
> IMHO the refactoring should be a pre-requisite of this series. I've seen
> too many times where future refactoring was promised but never arrived
> because the motivation to fix it is gone once the main series is committed.
> It is up to QEMU maintainers though - this is just my personal opinion.

Just this is already a good shape and the refactoring may need
a little more time since some details might needs to be considered
and better to discuss in a separate thread.

I am happy to take any chance to contribute to community, as I can
learn a lot from you guys and it's really good experience that my
work could be useful to lots of people. And I believe this is not
my last patch for it. :)

>
> Regards,
> Daniel


-- 
Lei

Re: [Qemu-devel] [PATCH 06/17] migration-local: add send_pipefd()

[Lei Li]

On 12/03/2013 07:52 PM, Paolo Bonzini wrote:
> Il 03/12/2013 12:19, Lei Li ha scritto:
>> On 12/02/2013 05:33 PM, Daniel P. Berrange wrote:
>>> On Mon, Dec 02, 2013 at 05:19:06PM +0800, Lei Li wrote:
>>>> This patch adds send_pipefd() to pass the pipe file descriptor
>>>> to destination process.
>>>>
>>>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>>>> ---
>>>>    migration-local.c |   46
>>>> ++++++++++++++++++++++++++++++++++++++++++++++
>>>>    1 files changed, 46 insertions(+), 0 deletions(-)
>>>>
>>>> diff --git a/migration-local.c b/migration-local.c
>>>> index 929ed60..f479530 100644
>>>> --- a/migration-local.c
>>>> +++ b/migration-local.c
>>>> @@ -167,3 +167,49 @@ fail:
>>>>        g_free(s);
>>>>        return NULL;
>>>>    }
>>>> +
>>>> +
>>>> +/*
>>>> + * Pass a pipe file descriptor to another process.
>>>> + *
>>>> + * Return negative value If pipefd < 0. Return 0 on
>>>> + * success.
>>>> + *
>>>> + */
>>>> +static int send_pipefd(int sockfd, int pipefd)
>>>> +{
>>>> +    struct msghdr msg;
>>>> +    struct iovec iov[1];
>>>> +    ssize_t ret;
>>>> +    char req[1] = { 0x01 };
>>>> +
>>>> +    union {
>>>> +      struct cmsghdr cm;
>>>> +      char control[CMSG_SPACE(sizeof(int))];
>>>> +    } control_un;
>>>> +    struct cmsghdr *cmptr;
>>>> +
>>>> +    msg.msg_control = control_un.control;
>>>> +    msg.msg_controllen = sizeof(control_un.control);
>>>> +
>>>> +    cmptr = CMSG_FIRSTHDR(&msg);
>>>> +    cmptr->cmsg_len = CMSG_LEN(sizeof(int));
>>>> +    cmptr->cmsg_level = SOL_SOCKET;
>>>> +    cmptr->cmsg_type = SCM_RIGHTS;
>>>> +    *((int *) CMSG_DATA(cmptr)) = pipefd;
>>>> +
>>>> +    msg.msg_name = NULL;
>>>> +    msg.msg_namelen = 0;
>>>> +
>>>> +    iov[0].iov_base = req;
>>>> +    iov[0].iov_len = sizeof(req);
>>>> +    msg.msg_iov = iov;
>>>> +    msg.msg_iovlen = 1;
>>>> +
>>>> +    ret = sendmsg(sockfd, &msg, 0);
>>>> +    if (ret <= 0) {
>>>> +        DPRINTF("sendmsg error: %s\n", strerror(errno));
>>>> +    }
>>>> +
>>>> +    return ret;
>>>> +}
>>> Just a reminder about my comments from previous posting. This is
>>> introducing a 3rd private function for sending FDs. The existing
>>> code should be refactored into qemu-socket.{c,h} and shared.
>> Hi Daniel,
>>
>> Yes, I remembered your suggestion. As my reply in the previous version,
>> I'll make this refactoring in a separate thread. There are some differences
>> between these private functions (like data type and length of bytes
>> transmitted), may need a little time to get the common method settle down,
>> and would be better to do some test to make sure there is no impact on
>> them.
> You would have to implement it in such a way that the buffer is
> specified in the function, for example:
>
> ssize_t qemu_send_with_fd(int sockfd, int passed_fd, const void *buf,
>                            size_t len);
> ssize_t qemu_recv_with_fd(int sockfd, int *passed_fd, void *buf,
>                            size_t len);
>
> The functions can go in util/ (I think not in qemu-socket.c, a new file
> is preferrable).
>
> I don't think it's particularly important, but it's definitely welcome.

Hi Paolo,

Thanks for your specified suggestion! As it needs to test the related
code (tap/bridge & Proxy FS & flipping migration), I will work on it
after back from my vacation next week.  :-)

>
> Paolo
>


-- 
Lei

Re: [Qemu-devel] [PATCH 07/17] save_page: replace block_offset with a MemoryRegion

[Michael R. Hines]

On 12/02/2013 05:19 PM, Lei Li wrote:
> This patch exports MemoryRegion to save_page hook, replacing
> argument ram_addr_t block_offset with a MemoryRegion suggested
> by Paolo Bonzini.
>
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   arch_init.c                   |    4 ++--
>   include/migration/migration.h |    2 +-
>   include/migration/qemu-file.h |    8 ++++----
>   migration-rdma.c              |    4 ++--
>   savevm.c                      |    8 ++++----
>   5 files changed, 13 insertions(+), 13 deletions(-)
>
> diff --git a/arch_init.c b/arch_init.c
> index e0acbc5..daaa519 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -485,8 +485,8 @@ static int ram_save_block(QEMUFile *f, bool last_stage)
>
>               /* In doubt sent page as normal */
>               bytes_sent = -1;
> -            ret = ram_control_save_page(f, block->offset,
> -                               offset, TARGET_PAGE_SIZE, &bytes_sent);
> +            ret = ram_control_save_page(f, mr, offset, TARGET_PAGE_SIZE,
> +                                        &bytes_sent);
>
>               if (ret != RAM_SAVE_CONTROL_NOT_SUPP) {
>                   if (ret != RAM_SAVE_CONTROL_DELAYED) {
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index 7e5d01a..ca852a8 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -161,7 +161,7 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags);
>   #define RAM_SAVE_CONTROL_NOT_SUPP -1000
>   #define RAM_SAVE_CONTROL_DELAYED  -2000
>
> -size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
> +size_t ram_control_save_page(QEMUFile *f, MemoryRegion *mr,
>                                ram_addr_t offset, size_t size,
>                                int *bytes_sent);
>
> diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
> index f9b104a..6646e89 100644
> --- a/include/migration/qemu-file.h
> +++ b/include/migration/qemu-file.h
> @@ -77,10 +77,10 @@ typedef int (QEMURamHookFunc)(QEMUFile *f, void *opaque, uint64_t flags);
>    * is saved (such as RDMA, for example.)
>    */
>   typedef size_t (QEMURamSaveFunc)(QEMUFile *f, void *opaque,
> -                               ram_addr_t block_offset,
> -                               ram_addr_t offset,
> -                               size_t size,
> -                               int *bytes_sent);
> +                                 MemoryRegion *mr,
> +                                 ram_addr_t offset,
> +                                 size_t size,
> +                                 int *bytes_sent);
>
>   typedef struct QEMUFileOps {
>       QEMUFilePutBufferFunc *put_buffer;
> diff --git a/migration-rdma.c b/migration-rdma.c
> index f94f3b4..ae04de4 100644
> --- a/migration-rdma.c
> +++ b/migration-rdma.c
> @@ -2699,7 +2699,7 @@ static int qemu_rdma_close(void *opaque)
>    *                  the protocol because most transfers are sent asynchronously.
>    */
>   static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque,
> -                                  ram_addr_t block_offset, ram_addr_t offset,
> +                                  MemoryRegion *mr, ram_addr_t offset,
>                                     size_t size, int *bytes_sent)
>   {
>       QEMUFileRDMA *rfile = opaque;
> @@ -2716,7 +2716,7 @@ static size_t qemu_rdma_save_page(QEMUFile *f, void *opaque,
>            * is full, or the page doen't belong to the current chunk,
>            * an actual RDMA write will occur and a new chunk will be formed.
>            */
> -        ret = qemu_rdma_write(f, rdma, block_offset, offset, size);
> +        ret = qemu_rdma_write(f, rdma, mr->ram_addr, offset, size);
>           if (ret < 0) {
>               fprintf(stderr, "rdma migration: write error! %d\n", ret);
>               goto err;
> diff --git a/savevm.c b/savevm.c
> index 3f912dd..06c1f29 100644
> --- a/savevm.c
> +++ b/savevm.c
> @@ -661,12 +661,12 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags)
>       }
>   }
>
> -size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
> -                         ram_addr_t offset, size_t size, int *bytes_sent)
> +size_t ram_control_save_page(QEMUFile *f, MemoryRegion *mr, ram_addr_t offset,
> +                             size_t size, int *bytes_sent)
>   {
>       if (f->ops->save_page) {
> -        int ret = f->ops->save_page(f, f->opaque, block_offset,
> -                                    offset, size, bytes_sent);
> +        int ret = f->ops->save_page(f, f->opaque, mr, offset,
> +                                    size, bytes_sent);
>
>           if (ret != RAM_SAVE_CONTROL_DELAYED) {
>               if (bytes_sent && *bytes_sent > 0) {
Reviewed-by: Michael R. Hines <mrhines@us.ibm.com>

Re: [Qemu-devel] [PATCH 11/17] add argument ram_addr_t to hook_ram_load

[Michael R. Hines]

On 12/02/2013 05:19 PM, Lei Li wrote:
> Adds argument ram_addr_t to hook_ram_load, and replaces
> QEMURamHookFunc with QEMURamLoadHookFunc for it. With this
> new argument, it will allow cut almost half of the data
> transferred on the Unix socket using by page flipping
> migraton.
>
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   arch_init.c                   |    2 +-
>   include/migration/migration.h |    2 +-
>   include/migration/qemu-file.h |   11 ++++++++++-
>   migration-rdma.c              |    2 +-
>   savevm.c                      |    4 ++--
>   5 files changed, 15 insertions(+), 6 deletions(-)
>
> diff --git a/arch_init.c b/arch_init.c
> index daaa519..0621893 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -945,7 +945,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>                   goto done;
>               }
>           } else if (flags & RAM_SAVE_FLAG_HOOK) {
> -            ram_control_load_hook(f, flags);
> +            ram_control_load_hook(f, addr, flags);
>           }
>           error = qemu_file_get_error(f);
>           if (error) {
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index ca852a8..300e52c 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -149,7 +149,7 @@ int64_t xbzrle_cache_resize(int64_t new_size);
>
>   void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
>   void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
> -void ram_control_load_hook(QEMUFile *f, uint64_t flags);
> +void ram_control_load_hook(QEMUFile *f, ram_addr_t addr, uint64_t flags);
>
>   /* Whenever this is found in the data stream, the flags
>    * will be passed to ram_control_load_hook in the incoming-migration
> diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
> index 6646e89..176c2d9 100644
> --- a/include/migration/qemu-file.h
> +++ b/include/migration/qemu-file.h
> @@ -65,6 +65,15 @@ typedef ssize_t (QEMUFileWritevBufferFunc)(void *opaque, struct iovec *iov,
>   typedef int (QEMURamHookFunc)(QEMUFile *f, void *opaque, uint64_t flags);
>
>   /*
> + * This function provides load hook for RAM migration, allows
> + * override of where the RAM page is loaded (such as page
> + * flipping for example).
> + */
> +typedef int (QEMURamLoadHookFunc)(QEMUFile *f, void *opaque,
> +                                  ram_addr_t addr,
> +                                  uint64_t flags);
> +
> +/*
>    * Constants used by ram_control_* hooks
>    */
>   #define RAM_CONTROL_SETUP    0
> @@ -90,7 +99,7 @@ typedef struct QEMUFileOps {
>       QEMUFileWritevBufferFunc *writev_buffer;
>       QEMURamHookFunc *before_ram_iterate;
>       QEMURamHookFunc *after_ram_iterate;
> -    QEMURamHookFunc *hook_ram_load;
> +    QEMURamLoadHookFunc *hook_ram_load;
>       QEMURamSaveFunc *save_page;
>   } QEMUFileOps;
>
> diff --git a/migration-rdma.c b/migration-rdma.c
> index ae04de4..732ec1a 100644
> --- a/migration-rdma.c
> +++ b/migration-rdma.c
> @@ -2938,7 +2938,7 @@ err_rdma_dest_wait:
>    * Keep doing this until the source tells us to stop.
>    */
>   static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque,
> -                                         uint64_t flags)
> +                                         ram_addr_t offset, uint64_t flags)
>   {
>       RDMAControlHeader reg_resp = { .len = sizeof(RDMARegisterResult),
>                                  .type = RDMA_CONTROL_REGISTER_RESULT,
> diff --git a/savevm.c b/savevm.c
> index 137e74f..75e397c 100644
> --- a/savevm.c
> +++ b/savevm.c
> @@ -647,12 +647,12 @@ void ram_control_after_iterate(QEMUFile *f, uint64_t flags)
>       }
>   }
>
> -void ram_control_load_hook(QEMUFile *f, uint64_t flags)
> +void ram_control_load_hook(QEMUFile *f, ram_addr_t offset, uint64_t flags)
>   {
>       int ret = -EINVAL;
>
>       if (f->ops->hook_ram_load) {
> -        ret = f->ops->hook_ram_load(f, f->opaque, flags);
> +        ret = f->ops->hook_ram_load(f, f->opaque, offset, flags);
>           if (ret < 0) {
>               qemu_file_set_error(f, ret);
>           }
Reviewed-by: Michael R. Hines <mrhines@us.ibm.com>

Re: [Qemu-devel] [PATCH 01/17] QAPI: introduce migration capability x_unix_page_flipping

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]

On 12/02/2013 02:19 AM, Lei Li wrote:
> Introduce x_unix_page_flipping to MigrationCapability for
> localhost migration.
> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>  qapi-schema.json |   12 +++++++++++-
>  1 files changed, 11 insertions(+), 1 deletions(-)

I don't see this in git; is it something you are still interested in
adding?  If so, your subject line and commit message use '_'...

> 
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 83fa485..ea910ef 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -685,10 +685,20 @@
>  # @auto-converge: If enabled, QEMU will automatically throttle down the guest
>  #          to speed up convergence of RAM migration. (since 1.6)
>  #
> +# @x-unix-page-flipping: If enabled, QEMU can optimize migration when the

...while the code correctly uses '-'.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]