[Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration

[Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration

[Lei Li]

Hi,

This patch series tries to add localhost migration support to 
Qemu. 

When doing localhost migration, the host memory will balloon
up during the period, might consume double memories for some time.
So we want to add a new live migration mechanism localhost
migration.

Following I copied from last version that Anthony added for the 
benefit of the other reviewers:

The goal here is to allow "live upgrade" of a running QEMU instance.  
The work flow would look like this:

1) Guests are running QEMU release 1.6.1
2) Admin installs QEMU release 1.6.2 via RPM or deb
3) Admin does localhost migration with page flipping to use new version
   of QEMU.

Page flipping is used in order to avoid requiring that there is enough
free memory to fit an additional copy of the largest guest which is the
requirement today with localhost migration.

You can also read from the link below:
http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html

The plan is:

1) Add new command to do localhost migration.

   The qmp interface introduced like:

   { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }

2) Use different mechanism than current live migration.

   The very basic work flow like:

       qemu on the source (the source and destination are both on localhost)
              |
              V
           Stop VM
              |
              V   
          Create threads
              |
              V
       Page flipping through vmspice
              |
              V
       MADV_DONTNEED the ram pages which are already flipped
              |
              V
       Migration completes

   As stopping VM first, we expect/resume the page flipping through vmspice
   is fast enough to meet *live migration (low downtime).

Notes:
Currently the work flow is not exactly the same as description
above. For the first step, the work flow we implemented is:
stop VM and copy ram pages via unix domain socket, MADV_DONTNEED 
ram pages that already copied. After that, will replace to vmsplice
mechanism instead of copying pages.

Now it's still a draft version, and as it is implemented separately 
to the current migration code for a easy start, the next step will 
be trying to integrate it into the current migration implementation 
closely. To make sure we are on the right direction that should be 
headed, please let me know your suggestions on this.

For the interface, as Anthony has suggested using a flag or a 
capability, which one would you prefer or any ideas?

Your comments are very welcome!

TODO:
- Integrate to the current implement of migration closely?
- Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
- benchmark/evaluation.

Lei Li (12):
  migration: export MIG_STATE_xxx flags 
  savevm: export qemu_save_device_state()
  rename is_active to is_block_active
  arch_init: introduce ram_page_save()
  arch_init: introduce ram_save_local()
  arch_init: add save_local_setup to savevm_ram_handlers
  savevm: introduce qemu_savevm_local()
  savevm: adjust is_ram check in register_savevm_live()
  migration-local: implementation of outgoing part
  migration-local: implementation of incoming part
  migration-local: add option to command line for local incoming
  hmp:add hmp_localhost_migration interface

 Makefile.objs                 |    1 +
 arch_init.c                   |   110 +++++++++++++++++++
 block-migration.c             |    2 +-
 hmp-commands.hx               |   17 ++++
 hmp.c                         |   13 +++
 hmp.h                         |    1 +
 include/migration/migration.h |   32 +++++++
 include/sysemu/sysemu.h       |    1 +
 migration-local.c             |  228 +++++++++++++++++++++++++++++++++++++++++
 migration-unix.c              |   60 ++++++++++++
 migration.c                   |    8 --
 qapi-schema.json              |   14 +++
 qemu-options.hx               |    9 ++
 qmp-commands.hx               |   22 +++++
 savevm.c                      |   100 +++++++++++++++--
 vl.c                          |   14 +++
 16 files changed, 613 insertions(+), 19 deletions(-)
 create mode 100644 migration-local.c

[Qemu-devel] [PATCH 01/12] migration: export MIG_STATE_xxx flags

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/migration.h |    9 +++++++++
 migration.c                   |    8 --------
 2 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index e2acec6..a821c80 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -27,6 +27,15 @@ struct MigrationParams {
     bool shared;
 };
 
+/* Migration status */
+enum {
+    MIG_STATE_ERROR,      /* There has been an error. */
+    MIG_STATE_SETUP,      /* Setup stage for a migration. */
+    MIG_STATE_CANCELLED,  /* The migration has been cancelled. */
+    MIG_STATE_ACTIVE,     /* Whether the migration is active. */
+    MIG_STATE_COMPLETED,  /* The migration is successful completed. */
+};
+
 typedef struct MigrationState MigrationState;
 
 struct MigrationState
diff --git a/migration.c b/migration.c
index 058f9e6..0921ace 100644
--- a/migration.c
+++ b/migration.c
@@ -35,14 +35,6 @@
     do { } while (0)
 #endif
 
-enum {
-    MIG_STATE_ERROR,
-    MIG_STATE_SETUP,
-    MIG_STATE_CANCELLED,
-    MIG_STATE_ACTIVE,
-    MIG_STATE_COMPLETED,
-};
-
 #define MAX_THROTTLE  (32 << 20)      /* Migration speed throttling */
 
 /* Amount of time to allocate to each "chunk" of bandwidth-throttled
-- 
1.7.7.6

[Qemu-devel] [PATCH 02/12] savevm: export qemu_save_device_state()

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/sysemu/sysemu.h |    1 +
 savevm.c                |    7 ++++++-
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 2fb71af..5b90027 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -80,6 +80,7 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+int qemu_save_device_state(QEMUFile *f);
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
diff --git a/savevm.c b/savevm.c
index ff5ece6..1695102 100644
--- a/savevm.c
+++ b/savevm.c
@@ -2006,7 +2006,12 @@ static int qemu_savevm_state(QEMUFile *f)
     return ret;
 }
 
-static int qemu_save_device_state(QEMUFile *f)
+/**
+ * Save all of the device states to stream QEMUFile
+ *
+ * Return negtive if there has been an error
+ **/
+int qemu_save_device_state(QEMUFile *f)
 {
     SaveStateEntry *se;
 
-- 
1.7.7.6

[Qemu-devel] [PATCH 03/12] rename is_active to is_block_active

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 block-migration.c           |    2 +-
 include/migration/vmstate.h |    2 +-
 savevm.c                    |   16 ++++++++--------
 3 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/block-migration.c b/block-migration.c
index 2fd7699..171b8b8 100644
--- a/block-migration.c
+++ b/block-migration.c
@@ -814,7 +814,7 @@ SaveVMHandlers savevm_block_handlers = {
     .save_live_pending = block_save_pending,
     .load_state = block_load,
     .cancel = block_migration_cancel,
-    .is_active = block_is_active,
+    .is_block_active = block_is_active,
 };
 
 void blk_mig_init(void)
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index ebc4d09..acf847b 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -40,7 +40,7 @@ typedef struct SaveVMHandlers {
     int (*save_live_complete)(QEMUFile *f, void *opaque);
 
     /* This runs both outside and inside the iothread lock.  */
-    bool (*is_active)(void *opaque);
+    bool (*is_block_active)(void *opaque);
 
     /* This runs outside the iothread lock in the migration case, and
      * within the lock in the savevm case.  The callback had better only
diff --git a/savevm.c b/savevm.c
index 1695102..e18ca22 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1817,8 +1817,8 @@ void qemu_savevm_state_begin(QEMUFile *f,
         if (!se->ops || !se->ops->save_live_setup) {
             continue;
         }
-        if (se->ops && se->ops->is_active) {
-            if (!se->ops->is_active(se->opaque)) {
+        if (se->ops && se->ops->is_block_active) {
+            if (!se->ops->is_block_active(se->opaque)) {
                 continue;
             }
         }
@@ -1857,8 +1857,8 @@ int qemu_savevm_state_iterate(QEMUFile *f)
         if (!se->ops || !se->ops->save_live_iterate) {
             continue;
         }
-        if (se->ops && se->ops->is_active) {
-            if (!se->ops->is_active(se->opaque)) {
+        if (se->ops && se->ops->is_block_active) {
+            if (!se->ops->is_block_active(se->opaque)) {
                 continue;
             }
         }
@@ -1898,8 +1898,8 @@ void qemu_savevm_state_complete(QEMUFile *f)
         if (!se->ops || !se->ops->save_live_complete) {
             continue;
         }
-        if (se->ops && se->ops->is_active) {
-            if (!se->ops->is_active(se->opaque)) {
+        if (se->ops && se->ops->is_block_active) {
+            if (!se->ops->is_block_active(se->opaque)) {
                 continue;
             }
         }
@@ -1952,8 +1952,8 @@ uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size)
         if (!se->ops || !se->ops->save_live_pending) {
             continue;
         }
-        if (se->ops && se->ops->is_active) {
-            if (!se->ops->is_active(se->opaque)) {
+        if (se->ops && se->ops->is_block_active) {
+            if (!se->ops->is_block_active(se->opaque)) {
                 continue;
             }
         }
-- 
1.7.7.6

[Qemu-devel] [PATCH 04/12] arch_init: introduce ram_page_save()

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 arch_init.c |   58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 58 insertions(+), 0 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index a8b91ee..a418071 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -699,6 +699,64 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
     return remaining_size;
 }
 
+/* This is the last block from where we have sent data for local migration */
+static RAMBlock *last_block_local;
+static ram_addr_t last_offset_local;
+
+static int ram_page_save(QEMUFile *f)
+{
+    RAMBlock *block = last_block_local;
+    ram_addr_t offset = last_offset_local;
+    MemoryRegion *mr = block->mr;
+    int bytes_sent = 0;
+
+    if (!block) {
+        block = QTAILQ_FIRST(&ram_list.blocks);
+    }
+
+    do {
+        mr = block->mr;
+        uint8_t *p;
+        int cont = (block == last_block_local) ? RAM_SAVE_FLAG_CONTINUE : 0;
+
+        p = memory_region_get_ram_ptr(mr) + offset;
+
+        if (is_zero_page(p)) {
+            qemu_put_be64(f, offset | cont | RAM_SAVE_FLAG_COMPRESS);
+            if (!cont) {
+                qemu_put_byte(f, strlen(block->idstr));
+                qemu_put_buffer(f, (uint8_t *)block->idstr,
+                                strlen(block->idstr));
+            }
+            qemu_put_byte(f, *p);
+            bytes_sent = 1;
+        } else {
+            qemu_put_be64(f, offset | cont | RAM_SAVE_FLAG_PAGE);
+            if (!cont) {
+                qemu_put_byte(f, strlen(block->idstr));
+                qemu_put_buffer(f, (uint8_t *)block->idstr,
+                                strlen(block->idstr));
+            }
+            qemu_put_buffer(f, p, TARGET_PAGE_SIZE);
+            bytes_sent = TARGET_PAGE_SIZE;
+        }
+
+        offset += TARGET_PAGE_SIZE;
+        if (offset >= block->length) {
+            offset = 0;
+            block = QTAILQ_NEXT(block, next);
+            if (!block) {
+                block = QTAILQ_FIRST(&ram_list.blocks);
+            }
+        }
+    } while (block != last_block_local || offset != last_offset_local);
+
+    last_block_local = block;
+    last_offset_local = offset;
+
+    return bytes_sent;
+}
+
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
 {
     int ret, rc = 0;
-- 
1.7.7.6

[Qemu-devel] [PATCH 05/12] arch_init: introduce ram_save_local()

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 arch_init.c |   32 ++++++++++++++++++++++++++++++++
 1 files changed, 32 insertions(+), 0 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index a418071..7eeb52f 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -757,6 +757,38 @@ static int ram_page_save(QEMUFile *f)
     return bytes_sent;
 }
 
+static int ram_save_local(QEMUFile *f, void *opaque)
+{
+    uint64_t bytes_sent = 0;
+    uint64_t total_ram_bytes = ram_bytes_total();
+
+    qemu_mutex_lock_ramlist();
+
+    while (total_ram_bytes) {
+       void *ram;
+       MemoryRegion *mr;
+
+       bytes_sent = ram_page_save(f);
+       /* No more ram pages. */
+       if (bytes_sent == 0) {
+           return bytes_sent;
+       }
+
+       mr = last_block_local->mr;
+       ram = memory_region_get_ram_ptr(mr) + last_offset_local;
+
+       /* DONTNEED the ram page that has already copied. */
+       qemu_madvise(ram, bytes_sent, QEMU_MADV_DONTNEED);
+       total_ram_bytes -= bytes_sent;
+    }
+
+    qemu_mutex_unlock_ramlist();
+
+    qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
+
+    return bytes_sent;
+}
+
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
 {
     int ret, rc = 0;
-- 
1.7.7.6

[Qemu-devel] [PATCH 06/12] arch_init: add save_local_setup to savevm_ram_handlers

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 arch_init.c                 |    1 +
 include/migration/vmstate.h |    2 ++
 2 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 7eeb52f..5c25005 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -973,6 +973,7 @@ SaveVMHandlers savevm_ram_handlers = {
     .save_live_iterate = ram_save_iterate,
     .save_live_complete = ram_save_complete,
     .save_live_pending = ram_save_pending,
+    .save_local_setup = ram_save_local,
     .load_state = ram_load,
     .cancel = ram_migration_cancel,
 };
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index acf847b..c534254 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -53,6 +53,8 @@ typedef struct SaveVMHandlers {
     int (*save_live_setup)(QEMUFile *f, void *opaque);
     uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
 
+    int (*save_local_setup)(QEMUFile *f, void *opaque);
+
     LoadStateHandler *load_state;
 } SaveVMHandlers;
 
-- 
1.7.7.6

[Qemu-devel] [PATCH 07/12] savevm: introduce qemu_savevm_local()

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/sysemu/sysemu.h |    1 +
 savevm.c                |   55 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+), 0 deletions(-)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 5b90027..9abe4c9 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -80,6 +80,7 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+int qemu_savevm_local(QEMUFile *f);
 int qemu_save_device_state(QEMUFile *f);
 int qemu_loadvm_state(QEMUFile *f);
 
diff --git a/savevm.c b/savevm.c
index e18ca22..c183369 100644
--- a/savevm.c
+++ b/savevm.c
@@ -2007,6 +2007,61 @@ static int qemu_savevm_state(QEMUFile *f)
 }
 
 /**
+ * Save all of the ram pages to stream QEMUFile
+ *
+ * Return: negtive for an error
+ */
+int qemu_savevm_local(QEMUFile *f)
+{
+    SaveStateEntry *se;
+    int ret;
+
+    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
+        if (!se->ops) {
+            continue;
+        }
+    }
+
+    qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
+    qemu_put_be32(f, QEMU_VM_FILE_VERSION);
+
+    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
+        int len;
+
+        if (!se->ops || !se->ops->save_local_setup) {
+            continue;
+        }
+        if (se->ops && se->ops->is_block_active) {
+            continue;
+        }
+
+        /* Section type */
+        qemu_put_byte(f, QEMU_VM_SECTION_START);
+        qemu_put_be32(f, se->section_id);
+
+        /* ID string */
+        len = strlen(se->idstr);
+        qemu_put_byte(f, len);
+        qemu_put_buffer(f, (uint8_t *)se->idstr, len);
+
+        qemu_put_be32(f, se->instance_id);
+
+        qemu_put_be32(f, se->version_id);
+
+        ret = se->ops->save_local_setup(f, se->opaque);
+        if (ret < 0) {
+            qemu_file_set_error(f, ret);
+            break;
+        }
+    }
+
+    qemu_put_byte(f, QEMU_VM_EOF);
+    qemu_fflush(f);
+
+    return qemu_file_get_error(f);
+}
+
+/**
  * Save all of the device states to stream QEMUFile
  *
  * Return negtive if there has been an error
-- 
1.7.7.6

[Qemu-devel] [PATCH 08/12] savevm: adjust is_ram check in register_savevm_live()

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 savevm.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/savevm.c b/savevm.c
index c183369..e826f64 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1484,7 +1484,8 @@ int register_savevm_live(DeviceState *dev,
     se->vmsd = NULL;
     se->no_migrate = 0;
     /* if this is a live_savem then set is_ram */
-    if (ops->save_live_setup != NULL) {
+    if ((ops->save_live_setup != NULL)
+         || (ops->save_local_setup != NULL)) {
         se->is_ram = 1;
     }
 
-- 
1.7.7.6

[Qemu-devel] [PATCH 09/12] migration-local: implementation of outgoing part

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 Makefile.objs                 |    1 +
 include/migration/migration.h |   15 ++++
 migration-local.c             |  158 +++++++++++++++++++++++++++++++++++++++++
 migration-unix.c              |   13 ++++
 qapi-schema.json              |   14 ++++
 qmp-commands.hx               |   22 ++++++
 6 files changed, 223 insertions(+), 0 deletions(-)
 create mode 100644 migration-local.c

diff --git a/Makefile.objs b/Makefile.objs
index 5b288ba..2a3d9a5 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -53,6 +53,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/
 common-obj-y += migration.o migration-tcp.o
 common-obj-y += qemu-char.o #aio.o
 common-obj-y += block-migration.o
+common-obj-y += migration-local.o
 common-obj-y += page_cache.o xbzrle.o
 
 common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
diff --git a/include/migration/migration.h b/include/migration/migration.h
index a821c80..a690e18 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -58,6 +58,17 @@ struct MigrationState
     int64_t xbzrle_cache_size;
 };
 
+
+typedef struct LocalMigState LocalMigState;
+
+struct LocalMigState
+{
+    int fd;
+    int state;
+    QEMUFile *file;
+    QemuThread thread;
+};
+
 void process_incoming_migration(QEMUFile *f);
 
 void qemu_start_incoming_migration(const char *uri, Error **errp);
@@ -80,6 +91,8 @@ void unix_start_incoming_migration(const char *path, Error **errp);
 
 void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
 
+void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp);
+
 void fd_start_incoming_migration(const char *path, Error **errp);
 
 void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp);
@@ -88,6 +101,8 @@ void migrate_fd_error(MigrationState *s);
 
 void migrate_fd_connect(MigrationState *s);
 
+void local_migration_fd_connect(LocalMigState *s);
+
 int migrate_fd_close(MigrationState *s);
 
 void add_migration_state_change_notifier(Notifier *notify);
diff --git a/migration-local.c b/migration-local.c
new file mode 100644
index 0000000..5bd1ed0
--- /dev/null
+++ b/migration-local.c
@@ -0,0 +1,158 @@
+/*
+ * QEMU localhost migration
+ *
+ * Copyright IBM, Corp. 2013
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions are licensed under the terms of the GNU GPL, version 2
+ * or (at your option) any later version.
+ */
+
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "monitor/monitor.h"
+#include "migration/qemu-file.h"
+#include "sysemu/sysemu.h"
+#include "block/block.h"
+#include "qemu/sockets.h"
+#include "migration/block.h"
+#include "qemu/thread.h"
+#include "qmp-commands.h"
+#include "exec/memory.h"
+#include "trace.h"
+#include "qemu/osdep.h"
+
+//#define DEBUG_MIGRATION_LOCAL
+
+#ifdef DEBUG_MIGRATION_LOCAL
+#define DPRINTF(fmt, ...) \
+    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+    do { } while (0)
+#endif
+
+
+/************************************************************************
+ * Outgoing part
+ */
+
+static LocalMigState *local_migration_init(void)
+{
+    LocalMigState *s = g_malloc0(sizeof(*s));
+
+    s->state = MIG_STATE_SETUP;
+    trace_migrate_set_state(MIG_STATE_SETUP);
+    s->fd = -1;
+
+    return s;
+}
+
+static void local_migration_error(LocalMigState *s)
+{
+    assert(s->file == NULL);
+
+    s->state = MIG_STATE_ERROR;
+    trace_migrate_set_state(MIG_STATE_ERROR);
+}
+
+static void local_outgoing_completed(LocalMigState *s)
+{
+    s->state = MIG_STATE_COMPLETED;
+    trace_migrate_set_state(MIG_STATE_COMPLETED);
+}
+
+static void *migration_local_thread(void *opaque)
+{
+    LocalMigState *s = opaque;
+    int ret;
+
+    DPRINTF("Beginning savevm\n");
+
+    while (s->state == MIG_STATE_ACTIVE) {
+        qemu_mutex_lock_iothread();
+        ret = qemu_savevm_local(s->file);
+        qemu_mutex_unlock_iothread();
+
+        /* No need to send device states if ram pages fails to to sent. */
+        if (ret < 0) {
+            local_migration_error(s);
+            break;
+        }
+
+        qemu_save_device_state(s->file);
+        qemu_fclose(s->file);
+    }
+
+    ret = qemu_file_get_error(s->file);
+    if (ret < 0) {
+        local_migration_error(s);
+    } else {
+        local_outgoing_completed(s);
+    }
+
+    qemu_mutex_lock_iothread();
+
+    if (s->state == MIG_STATE_COMPLETED) {
+        runstate_set(RUN_STATE_POSTMIGRATE);
+    }
+
+    qemu_mutex_unlock_iothread();
+
+    return NULL;
+}
+
+void local_migration_fd_connect(LocalMigState *s)
+{
+    s->state = MIG_STATE_ACTIVE;
+    trace_migrate_set_state(MIG_STATE_ACTIVE);
+
+    qemu_thread_create(&s->thread, migration_local_thread, s,
+                       QEMU_THREAD_JOINABLE);
+}
+
+void qmp_localhost_migrate(const char *uri, Error **errp)
+{
+    const char *path;
+    Error *local_err = NULL;
+    int is_vm_running;
+    LocalMigState *s;
+
+    if (qemu_savevm_state_blocked(errp)) {
+        return;
+    }
+
+    s = local_migration_init();
+    bdrv_flush_all();
+
+    is_vm_running = runstate_is_running();
+
+    /* Stop the VM first */
+    if (is_vm_running) {
+        vm_stop(RUN_STATE_SAVE_VM);
+    }
+
+    /* Start outgoing migration by unix socket. */
+    if (strstart(uri, "unix:", &path)) {
+        /* XXX. Creat a new unix_start_outgoing_migration_* is not necessary,
+         * just for the first step. This will be replaced by vmsplice
+         * mechanism. */
+        unix_start_local_outgoing_migration(s, path, &local_err);
+    } else {
+        error_set(errp, QERR_INVALID_PARAMETER_VALUE, "uri", "a valid migration protocol");
+        goto fail;
+    }
+
+    if (local_err) {
+        s->state = MIG_STATE_ERROR;
+        error_propagate(errp, local_err);
+        goto fail;
+    }
+
+fail:
+    if (!is_vm_running) {
+        vm_start();
+    }
+}
diff --git a/migration-unix.c b/migration-unix.c
index 94b7022..ec20c45 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -49,6 +49,19 @@ void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **
     unix_nonblocking_connect(path, unix_wait_for_connect, s, errp);
 }
 
+void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp)
+{
+    s->fd = unix_connect(path, errp);
+    if (s->fd < 0) {
+        s->file = NULL;
+        /* There should be a fd_error_set function */
+        s->state = MIG_STATE_ERROR;
+    } else {
+        s->file = qemu_fopen_socket(s->fd, "wb");
+        local_migration_fd_connect(s);
+    }
+}
+
 static void unix_accept_incoming_migration(void *opaque)
 {
     struct sockaddr_un addr;
diff --git a/qapi-schema.json b/qapi-schema.json
index a80ee40..7431a41 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -2336,6 +2336,20 @@
 { 'command': 'migrate',
   'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool', '*detach': 'bool' } }
 
+##
+# @localhost-migrate
+#
+# Migrates the current running guest to the localhost VM.
+#
+# @uri: the Uniform Resource Identifier of the destination VM
+#
+# Returns: nothing on success
+#
+# Since: 1.7
+##
+{ 'command': 'localhost-migrate',
+  'data': {'uri': 'str'} }
+
 # @xen-save-devices-state:
 #
 # Save the state of all devices to file. The RAM and the block devices
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 8cea5e5..aba2327 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -640,6 +640,28 @@ Notes:
 EQMP
 
     {
+        .name      = "localhost-migrate",
+        .args_type = "uri:s",
+        .mhandler.cmd_new = qmp_marshal_input_localhost_migrate,
+    },
+
+SQMP
+localhost-migrate
+
+Migrate VM in localhost.
+
+Arguments:
+
+- "uri": Destination URI (json-string)
+
+Example:
+
+-> { "execute": "localhost-migrate", "arguments": { "uri": "UNIX-SOCKET" }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "migrate_cancel",
         .args_type  = "",
         .mhandler.cmd_new = qmp_marshal_input_migrate_cancel,
-- 
1.7.7.6

[Qemu-devel] [PATCH 10/12] migration-local: implementation of incoming part

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/migration.h |    6 +++++
 migration-local.c             |   39 ++++++++++++++++++++++++++++++++++
 migration-unix.c              |   47 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 0 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index a690e18..5e7416f 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -73,6 +73,10 @@ void process_incoming_migration(QEMUFile *f);
 
 void qemu_start_incoming_migration(const char *uri, Error **errp);
 
+void start_local_incoming_migration(QEMUFile *f);
+
+void qemu_start_local_incoming_migration(const char *uri, Error **errp);
+
 uint64_t migrate_max_downtime(void);
 
 void do_info_migrate_print(Monitor *mon, const QObject *data);
@@ -89,6 +93,8 @@ void tcp_start_outgoing_migration(MigrationState *s, const char *host_port, Erro
 
 void unix_start_incoming_migration(const char *path, Error **errp);
 
+void unix_start_local_incoming_migration(const char *path, Error **errp);
+
 void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
 
 void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp);
diff --git a/migration-local.c b/migration-local.c
index 5bd1ed0..479b796 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -156,3 +156,42 @@ fail:
         vm_start();
     }
 }
+
+/**********************************************************************
+ * Incoming part
+ */
+
+void qemu_start_local_incoming_migration(const char *uri, Error **errp)
+{
+    const char *p;
+
+    if (strstart(uri, "unix:", &p)) {
+        unix_start_local_incoming_migration(p, errp);
+    } else {
+        error_setg(errp, "unknown migration protocol: %s", uri);
+    }
+}
+
+void start_local_incoming_migration(QEMUFile *f)
+{
+    int ret;
+
+    ret = qemu_loadvm_state(f);
+    if (ret < 0) {
+        fprintf(stderr, "load of migration failed\n");
+        exit(EXIT_FAILURE);
+    }
+    qemu_announce_self();
+
+    DPRINTF("successfully loaded vm state\n");
+
+    bdrv_clear_incoming_migration_all();
+    /* Make sure all file formats flush their mutable metadata */
+    bdrv_invalidate_cache_all();
+
+    if (autostart) {
+        vm_start();
+    } else {
+        runstate_set(RUN_STATE_PAUSED);
+    }
+}
diff --git a/migration-unix.c b/migration-unix.c
index ec20c45..70c847a 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -108,3 +108,50 @@ void unix_start_incoming_migration(const char *path, Error **errp)
     qemu_set_fd_handler2(s, NULL, unix_accept_incoming_migration, NULL,
                          (void *)(intptr_t)s);
 }
+
+static void unix_accept_local_incoming_migration(void *opaque)
+{
+    struct sockaddr_un addr;
+    socklen_t addrlen = sizeof(addr);
+    int s = (intptr_t)opaque;
+    QEMUFile *f;
+    int c;
+
+    do {
+        c = qemu_accept(s, (struct sockaddr *)&addr, &addrlen);
+    } while (c == -1 && errno == EINTR);
+    qemu_set_fd_handler2(s, NULL, NULL, NULL, NULL);
+    close(s);
+
+    DPRINTF("accepted migration\n");
+
+    if (c == -1) {
+        fprintf(stderr, "could not accept migration connection\n");
+        goto out;
+    }
+
+    f = qemu_fopen_socket(c, "rb");
+    if (f == NULL) {
+        fprintf(stderr, "could not qemu_fopen socket\n");
+        goto out;
+    }
+
+    start_local_incoming_migration(f);
+    return;
+
+out:
+    close(c);
+}
+
+void unix_start_local_incoming_migration(const char *path, Error **errp)
+{
+    int ret;
+
+    ret = unix_listen(path, NULL, 0, errp);
+    if (ret < 0) {
+        return;
+    }
+
+    qemu_set_fd_handler2(ret, NULL, unix_accept_local_incoming_migration, NULL,
+                         (void *)(intptr_t)ret);
+}
-- 
1.7.7.6

[Qemu-devel] [PATCH 11/12] migration-local: add option to commandline for incoming-local

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 qemu-options.hx |    9 +++++++++
 vl.c            |   14 ++++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/qemu-options.hx b/qemu-options.hx
index 8355f9b..a975e83 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -2912,6 +2912,15 @@ STEXI
 Prepare for incoming migration, listen on @var{port}.
 ETEXI
 
+DEF("incoming-local", HAS_ARG, QEMU_OPTION_incoming_local, \
+    "-incoming-local p prepare for localhost incoming migration, listen on domain unix socket p\n",
+    QEMU_ARCH_ALL)
+STEXI
+@item -incoming-local @var{port}
+@findex -incoming-local
+Prepare for localhost incoming migration, listen on @var{port}
+ETEXI
+
 DEF("nodefaults", 0, QEMU_OPTION_nodefaults, \
     "-nodefaults     don't create default devices\n", QEMU_ARCH_ALL)
 STEXI
diff --git a/vl.c b/vl.c
index 767e020..b820db5 100644
--- a/vl.c
+++ b/vl.c
@@ -2854,6 +2854,7 @@ int main(int argc, char **argv, char **envp)
     const char *vga_model = "none";
     const char *pid_file = NULL;
     const char *incoming = NULL;
+    const char *incoming_local = NULL;
 #ifdef CONFIG_VNC
     int show_vnc_port = 0;
 #endif
@@ -3691,6 +3692,10 @@ int main(int argc, char **argv, char **envp)
                 incoming = optarg;
                 runstate_set(RUN_STATE_INMIGRATE);
                 break;
+            case QEMU_OPTION_incoming_local:
+                incoming_local = optarg;
+                runstate_set(RUN_STATE_INMIGRATE);
+                break;
             case QEMU_OPTION_nodefaults:
                 default_serial = 0;
                 default_parallel = 0;
@@ -4377,6 +4382,15 @@ int main(int argc, char **argv, char **envp)
             error_free(local_err);
             exit(1);
         }
+    } else if (incoming_local) {
+        Error *local_err = NULL;
+        qemu_start_local_incoming_migration(incoming_local, &local_err);
+        if (local_err) {
+            fprintf(stderr, "-incoming_local %s: %s\n", incoming_local,
+                    error_get_pretty(local_err));
+            error_free(local_err);
+            exit(1);
+        }
     } else if (autostart) {
         vm_start();
     }
-- 
1.7.7.6

[Qemu-devel] [PATCH 12/12] hmp: add hmp_localhost_migration interface

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 hmp-commands.hx |   17 +++++++++++++++++
 hmp.c           |   13 +++++++++++++
 hmp.h           |    1 +
 3 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index 915b0d1..ed21970 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -989,6 +989,23 @@ STEXI
 Set the spice/vnc connection info for the migration target.  The spice/vnc
 server will ask the spice/vnc client to automatically reconnect using the
 new parameters (if specified) once the vm migration finished successfully.
+
+ETEXI
+
+    {
+        .name       = "localhost_migrate",
+        .args_type  = "uri:s",
+        .params     = "uri",
+        .help       = "migrate to domain socket URI on localhost",
+        .mhandler.cmd = hmp_localhost_migrate,
+    },
+
+
+STEXI
+@item localhost_migrate @var{uri}
+@findex localhost_migrate
+Migrate to @var{uri}.
+
 ETEXI
 
     {
diff --git a/hmp.c b/hmp.c
index 494a9aa..a7e779f 100644
--- a/hmp.c
+++ b/hmp.c
@@ -1191,6 +1191,19 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_localhost_migrate(Monitor *mon, const QDict *qdict)
+{
+    const char *uri = qdict_get_str(qdict, "uri");
+    Error *errp = NULL;
+
+    qmp_localhost_migrate(uri, &errp);
+    if (errp) {
+        monitor_printf(mon, "local_migrate: %s\n", error_get_pretty(errp));
+        error_free(errp);
+        return;
+    }
+}
+
 void hmp_device_del(Monitor *mon, const QDict *qdict)
 {
     const char *id = qdict_get_str(qdict, "id");
diff --git a/hmp.h b/hmp.h
index 56d2e92..9124c0e 100644
--- a/hmp.h
+++ b/hmp.h
@@ -60,6 +60,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
 void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
+void hmp_localhost_migrate(Monitor *mon, const QDict *qdict);
 void hmp_set_password(Monitor *mon, const QDict *qdict);
 void hmp_expire_password(Monitor *mon, const QDict *qdict);
 void hmp_eject(Monitor *mon, const QDict *qdict);
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration

[Paolo Bonzini]

Il 25/07/2013 22:18, Lei Li ha scritto:
> Hi,
> 
> This patch series tries to add localhost migration support to 
> Qemu. 
> 
> When doing localhost migration, the host memory will balloon
> up during the period, might consume double memories for some time.
> So we want to add a new live migration mechanism localhost
> migration.
> 
> Following I copied from last version that Anthony added for the 
> benefit of the other reviewers:
> 
> The goal here is to allow "live upgrade" of a running QEMU instance.  
> The work flow would look like this:
> 
> 1) Guests are running QEMU release 1.6.1
> 2) Admin installs QEMU release 1.6.2 via RPM or deb
> 3) Admin does localhost migration with page flipping to use new version
>    of QEMU.
> 
> Page flipping is used in order to avoid requiring that there is enough
> free memory to fit an additional copy of the largest guest which is the
> requirement today with localhost migration.
> 
> You can also read from the link below:
> http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html
> 
> The plan is:
> 
> 1) Add new command to do localhost migration.
> 
>    The qmp interface introduced like:
> 
>    { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
> 
> 2) Use different mechanism than current live migration.
> 
>    The very basic work flow like:
> 
>        qemu on the source (the source and destination are both on localhost)
>               |
>               V
>            Stop VM
>               |
>               V   
>           Create threads
>               |
>               V
>        Page flipping through vmspice
>               |
>               V
>        MADV_DONTNEED the ram pages which are already flipped
>               |
>               V
>        Migration completes
> 
>    As stopping VM first, we expect/resume the page flipping through vmspice
>    is fast enough to meet *live migration (low downtime).
> 
> Notes:
> Currently the work flow is not exactly the same as description
> above. For the first step, the work flow we implemented is:
> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED 
> ram pages that already copied. After that, will replace to vmsplice
> mechanism instead of copying pages.
> 
> Now it's still a draft version, and as it is implemented separately 
> to the current migration code for a easy start, the next step will 
> be trying to integrate it into the current migration implementation 
> closely. To make sure we are on the right direction that should be 
> headed, please let me know your suggestions on this.
> 
> For the interface, as Anthony has suggested using a flag or a 
> capability, which one would you prefer or any ideas?

Using a capability on the source makes sense; I don't think anything
special is needed on the destination.  The destination just sees a
special packet telling it that a pipe is available via SCM_RIGHTS; then
it fetches the file descriptor and uses it for the new protocol.

It looks like this could reuse a lot of the RAM copying hooks that we
introduced for RDMA.  You shouldn't need to change anything in savevm.c
or arch_init.c

Paolo

> Your comments are very welcome!
> 
> TODO:
> - Integrate to the current implement of migration closely?
> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
> - benchmark/evaluation.
> 
> Lei Li (12):
>   migration: export MIG_STATE_xxx flags 
>   savevm: export qemu_save_device_state()
>   rename is_active to is_block_active
>   arch_init: introduce ram_page_save()
>   arch_init: introduce ram_save_local()
>   arch_init: add save_local_setup to savevm_ram_handlers
>   savevm: introduce qemu_savevm_local()
>   savevm: adjust is_ram check in register_savevm_live()
>   migration-local: implementation of outgoing part
>   migration-local: implementation of incoming part
>   migration-local: add option to command line for local incoming
>   hmp:add hmp_localhost_migration interface
> 
>  Makefile.objs                 |    1 +
>  arch_init.c                   |   110 +++++++++++++++++++
>  block-migration.c             |    2 +-
>  hmp-commands.hx               |   17 ++++
>  hmp.c                         |   13 +++
>  hmp.h                         |    1 +
>  include/migration/migration.h |   32 +++++++
>  include/sysemu/sysemu.h       |    1 +
>  migration-local.c             |  228 +++++++++++++++++++++++++++++++++++++++++
>  migration-unix.c              |   60 ++++++++++++
>  migration.c                   |    8 --
>  qapi-schema.json              |   14 +++
>  qemu-options.hx               |    9 ++
>  qmp-commands.hx               |   22 +++++
>  savevm.c                      |   100 +++++++++++++++--
>  vl.c                          |   14 +++
>  16 files changed, 613 insertions(+), 19 deletions(-)
>  create mode 100644 migration-local.c
> 

Re: [Qemu-devel] [PATCH 04/12] arch_init: introduce ram_page_save()

[Michael R. Hines]

On 07/25/2013 04:18 PM, Lei Li wrote:
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   arch_init.c |   58 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 files changed, 58 insertions(+), 0 deletions(-)
>
> diff --git a/arch_init.c b/arch_init.c
> index a8b91ee..a418071 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -699,6 +699,64 @@ static uint64_t ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size)
>       return remaining_size;
>   }
>
> +/* This is the last block from where we have sent data for local migration */
> +static RAMBlock *last_block_local;
> +static ram_addr_t last_offset_local;
> +
> +static int ram_page_save(QEMUFile *f)
> +{
> +    RAMBlock *block = last_block_local;
> +    ram_addr_t offset = last_offset_local;
> +    MemoryRegion *mr = block->mr;
> +    int bytes_sent = 0;
> +
> +    if (!block) {
> +        block = QTAILQ_FIRST(&ram_list.blocks);
> +    }
> +
> +    do {
> +        mr = block->mr;
> +        uint8_t *p;
> +        int cont = (block == last_block_local) ? RAM_SAVE_FLAG_CONTINUE : 0;
> +
> +        p = memory_region_get_ram_ptr(mr) + offset;
> +
> +        if (is_zero_page(p)) {
> +            qemu_put_be64(f, offset | cont | RAM_SAVE_FLAG_COMPRESS);
> +            if (!cont) {
> +                qemu_put_byte(f, strlen(block->idstr));
> +                qemu_put_buffer(f, (uint8_t *)block->idstr,
> +                                strlen(block->idstr));
> +            }
> +            qemu_put_byte(f, *p);
> +            bytes_sent = 1;
> +        } else {
> +            qemu_put_be64(f, offset | cont | RAM_SAVE_FLAG_PAGE);
> +            if (!cont) {
> +                qemu_put_byte(f, strlen(block->idstr));
> +                qemu_put_buffer(f, (uint8_t *)block->idstr,
> +                                strlen(block->idstr));
> +            }
> +            qemu_put_buffer(f, p, TARGET_PAGE_SIZE);
> +            bytes_sent = TARGET_PAGE_SIZE;
> +        }
> +
> +        offset += TARGET_PAGE_SIZE;
> +        if (offset >= block->length) {
> +            offset = 0;
> +            block = QTAILQ_NEXT(block, next);
> +            if (!block) {
> +                block = QTAILQ_FIRST(&ram_list.blocks);
> +            }
> +        }
> +    } while (block != last_block_local || offset != last_offset_local);
> +
> +    last_block_local = block;
> +    last_offset_local = offset;
> +
> +    return bytes_sent;
> +}
> +
>   static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
>   {
>       int ret, rc = 0;

Seems like there's a lot of duplicate code.

We have new hooks to override the way memory is saved in arch_init.c

See the QEMUFileOps......... save_page() / before_ram_iterate() / 
after_ram_iterate()

You might need to introduce a new "load_page()" pointer in QEMUFileOps.

Then all this code can be private to migration-local.c

- Michael

Re: [Qemu-devel] [PATCH 05/12] arch_init: introduce ram_save_local()

[Michael R. Hines]

On 07/25/2013 04:18 PM, Lei Li wrote:
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   arch_init.c |   32 ++++++++++++++++++++++++++++++++
>   1 files changed, 32 insertions(+), 0 deletions(-)
>
> diff --git a/arch_init.c b/arch_init.c
> index a418071..7eeb52f 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -757,6 +757,38 @@ static int ram_page_save(QEMUFile *f)
>       return bytes_sent;
>   }
>
> +static int ram_save_local(QEMUFile *f, void *opaque)
> +{
> +    uint64_t bytes_sent = 0;
> +    uint64_t total_ram_bytes = ram_bytes_total();
> +
> +    qemu_mutex_lock_ramlist();
> +
> +    while (total_ram_bytes) {
> +       void *ram;
> +       MemoryRegion *mr;
> +
> +       bytes_sent = ram_page_save(f);
> +       /* No more ram pages. */
> +       if (bytes_sent == 0) {
> +           return bytes_sent;
> +       }
> +
> +       mr = last_block_local->mr;
> +       ram = memory_region_get_ram_ptr(mr) + last_offset_local;
> +
> +       /* DONTNEED the ram page that has already copied. */
> +       qemu_madvise(ram, bytes_sent, QEMU_MADV_DONTNEED);
> +       total_ram_bytes -= bytes_sent;
> +    }
> +
> +    qemu_mutex_unlock_ramlist();
> +
> +    qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
> +
> +    return bytes_sent;
> +}
> +
>   static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
>   {
>       int ret, rc = 0;

You need to create a new private structure "QEMUFileLocal".

Then override f->save_page and point this to your own function.

See migration-rdma.c for an example.

- Michael

Re: [Qemu-devel] [PATCH 06/12] arch_init: add save_local_setup to savevm_ram_handlers

[Michael R. Hines]

On 07/25/2013 04:18 PM, Lei Li wrote:
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   arch_init.c                 |    1 +
>   include/migration/vmstate.h |    2 ++
>   2 files changed, 3 insertions(+), 0 deletions(-)
>
> diff --git a/arch_init.c b/arch_init.c
> index 7eeb52f..5c25005 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -973,6 +973,7 @@ SaveVMHandlers savevm_ram_handlers = {
>       .save_live_iterate = ram_save_iterate,
>       .save_live_complete = ram_save_complete,
>       .save_live_pending = ram_save_pending,
> +    .save_local_setup = ram_save_local,
>       .load_state = ram_load,
>       .cancel = ram_migration_cancel,
>   };
> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> index acf847b..c534254 100644
> --- a/include/migration/vmstate.h
> +++ b/include/migration/vmstate.h
> @@ -53,6 +53,8 @@ typedef struct SaveVMHandlers {
>       int (*save_live_setup)(QEMUFile *f, void *opaque);
>       uint64_t (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size);
>
> +    int (*save_local_setup)(QEMUFile *f, void *opaque);
> +
>       LoadStateHandler *load_state;
>   } SaveVMHandlers;
>

save_page() should be good enough for you - it has more parameters too 
.... see last email....

- Michael

Re: [Qemu-devel] [PATCH 09/12] migration-local: implementation of outgoing part

[Michael R. Hines]

On 07/25/2013 04:18 PM, Lei Li wrote:
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   Makefile.objs                 |    1 +
>   include/migration/migration.h |   15 ++++
>   migration-local.c             |  158 +++++++++++++++++++++++++++++++++++++++++
>   migration-unix.c              |   13 ++++
>   qapi-schema.json              |   14 ++++
>   qmp-commands.hx               |   22 ++++++
>   6 files changed, 223 insertions(+), 0 deletions(-)
>   create mode 100644 migration-local.c
>
> diff --git a/Makefile.objs b/Makefile.objs
> index 5b288ba..2a3d9a5 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -53,6 +53,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/
>   common-obj-y += migration.o migration-tcp.o
>   common-obj-y += qemu-char.o #aio.o
>   common-obj-y += block-migration.o
> +common-obj-y += migration-local.o
>   common-obj-y += page_cache.o xbzrle.o
>
>   common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index a821c80..a690e18 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -58,6 +58,17 @@ struct MigrationState
>       int64_t xbzrle_cache_size;
>   };
>
> +
> +typedef struct LocalMigState LocalMigState;
> +
> +struct LocalMigState
> +{
> +    int fd;
> +    int state;
> +    QEMUFile *file;
> +    QemuThread thread;
> +};
> +
>   void process_incoming_migration(QEMUFile *f);
>
>   void qemu_start_incoming_migration(const char *uri, Error **errp);
> @@ -80,6 +91,8 @@ void unix_start_incoming_migration(const char *path, Error **errp);
>
>   void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
>
> +void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp);
> +
>   void fd_start_incoming_migration(const char *path, Error **errp);
>
>   void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp);
> @@ -88,6 +101,8 @@ void migrate_fd_error(MigrationState *s);
>
>   void migrate_fd_connect(MigrationState *s);
>
> +void local_migration_fd_connect(LocalMigState *s);
> +
>   int migrate_fd_close(MigrationState *s);
>
>   void add_migration_state_change_notifier(Notifier *notify);
> diff --git a/migration-local.c b/migration-local.c
> new file mode 100644
> index 0000000..5bd1ed0
> --- /dev/null
> +++ b/migration-local.c
> @@ -0,0 +1,158 @@
> +/*
> + * QEMU localhost migration
> + *
> + * Copyright IBM, Corp. 2013
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2.  See
> + * the COPYING file in the top-level directory.
> + *
> + * Contributions are licensed under the terms of the GNU GPL, version 2
> + * or (at your option) any later version.
> + */
> +
> +#include "qemu-common.h"
> +#include "migration/migration.h"
> +#include "monitor/monitor.h"
> +#include "migration/qemu-file.h"
> +#include "sysemu/sysemu.h"
> +#include "block/block.h"
> +#include "qemu/sockets.h"
> +#include "migration/block.h"
> +#include "qemu/thread.h"
> +#include "qmp-commands.h"
> +#include "exec/memory.h"
> +#include "trace.h"
> +#include "qemu/osdep.h"
> +
> +//#define DEBUG_MIGRATION_LOCAL
> +
> +#ifdef DEBUG_MIGRATION_LOCAL
> +#define DPRINTF(fmt, ...) \
> +    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
> +#else
> +#define DPRINTF(fmt, ...) \
> +    do { } while (0)
> +#endif
> +
> +
> +/************************************************************************
> + * Outgoing part
> + */
> +
> +static LocalMigState *local_migration_init(void)
> +{
> +    LocalMigState *s = g_malloc0(sizeof(*s));
> +
> +    s->state = MIG_STATE_SETUP;
> +    trace_migrate_set_state(MIG_STATE_SETUP);
> +    s->fd = -1;
> +
> +    return s;
> +}
> +
> +static void local_migration_error(LocalMigState *s)
> +{
> +    assert(s->file == NULL);
> +
> +    s->state = MIG_STATE_ERROR;
> +    trace_migrate_set_state(MIG_STATE_ERROR);
> +}
> +
> +static void local_outgoing_completed(LocalMigState *s)
> +{
> +    s->state = MIG_STATE_COMPLETED;
> +    trace_migrate_set_state(MIG_STATE_COMPLETED);
> +}
> +
> +static void *migration_local_thread(void *opaque)
> +{
> +    LocalMigState *s = opaque;
> +    int ret;
> +
> +    DPRINTF("Beginning savevm\n");
> +
> +    while (s->state == MIG_STATE_ACTIVE) {
> +        qemu_mutex_lock_iothread();
> +        ret = qemu_savevm_local(s->file);
> +        qemu_mutex_unlock_iothread();
> +
> +        /* No need to send device states if ram pages fails to to sent. */
> +        if (ret < 0) {
> +            local_migration_error(s);
> +            break;
> +        }
> +
> +        qemu_save_device_state(s->file);
> +        qemu_fclose(s->file);
> +    }
> +
> +    ret = qemu_file_get_error(s->file);
> +    if (ret < 0) {
> +        local_migration_error(s);
> +    } else {
> +        local_outgoing_completed(s);
> +    }
> +
> +    qemu_mutex_lock_iothread();
> +
> +    if (s->state == MIG_STATE_COMPLETED) {
> +        runstate_set(RUN_STATE_POSTMIGRATE);
> +    }
> +
> +    qemu_mutex_unlock_iothread();
> +
> +    return NULL;
> +}
> +
> +void local_migration_fd_connect(LocalMigState *s)
> +{
> +    s->state = MIG_STATE_ACTIVE;
> +    trace_migrate_set_state(MIG_STATE_ACTIVE);
> +
> +    qemu_thread_create(&s->thread, migration_local_thread, s,
> +                       QEMU_THREAD_JOINABLE);
> +}
> +
> +void qmp_localhost_migrate(const char *uri, Error **errp)
> +{
> +    const char *path;
> +    Error *local_err = NULL;
> +    int is_vm_running;
> +    LocalMigState *s;
> +
> +    if (qemu_savevm_state_blocked(errp)) {
> +        return;
> +    }
> +
> +    s = local_migration_init();
> +    bdrv_flush_all();
> +
> +    is_vm_running = runstate_is_running();
> +
> +    /* Stop the VM first */
> +    if (is_vm_running) {
> +        vm_stop(RUN_STATE_SAVE_VM);
> +    }
> +
> +    /* Start outgoing migration by unix socket. */
> +    if (strstart(uri, "unix:", &path)) {
> +        /* XXX. Creat a new unix_start_outgoing_migration_* is not necessary,
> +         * just for the first step. This will be replaced by vmsplice
> +         * mechanism. */
> +        unix_start_local_outgoing_migration(s, path, &local_err);
> +    } else {
> +        error_set(errp, QERR_INVALID_PARAMETER_VALUE, "uri", "a valid migration protocol");
> +        goto fail;
> +    }
> +
> +    if (local_err) {
> +        s->state = MIG_STATE_ERROR;
> +        error_propagate(errp, local_err);
> +        goto fail;
> +    }
> +
> +fail:
> +    if (!is_vm_running) {
> +        vm_start();
> +    }
> +}
> diff --git a/migration-unix.c b/migration-unix.c
> index 94b7022..ec20c45 100644
> --- a/migration-unix.c
> +++ b/migration-unix.c
> @@ -49,6 +49,19 @@ void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **
>       unix_nonblocking_connect(path, unix_wait_for_connect, s, errp);
>   }
>
> +void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp)
> +{
> +    s->fd = unix_connect(path, errp);
> +    if (s->fd < 0) {
> +        s->file = NULL;
> +        /* There should be a fd_error_set function */
> +        s->state = MIG_STATE_ERROR;
> +    } else {
> +        s->file = qemu_fopen_socket(s->fd, "wb");
> +        local_migration_fd_connect(s);
> +    }
> +}
> +
>   static void unix_accept_incoming_migration(void *opaque)
>   {
>       struct sockaddr_un addr;
> diff --git a/qapi-schema.json b/qapi-schema.json
> index a80ee40..7431a41 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -2336,6 +2336,20 @@
>   { 'command': 'migrate',
>     'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool', '*detach': 'bool' } }
>
> +##
> +# @localhost-migrate
> +#
> +# Migrates the current running guest to the localhost VM.
> +#
> +# @uri: the Uniform Resource Identifier of the destination VM
> +#
> +# Returns: nothing on success
> +#
> +# Since: 1.7
> +##
> +{ 'command': 'localhost-migrate',
> +  'data': {'uri': 'str'} }
> +
>   # @xen-save-devices-state:
>   #
>   # Save the state of all devices to file. The RAM and the block devices
> diff --git a/qmp-commands.hx b/qmp-commands.hx
> index 8cea5e5..aba2327 100644
> --- a/qmp-commands.hx
> +++ b/qmp-commands.hx
> @@ -640,6 +640,28 @@ Notes:
>   EQMP
>
>       {
> +        .name      = "localhost-migrate",
> +        .args_type = "uri:s",
> +        .mhandler.cmd_new = qmp_marshal_input_localhost_migrate,
> +    },
> +
> +SQMP
> +localhost-migrate
> +
> +Migrate VM in localhost.
> +
> +Arguments:
> +
> +- "uri": Destination URI (json-string)
> +
> +Example:
> +
> +-> { "execute": "localhost-migrate", "arguments": { "uri": "UNIX-SOCKET" }
> +<- { "return": {} }
> +
> +EQMP
> +
> +    {
>           .name       = "migrate_cancel",
>           .args_type  = "",
>           .mhandler.cmd_new = qmp_marshal_input_migrate_cancel,

Why a separate thread? What's wrong with modifying the existing 
migration_thread() ?

- Michael

Re: [Qemu-devel] [PATCH 11/12] migration-local: add option to commandline for incoming-local

[Michael R. Hines]

On 07/25/2013 04:18 PM, Lei Li wrote:
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   qemu-options.hx |    9 +++++++++
>   vl.c            |   14 ++++++++++++++
>   2 files changed, 23 insertions(+), 0 deletions(-)
>
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 8355f9b..a975e83 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -2912,6 +2912,15 @@ STEXI
>   Prepare for incoming migration, listen on @var{port}.
>   ETEXI
>
> +DEF("incoming-local", HAS_ARG, QEMU_OPTION_incoming_local, \
> +    "-incoming-local p prepare for localhost incoming migration, listen on domain unix socket p\n",
> +    QEMU_ARCH_ALL)
> +STEXI
> +@item -incoming-local @var{port}
> +@findex -incoming-local
> +Prepare for localhost incoming migration, listen on @var{port}
> +ETEXI
> +
>   DEF("nodefaults", 0, QEMU_OPTION_nodefaults, \
>       "-nodefaults     don't create default devices\n", QEMU_ARCH_ALL)
>   STEXI
> diff --git a/vl.c b/vl.c
> index 767e020..b820db5 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -2854,6 +2854,7 @@ int main(int argc, char **argv, char **envp)
>       const char *vga_model = "none";
>       const char *pid_file = NULL;
>       const char *incoming = NULL;
> +    const char *incoming_local = NULL;
>   #ifdef CONFIG_VNC
>       int show_vnc_port = 0;
>   #endif
> @@ -3691,6 +3692,10 @@ int main(int argc, char **argv, char **envp)
>                   incoming = optarg;
>                   runstate_set(RUN_STATE_INMIGRATE);
>                   break;
> +            case QEMU_OPTION_incoming_local:
> +                incoming_local = optarg;
> +                runstate_set(RUN_STATE_INMIGRATE);
> +                break;
>               case QEMU_OPTION_nodefaults:
>                   default_serial = 0;
>                   default_parallel = 0;
> @@ -4377,6 +4382,15 @@ int main(int argc, char **argv, char **envp)
>               error_free(local_err);
>               exit(1);
>           }
> +    } else if (incoming_local) {
> +        Error *local_err = NULL;
> +        qemu_start_local_incoming_migration(incoming_local, &local_err);
> +        if (local_err) {
> +            fprintf(stderr, "-incoming_local %s: %s\n", incoming_local,
> +                    error_get_pretty(local_err));
> +            error_free(local_err);
> +            exit(1);
> +        }
>       } else if (autostart) {
>           vm_start();
>       }

Why can't we do: -incoming "local:" instead of adding a new flag?

Would be much more compatible with libvirt tools if you just add a new 
URI prefix.....

- Michael

Re: [Qemu-devel] [PATCH 12/12] hmp: add hmp_localhost_migration interface

[Michael R. Hines]

On 07/25/2013 04:18 PM, Lei Li wrote:
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   hmp-commands.hx |   17 +++++++++++++++++
>   hmp.c           |   13 +++++++++++++
>   hmp.h           |    1 +
>   3 files changed, 31 insertions(+), 0 deletions(-)
>
> diff --git a/hmp-commands.hx b/hmp-commands.hx
> index 915b0d1..ed21970 100644
> --- a/hmp-commands.hx
> +++ b/hmp-commands.hx
> @@ -989,6 +989,23 @@ STEXI
>   Set the spice/vnc connection info for the migration target.  The spice/vnc
>   server will ask the spice/vnc client to automatically reconnect using the
>   new parameters (if specified) once the vm migration finished successfully.
> +
> +ETEXI
> +
> +    {
> +        .name       = "localhost_migrate",
> +        .args_type  = "uri:s",
> +        .params     = "uri",
> +        .help       = "migrate to domain socket URI on localhost",
> +        .mhandler.cmd = hmp_localhost_migrate,
> +    },
> +
> +
> +STEXI
> +@item localhost_migrate @var{uri}
> +@findex localhost_migrate
> +Migrate to @var{uri}.
> +
>   ETEXI
>
>       {
> diff --git a/hmp.c b/hmp.c
> index 494a9aa..a7e779f 100644
> --- a/hmp.c
> +++ b/hmp.c
> @@ -1191,6 +1191,19 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
>       }
>   }
>
> +void hmp_localhost_migrate(Monitor *mon, const QDict *qdict)
> +{
> +    const char *uri = qdict_get_str(qdict, "uri");
> +    Error *errp = NULL;
> +
> +    qmp_localhost_migrate(uri, &errp);
> +    if (errp) {
> +        monitor_printf(mon, "local_migrate: %s\n", error_get_pretty(errp));
> +        error_free(errp);
> +        return;
> +    }
> +}
> +
>   void hmp_device_del(Monitor *mon, const QDict *qdict)
>   {
>       const char *id = qdict_get_str(qdict, "id");
> diff --git a/hmp.h b/hmp.h
> index 56d2e92..9124c0e 100644
> --- a/hmp.h
> +++ b/hmp.h
> @@ -60,6 +60,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const QDict *qdict);
>   void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
>   void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
>   void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
> +void hmp_localhost_migrate(Monitor *mon, const QDict *qdict);
>   void hmp_set_password(Monitor *mon, const QDict *qdict);
>   void hmp_expire_password(Monitor *mon, const QDict *qdict);
>   void hmp_eject(Monitor *mon, const QDict *qdict);

This could go away if you just create a new URI prefix instead of a new 
command-line switch.

- Michael

Re: [Qemu-devel] [PATCH 07/12] savevm: introduce qemu_savevm_local()

[Michael R. Hines]

On 07/25/2013 04:18 PM, Lei Li wrote:
> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>   include/sysemu/sysemu.h |    1 +
>   savevm.c                |   55 +++++++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 56 insertions(+), 0 deletions(-)
>
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index 5b90027..9abe4c9 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -80,6 +80,7 @@ int qemu_savevm_state_iterate(QEMUFile *f);
>   void qemu_savevm_state_complete(QEMUFile *f);
>   void qemu_savevm_state_cancel(void);
>   uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
> +int qemu_savevm_local(QEMUFile *f);
>   int qemu_save_device_state(QEMUFile *f);
>   int qemu_loadvm_state(QEMUFile *f);
>
> diff --git a/savevm.c b/savevm.c
> index e18ca22..c183369 100644
> --- a/savevm.c
> +++ b/savevm.c
> @@ -2007,6 +2007,61 @@ static int qemu_savevm_state(QEMUFile *f)
>   }
>
>   /**
> + * Save all of the ram pages to stream QEMUFile
> + *
> + * Return: negtive for an error
> + */
> +int qemu_savevm_local(QEMUFile *f)
> +{
> +    SaveStateEntry *se;
> +    int ret;
> +
> +    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
> +        if (!se->ops) {
> +            continue;
> +        }
> +    }
> +
> +    qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
> +    qemu_put_be32(f, QEMU_VM_FILE_VERSION);
> +
> +    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
> +        int len;
> +
> +        if (!se->ops || !se->ops->save_local_setup) {
> +            continue;
> +        }
> +        if (se->ops && se->ops->is_block_active) {
> +            continue;
> +        }
> +
> +        /* Section type */
> +        qemu_put_byte(f, QEMU_VM_SECTION_START);
> +        qemu_put_be32(f, se->section_id);
> +
> +        /* ID string */
> +        len = strlen(se->idstr);
> +        qemu_put_byte(f, len);
> +        qemu_put_buffer(f, (uint8_t *)se->idstr, len);
> +
> +        qemu_put_be32(f, se->instance_id);
> +
> +        qemu_put_be32(f, se->version_id);
> +
> +        ret = se->ops->save_local_setup(f, se->opaque);
> +        if (ret < 0) {
> +            qemu_file_set_error(f, ret);
> +            break;
> +        }
> +    }
> +
> +    qemu_put_byte(f, QEMU_VM_EOF);
> +    qemu_fflush(f);
> +
> +    return qemu_file_get_error(f);
> +}
> +
> +/**
>    * Save all of the device states to stream QEMUFile
>    *
>    * Return negtive if there has been an error

You don't have enough comments in your commit messages - please add more.

Why do you need a new savevm function?

- Michael

Re: [Qemu-devel] [PATCH 04/12] arch_init: introduce ram_page_save()

[Lei Li]

On 08/03/2013 03:40 AM, Michael R. Hines wrote:
> On 07/25/2013 04:18 PM, Lei Li wrote:
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   arch_init.c |   58 
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 files changed, 58 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch_init.c b/arch_init.c
>> index a8b91ee..a418071 100644
>> --- a/arch_init.c
>> +++ b/arch_init.c
>> @@ -699,6 +699,64 @@ static uint64_t ram_save_pending(QEMUFile *f, 
>> void *opaque, uint64_t max_size)
>>       return remaining_size;
>>   }
>>
>> +/* This is the last block from where we have sent data for local 
>> migration */
>> +static RAMBlock *last_block_local;
>> +static ram_addr_t last_offset_local;
>> +
>> +static int ram_page_save(QEMUFile *f)
>> +{
>> +    RAMBlock *block = last_block_local;
>> +    ram_addr_t offset = last_offset_local;
>> +    MemoryRegion *mr = block->mr;
>> +    int bytes_sent = 0;
>> +
>> +    if (!block) {
>> +        block = QTAILQ_FIRST(&ram_list.blocks);
>> +    }
>> +
>> +    do {
>> +        mr = block->mr;
>> +        uint8_t *p;
>> +        int cont = (block == last_block_local) ? 
>> RAM_SAVE_FLAG_CONTINUE : 0;
>> +
>> +        p = memory_region_get_ram_ptr(mr) + offset;
>> +
>> +        if (is_zero_page(p)) {
>> +            qemu_put_be64(f, offset | cont | RAM_SAVE_FLAG_COMPRESS);
>> +            if (!cont) {
>> +                qemu_put_byte(f, strlen(block->idstr));
>> +                qemu_put_buffer(f, (uint8_t *)block->idstr,
>> +                                strlen(block->idstr));
>> +            }
>> +            qemu_put_byte(f, *p);
>> +            bytes_sent = 1;
>> +        } else {
>> +            qemu_put_be64(f, offset | cont | RAM_SAVE_FLAG_PAGE);
>> +            if (!cont) {
>> +                qemu_put_byte(f, strlen(block->idstr));
>> +                qemu_put_buffer(f, (uint8_t *)block->idstr,
>> +                                strlen(block->idstr));
>> +            }
>> +            qemu_put_buffer(f, p, TARGET_PAGE_SIZE);
>> +            bytes_sent = TARGET_PAGE_SIZE;
>> +        }
>> +
>> +        offset += TARGET_PAGE_SIZE;
>> +        if (offset >= block->length) {
>> +            offset = 0;
>> +            block = QTAILQ_NEXT(block, next);
>> +            if (!block) {
>> +                block = QTAILQ_FIRST(&ram_list.blocks);
>> +            }
>> +        }
>> +    } while (block != last_block_local || offset != last_offset_local);
>> +
>> +    last_block_local = block;
>> +    last_offset_local = offset;
>> +
>> +    return bytes_sent;
>> +}
>> +
>>   static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
>>   {
>>       int ret, rc = 0;
>
> Seems like there's a lot of duplicate code.
>
> We have new hooks to override the way memory is saved in arch_init.c
>
> See the QEMUFileOps......... save_page() / before_ram_iterate() / 
> after_ram_iterate()
>
> You might need to introduce a new "load_page()" pointer in QEMUFileOps.
>
> Then all this code can be private to migration-local.c

Hi Michael,

This implementation was based on the tree that rdma migration you added have
not been merged...

Yes, Paolo also suggested that reuses the rdma hooks and I am trying to figure
it out. Your comments really helps, thanks for your suggestions!
  

>
> - Michael
>
>
>


-- 
Lei

Re: [Qemu-devel] [PATCH 07/12] savevm: introduce qemu_savevm_local()

[Lei Li]

On 08/03/2013 03:48 AM, Michael R. Hines wrote:
> On 07/25/2013 04:18 PM, Lei Li wrote:
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   include/sysemu/sysemu.h |    1 +
>>   savevm.c                |   55 
>> +++++++++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 56 insertions(+), 0 deletions(-)
>>
>> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
>> index 5b90027..9abe4c9 100644
>> --- a/include/sysemu/sysemu.h
>> +++ b/include/sysemu/sysemu.h
>> @@ -80,6 +80,7 @@ int qemu_savevm_state_iterate(QEMUFile *f);
>>   void qemu_savevm_state_complete(QEMUFile *f);
>>   void qemu_savevm_state_cancel(void);
>>   uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
>> +int qemu_savevm_local(QEMUFile *f);
>>   int qemu_save_device_state(QEMUFile *f);
>>   int qemu_loadvm_state(QEMUFile *f);
>>
>> diff --git a/savevm.c b/savevm.c
>> index e18ca22..c183369 100644
>> --- a/savevm.c
>> +++ b/savevm.c
>> @@ -2007,6 +2007,61 @@ static int qemu_savevm_state(QEMUFile *f)
>>   }
>>
>>   /**
>> + * Save all of the ram pages to stream QEMUFile
>> + *
>> + * Return: negtive for an error
>> + */
>> +int qemu_savevm_local(QEMUFile *f)
>> +{
>> +    SaveStateEntry *se;
>> +    int ret;
>> +
>> +    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
>> +        if (!se->ops) {
>> +            continue;
>> +        }
>> +    }
>> +
>> +    qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
>> +    qemu_put_be32(f, QEMU_VM_FILE_VERSION);
>> +
>> +    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
>> +        int len;
>> +
>> +        if (!se->ops || !se->ops->save_local_setup) {
>> +            continue;
>> +        }
>> +        if (se->ops && se->ops->is_block_active) {
>> +            continue;
>> +        }
>> +
>> +        /* Section type */
>> +        qemu_put_byte(f, QEMU_VM_SECTION_START);
>> +        qemu_put_be32(f, se->section_id);
>> +
>> +        /* ID string */
>> +        len = strlen(se->idstr);
>> +        qemu_put_byte(f, len);
>> +        qemu_put_buffer(f, (uint8_t *)se->idstr, len);
>> +
>> +        qemu_put_be32(f, se->instance_id);
>> +
>> +        qemu_put_be32(f, se->version_id);
>> +
>> +        ret = se->ops->save_local_setup(f, se->opaque);
>> +        if (ret < 0) {
>> +            qemu_file_set_error(f, ret);
>> +            break;
>> +        }
>> +    }
>> +
>> +    qemu_put_byte(f, QEMU_VM_EOF);
>> +    qemu_fflush(f);
>> +
>> +    return qemu_file_get_error(f);
>> +}
>> +
>> +/**
>>    * Save all of the device states to stream QEMUFile
>>    *
>>    * Return negtive if there has been an error
>
> You don't have enough comments in your commit messages - please add more.

Sorry for the missing of commit messages..

>
> Why do you need a new savevm function?

qemu_savevm_local() is introduced to savevm layer that begin
and do the local migration by calling ops->save_local_setup which
is added to savevm_ram_handlers.

>
> - Michael


-- 
Lei

Re: [Qemu-devel] [PATCH 09/12] migration-local: implementation of outgoing part

[Lei Li]

On 08/03/2013 03:45 AM, Michael R. Hines wrote:
> On 07/25/2013 04:18 PM, Lei Li wrote:
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   Makefile.objs                 |    1 +
>>   include/migration/migration.h |   15 ++++
>>   migration-local.c             |  158 
>> +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   13 ++++
>>   qapi-schema.json              |   14 ++++
>>   qmp-commands.hx               |   22 ++++++
>>   6 files changed, 223 insertions(+), 0 deletions(-)
>>   create mode 100644 migration-local.c
>>
>> diff --git a/Makefile.objs b/Makefile.objs
>> index 5b288ba..2a3d9a5 100644
>> --- a/Makefile.objs
>> +++ b/Makefile.objs
>> @@ -53,6 +53,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/
>>   common-obj-y += migration.o migration-tcp.o
>>   common-obj-y += qemu-char.o #aio.o
>>   common-obj-y += block-migration.o
>> +common-obj-y += migration-local.o
>>   common-obj-y += page_cache.o xbzrle.o
>>
>>   common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o 
>> migration-fd.o
>> diff --git a/include/migration/migration.h 
>> b/include/migration/migration.h
>> index a821c80..a690e18 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -58,6 +58,17 @@ struct MigrationState
>>       int64_t xbzrle_cache_size;
>>   };
>>
>> +
>> +typedef struct LocalMigState LocalMigState;
>> +
>> +struct LocalMigState
>> +{
>> +    int fd;
>> +    int state;
>> +    QEMUFile *file;
>> +    QemuThread thread;
>> +};
>> +
>>   void process_incoming_migration(QEMUFile *f);
>>
>>   void qemu_start_incoming_migration(const char *uri, Error **errp);
>> @@ -80,6 +91,8 @@ void unix_start_incoming_migration(const char 
>> *path, Error **errp);
>>
>>   void unix_start_outgoing_migration(MigrationState *s, const char 
>> *path, Error **errp);
>>
>> +void unix_start_local_outgoing_migration(LocalMigState *s, const 
>> char *path, Error **errp);
>> +
>>   void fd_start_incoming_migration(const char *path, Error **errp);
>>
>>   void fd_start_outgoing_migration(MigrationState *s, const char 
>> *fdname, Error **errp);
>> @@ -88,6 +101,8 @@ void migrate_fd_error(MigrationState *s);
>>
>>   void migrate_fd_connect(MigrationState *s);
>>
>> +void local_migration_fd_connect(LocalMigState *s);
>> +
>>   int migrate_fd_close(MigrationState *s);
>>
>>   void add_migration_state_change_notifier(Notifier *notify);
>> diff --git a/migration-local.c b/migration-local.c
>> new file mode 100644
>> index 0000000..5bd1ed0
>> --- /dev/null
>> +++ b/migration-local.c
>> @@ -0,0 +1,158 @@
>> +/*
>> + * QEMU localhost migration
>> + *
>> + * Copyright IBM, Corp. 2013
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2.  
>> See
>> + * the COPYING file in the top-level directory.
>> + *
>> + * Contributions are licensed under the terms of the GNU GPL, version 2
>> + * or (at your option) any later version.
>> + */
>> +
>> +#include "qemu-common.h"
>> +#include "migration/migration.h"
>> +#include "monitor/monitor.h"
>> +#include "migration/qemu-file.h"
>> +#include "sysemu/sysemu.h"
>> +#include "block/block.h"
>> +#include "qemu/sockets.h"
>> +#include "migration/block.h"
>> +#include "qemu/thread.h"
>> +#include "qmp-commands.h"
>> +#include "exec/memory.h"
>> +#include "trace.h"
>> +#include "qemu/osdep.h"
>> +
>> +//#define DEBUG_MIGRATION_LOCAL
>> +
>> +#ifdef DEBUG_MIGRATION_LOCAL
>> +#define DPRINTF(fmt, ...) \
>> +    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
>> +#else
>> +#define DPRINTF(fmt, ...) \
>> +    do { } while (0)
>> +#endif
>> +
>> +
>> +/************************************************************************ 
>>
>> + * Outgoing part
>> + */
>> +
>> +static LocalMigState *local_migration_init(void)
>> +{
>> +    LocalMigState *s = g_malloc0(sizeof(*s));
>> +
>> +    s->state = MIG_STATE_SETUP;
>> +    trace_migrate_set_state(MIG_STATE_SETUP);
>> +    s->fd = -1;
>> +
>> +    return s;
>> +}
>> +
>> +static void local_migration_error(LocalMigState *s)
>> +{
>> +    assert(s->file == NULL);
>> +
>> +    s->state = MIG_STATE_ERROR;
>> +    trace_migrate_set_state(MIG_STATE_ERROR);
>> +}
>> +
>> +static void local_outgoing_completed(LocalMigState *s)
>> +{
>> +    s->state = MIG_STATE_COMPLETED;
>> +    trace_migrate_set_state(MIG_STATE_COMPLETED);
>> +}
>> +
>> +static void *migration_local_thread(void *opaque)
>> +{
>> +    LocalMigState *s = opaque;
>> +    int ret;
>> +
>> +    DPRINTF("Beginning savevm\n");
>> +
>> +    while (s->state == MIG_STATE_ACTIVE) {
>> +        qemu_mutex_lock_iothread();
>> +        ret = qemu_savevm_local(s->file);
>> +        qemu_mutex_unlock_iothread();
>> +
>> +        /* No need to send device states if ram pages fails to to 
>> sent. */
>> +        if (ret < 0) {
>> +            local_migration_error(s);
>> +            break;
>> +        }
>> +
>> +        qemu_save_device_state(s->file);
>> +        qemu_fclose(s->file);
>> +    }
>> +
>> +    ret = qemu_file_get_error(s->file);
>> +    if (ret < 0) {
>> +        local_migration_error(s);
>> +    } else {
>> +        local_outgoing_completed(s);
>> +    }
>> +
>> +    qemu_mutex_lock_iothread();
>> +
>> +    if (s->state == MIG_STATE_COMPLETED) {
>> +        runstate_set(RUN_STATE_POSTMIGRATE);
>> +    }
>> +
>> +    qemu_mutex_unlock_iothread();
>> +
>> +    return NULL;
>> +}
>> +
>> +void local_migration_fd_connect(LocalMigState *s)
>> +{
>> +    s->state = MIG_STATE_ACTIVE;
>> +    trace_migrate_set_state(MIG_STATE_ACTIVE);
>> +
>> +    qemu_thread_create(&s->thread, migration_local_thread, s,
>> +                       QEMU_THREAD_JOINABLE);
>> +}
>> +
>> +void qmp_localhost_migrate(const char *uri, Error **errp)
>> +{
>> +    const char *path;
>> +    Error *local_err = NULL;
>> +    int is_vm_running;
>> +    LocalMigState *s;
>> +
>> +    if (qemu_savevm_state_blocked(errp)) {
>> +        return;
>> +    }
>> +
>> +    s = local_migration_init();
>> +    bdrv_flush_all();
>> +
>> +    is_vm_running = runstate_is_running();
>> +
>> +    /* Stop the VM first */
>> +    if (is_vm_running) {
>> +        vm_stop(RUN_STATE_SAVE_VM);
>> +    }
>> +
>> +    /* Start outgoing migration by unix socket. */
>> +    if (strstart(uri, "unix:", &path)) {
>> +        /* XXX. Creat a new unix_start_outgoing_migration_* is not 
>> necessary,
>> +         * just for the first step. This will be replaced by vmsplice
>> +         * mechanism. */
>> +        unix_start_local_outgoing_migration(s, path, &local_err);
>> +    } else {
>> +        error_set(errp, QERR_INVALID_PARAMETER_VALUE, "uri", "a 
>> valid migration protocol");
>> +        goto fail;
>> +    }
>> +
>> +    if (local_err) {
>> +        s->state = MIG_STATE_ERROR;
>> +        error_propagate(errp, local_err);
>> +        goto fail;
>> +    }
>> +
>> +fail:
>> +    if (!is_vm_running) {
>> +        vm_start();
>> +    }
>> +}
>> diff --git a/migration-unix.c b/migration-unix.c
>> index 94b7022..ec20c45 100644
>> --- a/migration-unix.c
>> +++ b/migration-unix.c
>> @@ -49,6 +49,19 @@ void unix_start_outgoing_migration(MigrationState 
>> *s, const char *path, Error **
>>       unix_nonblocking_connect(path, unix_wait_for_connect, s, errp);
>>   }
>>
>> +void unix_start_local_outgoing_migration(LocalMigState *s, const 
>> char *path, Error **errp)
>> +{
>> +    s->fd = unix_connect(path, errp);
>> +    if (s->fd < 0) {
>> +        s->file = NULL;
>> +        /* There should be a fd_error_set function */
>> +        s->state = MIG_STATE_ERROR;
>> +    } else {
>> +        s->file = qemu_fopen_socket(s->fd, "wb");
>> +        local_migration_fd_connect(s);
>> +    }
>> +}
>> +
>>   static void unix_accept_incoming_migration(void *opaque)
>>   {
>>       struct sockaddr_un addr;
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index a80ee40..7431a41 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -2336,6 +2336,20 @@
>>   { 'command': 'migrate',
>>     'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool', '*detach': 
>> 'bool' } }
>>
>> +##
>> +# @localhost-migrate
>> +#
>> +# Migrates the current running guest to the localhost VM.
>> +#
>> +# @uri: the Uniform Resource Identifier of the destination VM
>> +#
>> +# Returns: nothing on success
>> +#
>> +# Since: 1.7
>> +##
>> +{ 'command': 'localhost-migrate',
>> +  'data': {'uri': 'str'} }
>> +
>>   # @xen-save-devices-state:
>>   #
>>   # Save the state of all devices to file. The RAM and the block devices
>> diff --git a/qmp-commands.hx b/qmp-commands.hx
>> index 8cea5e5..aba2327 100644
>> --- a/qmp-commands.hx
>> +++ b/qmp-commands.hx
>> @@ -640,6 +640,28 @@ Notes:
>>   EQMP
>>
>>       {
>> +        .name      = "localhost-migrate",
>> +        .args_type = "uri:s",
>> +        .mhandler.cmd_new = qmp_marshal_input_localhost_migrate,
>> +    },
>> +
>> +SQMP
>> +localhost-migrate
>> +
>> +Migrate VM in localhost.
>> +
>> +Arguments:
>> +
>> +- "uri": Destination URI (json-string)
>> +
>> +Example:
>> +
>> +-> { "execute": "localhost-migrate", "arguments": { "uri": 
>> "UNIX-SOCKET" }
>> +<- { "return": {} }
>> +
>> +EQMP
>> +
>> +    {
>>           .name       = "migrate_cancel",
>>           .args_type  = "",
>>           .mhandler.cmd_new = qmp_marshal_input_migrate_cancel,
>
> Why a separate thread? What's wrong with modifying the existing 
> migration_thread() ?

Well, as mentioned in the cover letter, it was implemented
separately to the current migration code just for a easier start..
I am looking for suggestions on the proper way to integrate it.

Yes, modify the existing migration_thread() is an option.

>
> - Michael
>
>


-- 
Lei

Re: [Qemu-devel] [PATCH 11/12] migration-local: add option to commandline for incoming-local

[Lei Li]

On 08/03/2013 03:46 AM, Michael R. Hines wrote:
> On 07/25/2013 04:18 PM, Lei Li wrote:
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   qemu-options.hx |    9 +++++++++
>>   vl.c            |   14 ++++++++++++++
>>   2 files changed, 23 insertions(+), 0 deletions(-)
>>
>> diff --git a/qemu-options.hx b/qemu-options.hx
>> index 8355f9b..a975e83 100644
>> --- a/qemu-options.hx
>> +++ b/qemu-options.hx
>> @@ -2912,6 +2912,15 @@ STEXI
>>   Prepare for incoming migration, listen on @var{port}.
>>   ETEXI
>>
>> +DEF("incoming-local", HAS_ARG, QEMU_OPTION_incoming_local, \
>> +    "-incoming-local p prepare for localhost incoming migration, 
>> listen on domain unix socket p\n",
>> +    QEMU_ARCH_ALL)
>> +STEXI
>> +@item -incoming-local @var{port}
>> +@findex -incoming-local
>> +Prepare for localhost incoming migration, listen on @var{port}
>> +ETEXI
>> +
>>   DEF("nodefaults", 0, QEMU_OPTION_nodefaults, \
>>       "-nodefaults     don't create default devices\n", QEMU_ARCH_ALL)
>>   STEXI
>> diff --git a/vl.c b/vl.c
>> index 767e020..b820db5 100644
>> --- a/vl.c
>> +++ b/vl.c
>> @@ -2854,6 +2854,7 @@ int main(int argc, char **argv, char **envp)
>>       const char *vga_model = "none";
>>       const char *pid_file = NULL;
>>       const char *incoming = NULL;
>> +    const char *incoming_local = NULL;
>>   #ifdef CONFIG_VNC
>>       int show_vnc_port = 0;
>>   #endif
>> @@ -3691,6 +3692,10 @@ int main(int argc, char **argv, char **envp)
>>                   incoming = optarg;
>>                   runstate_set(RUN_STATE_INMIGRATE);
>>                   break;
>> +            case QEMU_OPTION_incoming_local:
>> +                incoming_local = optarg;
>> +                runstate_set(RUN_STATE_INMIGRATE);
>> +                break;
>>               case QEMU_OPTION_nodefaults:
>>                   default_serial = 0;
>>                   default_parallel = 0;
>> @@ -4377,6 +4382,15 @@ int main(int argc, char **argv, char **envp)
>>               error_free(local_err);
>>               exit(1);
>>           }
>> +    } else if (incoming_local) {
>> +        Error *local_err = NULL;
>> +        qemu_start_local_incoming_migration(incoming_local, 
>> &local_err);
>> +        if (local_err) {
>> +            fprintf(stderr, "-incoming_local %s: %s\n", incoming_local,
>> +                    error_get_pretty(local_err));
>> +            error_free(local_err);
>> +            exit(1);
>> +        }
>>       } else if (autostart) {
>>           vm_start();
>>       }
>
> Why can't we do: -incoming "local:" instead of adding a new flag?
>
> Would be much more compatible with libvirt tools if you just add a new 
> URI prefix.....

Good suggestion, thanks!

>
> - Michael
>
>


-- 
Lei

Re: [Qemu-devel] [PATCH 12/12] hmp: add hmp_localhost_migration interface

[Lei Li]

On 08/03/2013 03:47 AM, Michael R. Hines wrote:
> On 07/25/2013 04:18 PM, Lei Li wrote:
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   hmp-commands.hx |   17 +++++++++++++++++
>>   hmp.c           |   13 +++++++++++++
>>   hmp.h           |    1 +
>>   3 files changed, 31 insertions(+), 0 deletions(-)
>>
>> diff --git a/hmp-commands.hx b/hmp-commands.hx
>> index 915b0d1..ed21970 100644
>> --- a/hmp-commands.hx
>> +++ b/hmp-commands.hx
>> @@ -989,6 +989,23 @@ STEXI
>>   Set the spice/vnc connection info for the migration target. The 
>> spice/vnc
>>   server will ask the spice/vnc client to automatically reconnect 
>> using the
>>   new parameters (if specified) once the vm migration finished 
>> successfully.
>> +
>> +ETEXI
>> +
>> +    {
>> +        .name       = "localhost_migrate",
>> +        .args_type  = "uri:s",
>> +        .params     = "uri",
>> +        .help       = "migrate to domain socket URI on localhost",
>> +        .mhandler.cmd = hmp_localhost_migrate,
>> +    },
>> +
>> +
>> +STEXI
>> +@item localhost_migrate @var{uri}
>> +@findex localhost_migrate
>> +Migrate to @var{uri}.
>> +
>>   ETEXI
>>
>>       {
>> diff --git a/hmp.c b/hmp.c
>> index 494a9aa..a7e779f 100644
>> --- a/hmp.c
>> +++ b/hmp.c
>> @@ -1191,6 +1191,19 @@ void hmp_migrate(Monitor *mon, const QDict 
>> *qdict)
>>       }
>>   }
>>
>> +void hmp_localhost_migrate(Monitor *mon, const QDict *qdict)
>> +{
>> +    const char *uri = qdict_get_str(qdict, "uri");
>> +    Error *errp = NULL;
>> +
>> +    qmp_localhost_migrate(uri, &errp);
>> +    if (errp) {
>> +        monitor_printf(mon, "local_migrate: %s\n", 
>> error_get_pretty(errp));
>> +        error_free(errp);
>> +        return;
>> +    }
>> +}
>> +
>>   void hmp_device_del(Monitor *mon, const QDict *qdict)
>>   {
>>       const char *id = qdict_get_str(qdict, "id");
>> diff --git a/hmp.h b/hmp.h
>> index 56d2e92..9124c0e 100644
>> --- a/hmp.h
>> +++ b/hmp.h
>> @@ -60,6 +60,7 @@ void hmp_migrate_set_downtime(Monitor *mon, const 
>> QDict *qdict);
>>   void hmp_migrate_set_speed(Monitor *mon, const QDict *qdict);
>>   void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
>>   void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
>> +void hmp_localhost_migrate(Monitor *mon, const QDict *qdict);
>>   void hmp_set_password(Monitor *mon, const QDict *qdict);
>>   void hmp_expire_password(Monitor *mon, const QDict *qdict);
>>   void hmp_eject(Monitor *mon, const QDict *qdict);
>
> This could go away if you just create a new URI prefix instead of a 
> new command-line switch.

Yes, you are right.
I think a new URI prefix is a good idea.

>
> - Michael


-- 
Lei

Re: [Qemu-devel] [PATCH 05/12] arch_init: introduce ram_save_local()

[Lei Li]

On 08/03/2013 03:42 AM, Michael R. Hines wrote:
> On 07/25/2013 04:18 PM, Lei Li wrote:
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   arch_init.c |   32 ++++++++++++++++++++++++++++++++
>>   1 files changed, 32 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch_init.c b/arch_init.c
>> index a418071..7eeb52f 100644
>> --- a/arch_init.c
>> +++ b/arch_init.c
>> @@ -757,6 +757,38 @@ static int ram_page_save(QEMUFile *f)
>>       return bytes_sent;
>>   }
>>
>> +static int ram_save_local(QEMUFile *f, void *opaque)
>> +{
>> +    uint64_t bytes_sent = 0;
>> +    uint64_t total_ram_bytes = ram_bytes_total();
>> +
>> +    qemu_mutex_lock_ramlist();
>> +
>> +    while (total_ram_bytes) {
>> +       void *ram;
>> +       MemoryRegion *mr;
>> +
>> +       bytes_sent = ram_page_save(f);
>> +       /* No more ram pages. */
>> +       if (bytes_sent == 0) {
>> +           return bytes_sent;
>> +       }
>> +
>> +       mr = last_block_local->mr;
>> +       ram = memory_region_get_ram_ptr(mr) + last_offset_local;
>> +
>> +       /* DONTNEED the ram page that has already copied. */
>> +       qemu_madvise(ram, bytes_sent, QEMU_MADV_DONTNEED);
>> +       total_ram_bytes -= bytes_sent;
>> +    }
>> +
>> +    qemu_mutex_unlock_ramlist();
>> +
>> +    qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
>> +
>> +    return bytes_sent;
>> +}
>> +
>>   static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
>>   {
>>       int ret, rc = 0;
>
> You need to create a new private structure "QEMUFileLocal".
>
> Then override f->save_page and point this to your own function.
>
> See migration-rdma.c for an example.

Yes, I am looking at it. :)

>
> - Michael


-- 
Lei

Re: [Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration

[Lei Li]

On 07/26/2013 05:41 PM, Paolo Bonzini wrote:
> Il 25/07/2013 22:18, Lei Li ha scritto:
>> Hi,
>>
>> This patch series tries to add localhost migration support to
>> Qemu.
>>
>> When doing localhost migration, the host memory will balloon
>> up during the period, might consume double memories for some time.
>> So we want to add a new live migration mechanism localhost
>> migration.
>>
>> Following I copied from last version that Anthony added for the
>> benefit of the other reviewers:
>>
>> The goal here is to allow "live upgrade" of a running QEMU instance.
>> The work flow would look like this:
>>
>> 1) Guests are running QEMU release 1.6.1
>> 2) Admin installs QEMU release 1.6.2 via RPM or deb
>> 3) Admin does localhost migration with page flipping to use new version
>>     of QEMU.
>>
>> Page flipping is used in order to avoid requiring that there is enough
>> free memory to fit an additional copy of the largest guest which is the
>> requirement today with localhost migration.
>>
>> You can also read from the link below:
>> http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html
>>
>> The plan is:
>>
>> 1) Add new command to do localhost migration.
>>
>>     The qmp interface introduced like:
>>
>>     { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
>>
>> 2) Use different mechanism than current live migration.
>>
>>     The very basic work flow like:
>>
>>         qemu on the source (the source and destination are both on localhost)
>>                |
>>                V
>>             Stop VM
>>                |
>>                V
>>            Create threads
>>                |
>>                V
>>         Page flipping through vmspice
>>                |
>>                V
>>         MADV_DONTNEED the ram pages which are already flipped
>>                |
>>                V
>>         Migration completes
>>
>>     As stopping VM first, we expect/resume the page flipping through vmspice
>>     is fast enough to meet *live migration (low downtime).
>>
>> Notes:
>> Currently the work flow is not exactly the same as description
>> above. For the first step, the work flow we implemented is:
>> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
>> ram pages that already copied. After that, will replace to vmsplice
>> mechanism instead of copying pages.
>>
>> Now it's still a draft version, and as it is implemented separately
>> to the current migration code for a easy start, the next step will
>> be trying to integrate it into the current migration implementation
>> closely. To make sure we are on the right direction that should be
>> headed, please let me know your suggestions on this.
>>
>> For the interface, as Anthony has suggested using a flag or a
>> capability, which one would you prefer or any ideas?
> Using a capability on the source makes sense; I don't think anything
> special is needed on the destination.  The destination just sees a
> special packet telling it that a pipe is available via SCM_RIGHTS; then
> it fetches the file descriptor and uses it for the new protocol.
>
> It looks like this could reuse a lot of the RAM copying hooks that we
> introduced for RDMA.  You shouldn't need to change anything in savevm.c
> or arch_init.c

Hi Paolo,

Thanks for your suggestions!

And sorry for the late reply, I am looking into RDMA implementation and
trying to figure out the way to integrate with its approach as you suggested.

For the interface on the source, I am OK with using a capability.
For the destination, currently we are copying ram pages via unix
domain socket as first step, will replace to pipe later. How about
add a new URI prefix as Michael R.Hines suggested?



> Paolo
>
>> Your comments are very welcome!
>>
>> TODO:
>> - Integrate to the current implement of migration closely?
>> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
>> - benchmark/evaluation.
>>
>> Lei Li (12):
>>    migration: export MIG_STATE_xxx flags
>>    savevm: export qemu_save_device_state()
>>    rename is_active to is_block_active
>>    arch_init: introduce ram_page_save()
>>    arch_init: introduce ram_save_local()
>>    arch_init: add save_local_setup to savevm_ram_handlers
>>    savevm: introduce qemu_savevm_local()
>>    savevm: adjust is_ram check in register_savevm_live()
>>    migration-local: implementation of outgoing part
>>    migration-local: implementation of incoming part
>>    migration-local: add option to command line for local incoming
>>    hmp:add hmp_localhost_migration interface
>>
>>   Makefile.objs                 |    1 +
>>   arch_init.c                   |   110 +++++++++++++++++++
>>   block-migration.c             |    2 +-
>>   hmp-commands.hx               |   17 ++++
>>   hmp.c                         |   13 +++
>>   hmp.h                         |    1 +
>>   include/migration/migration.h |   32 +++++++
>>   include/sysemu/sysemu.h       |    1 +
>>   migration-local.c             |  228 +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   60 ++++++++++++
>>   migration.c                   |    8 --
>>   qapi-schema.json              |   14 +++
>>   qemu-options.hx               |    9 ++
>>   qmp-commands.hx               |   22 +++++
>>   savevm.c                      |   100 +++++++++++++++--
>>   vl.c                          |   14 +++
>>   16 files changed, 613 insertions(+), 19 deletions(-)
>>   create mode 100644 migration-local.c
>>
>


-- 
Lei