[Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Lei Li]

Hi,

This patch series tries to add localhost live migration support
to Qemu. It is just a draft version which is far from completing
and I am still working on, send it out to have your suggestions
and make sure it is the way should be headed. Your comments are
very appreciated!

When doing localhost live migration, the host memory will balloon
up during the period, might consume double memories for some time.
So we want to add a new live migration mechanism localhost live 
migration.

The plan is:

1) Add new command to do localhost migration.

   The qmp interface introduced like:

   { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }

2) Use different mechanism than current live migration.

   The very basic work flow like:

       qemu on the source (the source and destination are both on localhost)
              |
              V
           Stop VM
              |
              V   
          Create threads
              |
              V
       Page flipping through vmspice
              |
              V
       MADV_DONTNEED the ram pages which are already flipped
              |
              V
       Migration completes

   As stopping VM first, we expect/resume the page flipping through vmspice
   is fast enough to meet *live migration (low downtime).

Notes:
Currently the work flow is not exactly the same as description
above. For the first step, the work flow we plan to implement is:
stop VM and copy ram pages via unix domain socket, MADV_DONTNEED 
ram pages that already copied. After that, will replace to vmsplice
mechanism instead of copying pages.

TODO:
- Working on the completing of ram copy and MADV_DONTNEED part.
- Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
- benchmark/evaluation.

Lei Li (7):
  migration: export MIG_STATE_xxx flags
  savevm: export qemu_save_device_state()
  arch_init: add ram_madvise_free()
  migration-local: implementation of outgoing part
  migration-local: impelementation of incoming part
  migration-local: add option to command line for local incoming
  hmp:add hmp_localhost_migration interface


 Makefile.objs                 |    1 +
 arch_init.c                   |   13 ++++
 hmp-commands.hx               |   17 +++++
 hmp.c                         |   13 ++++
 hmp.h                         |    1 +
 include/migration/migration.h |   32 +++++++++
 include/sysemu/sysemu.h       |    1 +
 migration-local.c             |  155 +++++++++++++++++++++++++++++++++++++++++
 migration-unix.c              |   60 ++++++++++++++++
 migration.c                   |    8 --
 qapi-schema.json              |   14 ++++
 qemu-options.hx               |    9 +++
 qmp-commands.hx               |   22 ++++++
 savevm.c                      |    2 +-
 vl.c                          |   14 ++++
 15 files changed, 353 insertions(+), 9 deletions(-)
 create mode 100644 migration-local.c

[Qemu-devel] [PATCH 1/7] migration: export MIG_STATE_xxx flags

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/migration.h |    8 ++++++++
 migration.c                   |    8 --------
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index e2acec6..8866c3c 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -22,6 +22,14 @@
 #include "migration/vmstate.h"
 #include "qapi-types.h"
 
+enum {
+    MIG_STATE_ERROR,
+    MIG_STATE_SETUP,
+    MIG_STATE_CANCELLED,
+    MIG_STATE_ACTIVE,
+    MIG_STATE_COMPLETED,
+};
+
 struct MigrationParams {
     bool blk;
     bool shared;
diff --git a/migration.c b/migration.c
index 058f9e6..0921ace 100644
--- a/migration.c
+++ b/migration.c
@@ -35,14 +35,6 @@
     do { } while (0)
 #endif
 
-enum {
-    MIG_STATE_ERROR,
-    MIG_STATE_SETUP,
-    MIG_STATE_CANCELLED,
-    MIG_STATE_ACTIVE,
-    MIG_STATE_COMPLETED,
-};
-
 #define MAX_THROTTLE  (32 << 20)      /* Migration speed throttling */
 
 /* Amount of time to allocate to each "chunk" of bandwidth-throttled
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 1/7] migration: export MIG_STATE_xxx flags

[Anthony Liguori]

Lei Li <lilei@linux.vnet.ibm.com> writes:

> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>  include/migration/migration.h |    8 ++++++++
>  migration.c                   |    8 --------
>  2 files changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index e2acec6..8866c3c 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -22,6 +22,14 @@
>  #include "migration/vmstate.h"
>  #include "qapi-types.h"
>  
> +enum {
> +    MIG_STATE_ERROR,
> +    MIG_STATE_SETUP,
> +    MIG_STATE_CANCELLED,
> +    MIG_STATE_ACTIVE,
> +    MIG_STATE_COMPLETED,
> +};
> +

If these flags are being exported, it would be nice to provide comments
describing what each state corresponded to.

Regards,

Anthony Liguori

>  struct MigrationParams {
>      bool blk;
>      bool shared;
> diff --git a/migration.c b/migration.c
> index 058f9e6..0921ace 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -35,14 +35,6 @@
>      do { } while (0)
>  #endif
>  
> -enum {
> -    MIG_STATE_ERROR,
> -    MIG_STATE_SETUP,
> -    MIG_STATE_CANCELLED,
> -    MIG_STATE_ACTIVE,
> -    MIG_STATE_COMPLETED,
> -};
> -
>  #define MAX_THROTTLE  (32 << 20)      /* Migration speed throttling */
>  
>  /* Amount of time to allocate to each "chunk" of bandwidth-throttled
> -- 
> 1.7.7.6

Re: [Qemu-devel] [PATCH 1/7] migration: export MIG_STATE_xxx flags

[Lei Li]

On 06/17/2013 12:01 AM, Anthony Liguori wrote:
> Lei Li <lilei@linux.vnet.ibm.com> writes:
>
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   include/migration/migration.h |    8 ++++++++
>>   migration.c                   |    8 --------
>>   2 files changed, 8 insertions(+), 8 deletions(-)
>>
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index e2acec6..8866c3c 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -22,6 +22,14 @@
>>   #include "migration/vmstate.h"
>>   #include "qapi-types.h"
>>   
>> +enum {
>> +    MIG_STATE_ERROR,
>> +    MIG_STATE_SETUP,
>> +    MIG_STATE_CANCELLED,
>> +    MIG_STATE_ACTIVE,
>> +    MIG_STATE_COMPLETED,
>> +};
>> +
> If these flags are being exported, it would be nice to provide comments
> describing what each state corresponded to.

Got it, thanks.

>
> Regards,
>
> Anthony Liguori
>
>>   struct MigrationParams {
>>       bool blk;
>>       bool shared;
>> diff --git a/migration.c b/migration.c
>> index 058f9e6..0921ace 100644
>> --- a/migration.c
>> +++ b/migration.c
>> @@ -35,14 +35,6 @@
>>       do { } while (0)
>>   #endif
>>   
>> -enum {
>> -    MIG_STATE_ERROR,
>> -    MIG_STATE_SETUP,
>> -    MIG_STATE_CANCELLED,
>> -    MIG_STATE_ACTIVE,
>> -    MIG_STATE_COMPLETED,
>> -};
>> -
>>   #define MAX_THROTTLE  (32 << 20)      /* Migration speed throttling */
>>   
>>   /* Amount of time to allocate to each "chunk" of bandwidth-throttled
>> -- 
>> 1.7.7.6


-- 
Lei

[Qemu-devel] [PATCH 2/7] savevm: export qemu_save_device_state()

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/sysemu/sysemu.h |    1 +
 savevm.c                |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 2fb71af..5b90027 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -80,6 +80,7 @@ int qemu_savevm_state_iterate(QEMUFile *f);
 void qemu_savevm_state_complete(QEMUFile *f);
 void qemu_savevm_state_cancel(void);
 uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+int qemu_save_device_state(QEMUFile *f);
 int qemu_loadvm_state(QEMUFile *f);
 
 /* SLIRP */
diff --git a/savevm.c b/savevm.c
index 2ce439f..e3ad5da 100644
--- a/savevm.c
+++ b/savevm.c
@@ -2006,7 +2006,7 @@ static int qemu_savevm_state(QEMUFile *f)
     return ret;
 }
 
-static int qemu_save_device_state(QEMUFile *f)
+int qemu_save_device_state(QEMUFile *f)
 {
     SaveStateEntry *se;
 
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 2/7] savevm: export qemu_save_device_state()

[Anthony Liguori]

Lei Li <lilei@linux.vnet.ibm.com> writes:

> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>  include/sysemu/sysemu.h |    1 +
>  savevm.c                |    2 +-
>  2 files changed, 2 insertions(+), 1 deletions(-)
>
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index 2fb71af..5b90027 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -80,6 +80,7 @@ int qemu_savevm_state_iterate(QEMUFile *f);
>  void qemu_savevm_state_complete(QEMUFile *f);
>  void qemu_savevm_state_cancel(void);
>  uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
> +int qemu_save_device_state(QEMUFile *f);
>  int qemu_loadvm_state(QEMUFile *f);

Please add comments describing what this function does.  I know they're
absent for the rest of the functions but we have to start somewhere.

Regards,

Anthony Liguori

>  
>  /* SLIRP */
> diff --git a/savevm.c b/savevm.c
> index 2ce439f..e3ad5da 100644
> --- a/savevm.c
> +++ b/savevm.c
> @@ -2006,7 +2006,7 @@ static int qemu_savevm_state(QEMUFile *f)
>      return ret;
>  }
>  
> -static int qemu_save_device_state(QEMUFile *f)
> +int qemu_save_device_state(QEMUFile *f)
>  {
>      SaveStateEntry *se;
>  
> -- 
> 1.7.7.6

Re: [Qemu-devel] [PATCH 2/7] savevm: export qemu_save_device_state()

[Lei Li]

On 06/17/2013 12:02 AM, Anthony Liguori wrote:
> Lei Li <lilei@linux.vnet.ibm.com> writes:
>
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   include/sysemu/sysemu.h |    1 +
>>   savevm.c                |    2 +-
>>   2 files changed, 2 insertions(+), 1 deletions(-)
>>
>> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
>> index 2fb71af..5b90027 100644
>> --- a/include/sysemu/sysemu.h
>> +++ b/include/sysemu/sysemu.h
>> @@ -80,6 +80,7 @@ int qemu_savevm_state_iterate(QEMUFile *f);
>>   void qemu_savevm_state_complete(QEMUFile *f);
>>   void qemu_savevm_state_cancel(void);
>>   uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
>> +int qemu_save_device_state(QEMUFile *f);
>>   int qemu_loadvm_state(QEMUFile *f);
> Please add comments describing what this function does.  I know they're
> absent for the rest of the functions but we have to start somewhere.

OK, will do.

>
> Regards,
>
> Anthony Liguori
>
>>   
>>   /* SLIRP */
>> diff --git a/savevm.c b/savevm.c
>> index 2ce439f..e3ad5da 100644
>> --- a/savevm.c
>> +++ b/savevm.c
>> @@ -2006,7 +2006,7 @@ static int qemu_savevm_state(QEMUFile *f)
>>       return ret;
>>   }
>>   
>> -static int qemu_save_device_state(QEMUFile *f)
>> +int qemu_save_device_state(QEMUFile *f)
>>   {
>>       SaveStateEntry *se;
>>   
>> -- 
>> 1.7.7.6


-- 
Lei

[Qemu-devel] [PATCH 3/7] arch_init: add ram_madvise_free()

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 arch_init.c                   |   13 +++++++++++++
 include/migration/migration.h |    3 +++
 2 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 872020e..fc66bd2 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -524,6 +524,19 @@ uint64_t ram_bytes_total(void)
     return total;
 }
 
+void ram_madvise_free(ram_addr_t size)
+{
+    void *ram;
+    RAMBlock *block = NULL;
+
+    ram = memory_region_get_ram_ptr(block->mr);
+
+    /* XXX. Here just simplely madvise(.., MADV_DONTNEED) the whole ram
+     * pages, need more work to keep MADV_DONTNEED ram pages that
+     * already sent. */
+    qemu_madvise(ram, size, MADV_DONTNEED);
+}
+
 static void migration_end(void)
 {
     if (migration_bitmap) {
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 8866c3c..9cc5285 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -21,6 +21,7 @@
 #include "qapi/error.h"
 #include "migration/vmstate.h"
 #include "qapi-types.h"
+#include "exec/memory.h"
 
 enum {
     MIG_STATE_ERROR,
@@ -100,6 +101,8 @@ uint64_t ram_bytes_remaining(void);
 uint64_t ram_bytes_transferred(void);
 uint64_t ram_bytes_total(void);
 
+void ram_madvise_free(ram_addr_t size);
+
 extern SaveVMHandlers savevm_ram_handlers;
 
 uint64_t dup_mig_bytes_transferred(void);
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 3/7] arch_init: add ram_madvise_free()

[Anthony Liguori]

Lei Li <lilei@linux.vnet.ibm.com> writes:

> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>  arch_init.c                   |   13 +++++++++++++
>  include/migration/migration.h |    3 +++
>  2 files changed, 16 insertions(+), 0 deletions(-)
>
> diff --git a/arch_init.c b/arch_init.c
> index 872020e..fc66bd2 100644
> --- a/arch_init.c
> +++ b/arch_init.c
> @@ -524,6 +524,19 @@ uint64_t ram_bytes_total(void)
>      return total;
>  }
>  
> +void ram_madvise_free(ram_addr_t size)
> +{
> +    void *ram;
> +    RAMBlock *block = NULL;
> +
> +    ram = memory_region_get_ram_ptr(block->mr);
> +
> +    /* XXX. Here just simplely madvise(.., MADV_DONTNEED) the whole ram
> +     * pages, need more work to keep MADV_DONTNEED ram pages that
> +     * already sent. */
> +    qemu_madvise(ram, size, MADV_DONTNEED);
> +}
> +

I don't think this is right at all.  There's no guarantee we have a
single linear mapping of all ram.

I think you need something a bit more clever than this.

>  static void migration_end(void)
>  {
>      if (migration_bitmap) {
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index 8866c3c..9cc5285 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -21,6 +21,7 @@
>  #include "qapi/error.h"
>  #include "migration/vmstate.h"
>  #include "qapi-types.h"
> +#include "exec/memory.h"
>  
>  enum {
>      MIG_STATE_ERROR,
> @@ -100,6 +101,8 @@ uint64_t ram_bytes_remaining(void);
>  uint64_t ram_bytes_transferred(void);
>  uint64_t ram_bytes_total(void);
>  
> +void ram_madvise_free(ram_addr_t size);
> +

If you introduce new interfaces, please include documentation in the
header.

Regards,

Anthony Liguori

>  extern SaveVMHandlers savevm_ram_handlers;
>  
>  uint64_t dup_mig_bytes_transferred(void);
> -- 
> 1.7.7.6

Re: [Qemu-devel] [PATCH 3/7] arch_init: add ram_madvise_free()

[Lei Li]

On 06/17/2013 12:04 AM, Anthony Liguori wrote:
> Lei Li <lilei@linux.vnet.ibm.com> writes:
>
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   arch_init.c                   |   13 +++++++++++++
>>   include/migration/migration.h |    3 +++
>>   2 files changed, 16 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch_init.c b/arch_init.c
>> index 872020e..fc66bd2 100644
>> --- a/arch_init.c
>> +++ b/arch_init.c
>> @@ -524,6 +524,19 @@ uint64_t ram_bytes_total(void)
>>       return total;
>>   }
>>   
>> +void ram_madvise_free(ram_addr_t size)
>> +{
>> +    void *ram;
>> +    RAMBlock *block = NULL;
>> +
>> +    ram = memory_region_get_ram_ptr(block->mr);
>> +
>> +    /* XXX. Here just simplely madvise(.., MADV_DONTNEED) the whole ram
>> +     * pages, need more work to keep MADV_DONTNEED ram pages that
>> +     * already sent. */
>> +    qemu_madvise(ram, size, MADV_DONTNEED);
>> +}
>> +
> I don't think this is right at all.  There's no guarantee we have a
> single linear mapping of all ram.

Ah, thanks for yourclarification!

>
> I think you need something a bit more clever than this.

Yes, this need more work.

>
>>   static void migration_end(void)
>>   {
>>       if (migration_bitmap) {
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index 8866c3c..9cc5285 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -21,6 +21,7 @@
>>   #include "qapi/error.h"
>>   #include "migration/vmstate.h"
>>   #include "qapi-types.h"
>> +#include "exec/memory.h"
>>   
>>   enum {
>>       MIG_STATE_ERROR,
>> @@ -100,6 +101,8 @@ uint64_t ram_bytes_remaining(void);
>>   uint64_t ram_bytes_transferred(void);
>>   uint64_t ram_bytes_total(void);
>>   
>> +void ram_madvise_free(ram_addr_t size);
>> +
> If you introduce new interfaces, please include documentation in the
> header.

Good to know, thanks!

> Regards,
>
> Anthony Liguori
>
>>   extern SaveVMHandlers savevm_ram_handlers;
>>   
>>   uint64_t dup_mig_bytes_transferred(void);
>> -- 
>> 1.7.7.6


-- 
Lei

Re: [Qemu-devel] [PATCH 3/7] arch_init: add ram_madvise_free()

[Michael R. Hines]

On 06/16/2013 12:04 PM, Anthony Liguori wrote:
> Lei Li <lilei@linux.vnet.ibm.com> writes:
>
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   arch_init.c                   |   13 +++++++++++++
>>   include/migration/migration.h |    3 +++
>>   2 files changed, 16 insertions(+), 0 deletions(-)
>>
>> diff --git a/arch_init.c b/arch_init.c
>> index 872020e..fc66bd2 100644
>> --- a/arch_init.c
>> +++ b/arch_init.c
>> @@ -524,6 +524,19 @@ uint64_t ram_bytes_total(void)
>>       return total;
>>   }
>>   
>> +void ram_madvise_free(ram_addr_t size)
>> +{
>> +    void *ram;
>> +    RAMBlock *block = NULL;
>> +
>> +    ram = memory_region_get_ram_ptr(block->mr);
>> +
>> +    /* XXX. Here just simplely madvise(.., MADV_DONTNEED) the whole ram
>> +     * pages, need more work to keep MADV_DONTNEED ram pages that
>> +     * already sent. */
>> +    qemu_madvise(ram, size, MADV_DONTNEED);
>> +}
>> +
> I don't think this is right at all.  There's no guarantee we have a
> single linear mapping of all ram.
>
> I think you need something a bit more clever than this.
>
>>   static void migration_end(void)
>>   {
>>       if (migration_bitmap) {
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index 8866c3c..9cc5285 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -21,6 +21,7 @@
>>   #include "qapi/error.h"
>>   #include "migration/vmstate.h"
>>   #include "qapi-types.h"
>> +#include "exec/memory.h"
>>   
>>   enum {
>>       MIG_STATE_ERROR,
>> @@ -100,6 +101,8 @@ uint64_t ram_bytes_remaining(void);
>>   uint64_t ram_bytes_transferred(void);
>>   uint64_t ram_bytes_total(void);
>>   
>> +void ram_madvise_free(ram_addr_t size);
>> +
> If you introduce new interfaces, please include documentation in the
> header.
>
> Regards,
>
> Anthony Liguori
>
>>   extern SaveVMHandlers savevm_ram_handlers;
>>   
>>   uint64_t dup_mig_bytes_transferred(void);
>> -- 
>> 1.7.7.6
>

We have a new function in master...... ram_handle_compressed()  (was 
exported for RDMA).

Perhaps this could be used.....

- Michael

[Qemu-devel] [PATCH 4/7] migration-local: implementation of outgoing part

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 Makefile.objs                 |    1 +
 include/migration/migration.h |   15 +++++
 migration-local.c             |  116 +++++++++++++++++++++++++++++++++++++++++
 migration-unix.c              |   13 +++++
 qapi-schema.json              |   14 +++++
 qmp-commands.hx               |   22 ++++++++
 6 files changed, 181 insertions(+), 0 deletions(-)
 create mode 100644 migration-local.c

diff --git a/Makefile.objs b/Makefile.objs
index 5b288ba..2a3d9a5 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -53,6 +53,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/
 common-obj-y += migration.o migration-tcp.o
 common-obj-y += qemu-char.o #aio.o
 common-obj-y += block-migration.o
+common-obj-y += migration-local.o
 common-obj-y += page_cache.o xbzrle.o
 
 common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 9cc5285..12c75cf 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -58,6 +58,17 @@ struct MigrationState
     int64_t xbzrle_cache_size;
 };
 
+
+typedef struct LocalMigState LocalMigState;
+
+struct LocalMigState
+{
+    int fd;
+    int state;
+    QEMUFile *file;
+    QemuThread thread;
+};
+
 void process_incoming_migration(QEMUFile *f);
 
 void qemu_start_incoming_migration(const char *uri, Error **errp);
@@ -80,6 +91,8 @@ void unix_start_incoming_migration(const char *path, Error **errp);
 
 void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
 
+void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp);
+
 void fd_start_incoming_migration(const char *path, Error **errp);
 
 void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp);
@@ -90,6 +103,8 @@ void migrate_fd_connect(MigrationState *s);
 
 int migrate_fd_close(MigrationState *s);
 
+void migrate_fd_connect_local(LocalMigState *s);
+
 void add_migration_state_change_notifier(Notifier *notify);
 void remove_migration_state_change_notifier(Notifier *notify);
 bool migration_is_active(MigrationState *);
diff --git a/migration-local.c b/migration-local.c
new file mode 100644
index 0000000..a97dd3a
--- /dev/null
+++ b/migration-local.c
@@ -0,0 +1,116 @@
+/*
+ * QEMU localhost live migration
+ *
+ * Copyright IBM, Corp. 2013
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions are licensed under the terms of the GNU GPL, version 2
+ * or (at your option) any later version.
+ */
+
+#include "qemu-common.h"
+#include "migration/migration.h"
+#include "monitor/monitor.h"
+#include "migration/qemu-file.h"
+#include "sysemu/sysemu.h"
+#include "block/block.h"
+#include "qemu/sockets.h"
+#include "migration/block.h"
+#include "qemu/thread.h"
+#include "qmp-commands.h"
+#include "exec/memory.h"
+#include "trace.h"
+#include "qemu/osdep.h"
+
+//#define DEBUG_MIGRATION_LOCAL
+
+#ifdef DEBUG_MIGRATION_LOCAL
+#define DPRINTF(fmt, ...) \
+    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+    do { } while (0)
+#endif
+
+
+/************************************************************************
+ * Outgoing part
+ */
+
+static void *migration_local_thread(void *opaque)
+{
+    LocalMigState *s = opaque;
+    int ret;
+    ram_addr_t size;
+
+    DPRINTF("Beginning savevm\n");
+
+    /* This will be replaced by new function qemu_save_local_state(). */
+    ret = qemu_save_device_state(s->file);
+    qemu_fclose(s->file);
+
+    if (ret < 0) {
+        s->state = MIG_STATE_ERROR;
+        return NULL;
+    } else {
+        /* XXX: The logic will be changed, need more work here. */
+        s->state = MIG_STATE_COMPLETED;
+
+        size = ram_bytes_total();
+        ram_madvise_free(size);
+    }
+
+    return NULL;
+}
+
+void migrate_fd_connect_local(LocalMigState *s)
+{
+    s->state = MIG_STATE_ACTIVE;
+    trace_migrate_set_state(MIG_STATE_ACTIVE);
+
+    qemu_thread_create(&s->thread, migration_local_thread, s,
+                       QEMU_THREAD_JOINABLE);
+}
+
+void qmp_localhost_migrate(const char *uri, Error **errp)
+{
+    const char *path;
+    Error *local_err = NULL;
+    int is_vm_running;
+    LocalMigState *s;
+
+    is_vm_running = runstate_is_running();
+
+    /* Stop the VM first */
+    if (is_vm_running) {
+        vm_stop(RUN_STATE_SAVE_VM);
+    }
+
+    bdrv_flush_all();
+
+    s = g_malloc0(sizeof(LocalMigState));
+
+    /* Start outgoing migration by unix socket. */
+    if (strstart(uri, "unix:", &path)) {
+        /* XXX. Creat a new unix_start_outgoing_migration_* is not necessary,
+         * just for the first step. This will be replaced by vmsplice
+         * mechanism. */
+        unix_start_local_outgoing_migration(s, path, &local_err);
+    } else {
+        error_set(errp, QERR_INVALID_PARAMETER_VALUE, "uri", "a valid migration protocol");
+        goto fail;
+    }
+
+    if (local_err) {
+        s->state = MIG_STATE_ERROR;
+        error_propagate(errp, local_err);
+        goto fail;
+    }
+
+fail:
+    if (!is_vm_running) {
+        vm_start();
+    }
+}
diff --git a/migration-unix.c b/migration-unix.c
index 94b7022..a21262a 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -49,6 +49,19 @@ void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **
     unix_nonblocking_connect(path, unix_wait_for_connect, s, errp);
 }
 
+void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp)
+{
+    s->fd = unix_connect(path, errp);
+    if (s->fd < 0) {
+        s->file = NULL;
+        /* There should be a fd_error_set function */
+        s->state = MIG_STATE_ERROR;
+    } else {
+        s->file = qemu_fopen_socket(s->fd, "wb");
+        migrate_fd_connect_local(s);
+    }
+}
+
 static void unix_accept_incoming_migration(void *opaque)
 {
     struct sockaddr_un addr;
diff --git a/qapi-schema.json b/qapi-schema.json
index 5ad6894..2890f70 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -648,6 +648,20 @@
 { 'command': 'query-migrate-capabilities', 'returns':   ['MigrationCapabilityStatus']}
 
 ##
+# @localhost-migrate
+#
+# Migrates the current running guest to another localhost VM.
+#
+# @uri: the Uniform Resource Identifier of the destination VM
+#
+# Returns: nothing on success
+#
+# Since: 1.7
+##
+{ 'command': 'localhost-migrate',
+  'data': {'uri': 'str'} }
+
+##
 # @MouseInfo:
 #
 # Information about a mouse device.
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 8cea5e5..bf6d042 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -787,6 +787,28 @@ Example:
 EQMP
 
     {
+        .name      = "localhost-migrate",
+        .args_type = "uri:s",
+        .mhandler.cmd_new = qmp_marshal_input_localhost_migrate,
+    },
+
+SQMP
+localhost-migrate
+
+Migrate VM in localhost.
+
+Arguments:
+
+- "uri": Destination URI (json-string)
+
+Example:
+
+-> { "execute": "localhost-migrate", "arguments": { "uri": "/tmp/foo" }
+<- { "return": {} }
+
+EQMP
+
+    {
         .name       = "dump-guest-memory",
         .args_type  = "paging:b,protocol:s,begin:i?,end:i?",
         .params     = "-p protocol [begin] [length]",
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 4/7] migration-local: implementation of outgoing part

[Anthony Liguori]

Lei Li <lilei@linux.vnet.ibm.com> writes:

> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
> ---
>  Makefile.objs                 |    1 +
>  include/migration/migration.h |   15 +++++
>  migration-local.c             |  116 +++++++++++++++++++++++++++++++++++++++++
>  migration-unix.c              |   13 +++++
>  qapi-schema.json              |   14 +++++
>  qmp-commands.hx               |   22 ++++++++
>  6 files changed, 181 insertions(+), 0 deletions(-)
>  create mode 100644 migration-local.c
>
> diff --git a/Makefile.objs b/Makefile.objs
> index 5b288ba..2a3d9a5 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -53,6 +53,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/
>  common-obj-y += migration.o migration-tcp.o
>  common-obj-y += qemu-char.o #aio.o
>  common-obj-y += block-migration.o
> +common-obj-y += migration-local.o
>  common-obj-y += page_cache.o xbzrle.o
>  
>  common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index 9cc5285..12c75cf 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -58,6 +58,17 @@ struct MigrationState
>      int64_t xbzrle_cache_size;
>  };
>  
> +
> +typedef struct LocalMigState LocalMigState;
> +
> +struct LocalMigState
> +{
> +    int fd;
> +    int state;
> +    QEMUFile *file;
> +    QemuThread thread;
> +};
> +
>  void process_incoming_migration(QEMUFile *f);
>  
>  void qemu_start_incoming_migration(const char *uri, Error **errp);
> @@ -80,6 +91,8 @@ void unix_start_incoming_migration(const char *path, Error **errp);
>  
>  void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
>  
> +void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp);
> +
>  void fd_start_incoming_migration(const char *path, Error **errp);
>  
>  void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp);
> @@ -90,6 +103,8 @@ void migrate_fd_connect(MigrationState *s);
>  
>  int migrate_fd_close(MigrationState *s);
>  
> +void migrate_fd_connect_local(LocalMigState *s);
> +
>  void add_migration_state_change_notifier(Notifier *notify);
>  void remove_migration_state_change_notifier(Notifier *notify);
>  bool migration_is_active(MigrationState *);
> diff --git a/migration-local.c b/migration-local.c
> new file mode 100644
> index 0000000..a97dd3a
> --- /dev/null
> +++ b/migration-local.c
> @@ -0,0 +1,116 @@
> +/*
> + * QEMU localhost live migration
> + *
> + * Copyright IBM, Corp. 2013
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2.  See
> + * the COPYING file in the top-level directory.
> + *
> + * Contributions are licensed under the terms of the GNU GPL, version 2
> + * or (at your option) any later version.

You can just add the "or later" clause to the first paragraph.

> + */
> +
> +#include "qemu-common.h"
> +#include "migration/migration.h"
> +#include "monitor/monitor.h"
> +#include "migration/qemu-file.h"
> +#include "sysemu/sysemu.h"
> +#include "block/block.h"
> +#include "qemu/sockets.h"
> +#include "migration/block.h"
> +#include "qemu/thread.h"
> +#include "qmp-commands.h"
> +#include "exec/memory.h"
> +#include "trace.h"
> +#include "qemu/osdep.h"
> +
> +//#define DEBUG_MIGRATION_LOCAL
> +
> +#ifdef DEBUG_MIGRATION_LOCAL
> +#define DPRINTF(fmt, ...) \
> +    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
> +#else
> +#define DPRINTF(fmt, ...) \
> +    do { } while (0)
> +#endif
> +
> +
> +/************************************************************************
> + * Outgoing part
> + */
> +
> +static void *migration_local_thread(void *opaque)
> +{
> +    LocalMigState *s = opaque;
> +    int ret;
> +    ram_addr_t size;
> +
> +    DPRINTF("Beginning savevm\n");
> +
> +    /* This will be replaced by new function qemu_save_local_state(). */
> +    ret = qemu_save_device_state(s->file);
> +    qemu_fclose(s->file);
> +
> +    if (ret < 0) {
> +        s->state = MIG_STATE_ERROR;
> +        return NULL;
> +    } else {
> +        /* XXX: The logic will be changed, need more work here. */
> +        s->state = MIG_STATE_COMPLETED;
> +
> +        size = ram_bytes_total();
> +        ram_madvise_free(size);
> +    }
> +
> +    return NULL;
> +}
> +
> +void migrate_fd_connect_local(LocalMigState *s)
> +{
> +    s->state = MIG_STATE_ACTIVE;
> +    trace_migrate_set_state(MIG_STATE_ACTIVE);
> +
> +    qemu_thread_create(&s->thread, migration_local_thread, s,
> +                       QEMU_THREAD_JOINABLE);
> +}
> +
> +void qmp_localhost_migrate(const char *uri, Error **errp)
> +{
> +    const char *path;
> +    Error *local_err = NULL;
> +    int is_vm_running;
> +    LocalMigState *s;
> +
> +    is_vm_running = runstate_is_running();
> +
> +    /* Stop the VM first */
> +    if (is_vm_running) {
> +        vm_stop(RUN_STATE_SAVE_VM);
> +    }
> +
> +    bdrv_flush_all();
> +
> +    s = g_malloc0(sizeof(LocalMigState));
> +
> +    /* Start outgoing migration by unix socket. */
> +    if (strstart(uri, "unix:", &path)) {
> +        /* XXX. Creat a new unix_start_outgoing_migration_* is not necessary,
> +         * just for the first step. This will be replaced by vmsplice
> +         * mechanism. */
> +        unix_start_local_outgoing_migration(s, path, &local_err);
> +    } else {
> +        error_set(errp, QERR_INVALID_PARAMETER_VALUE, "uri", "a valid migration protocol");
> +        goto fail;
> +    }
> +
> +    if (local_err) {
> +        s->state = MIG_STATE_ERROR;
> +        error_propagate(errp, local_err);
> +        goto fail;
> +    }
> +
> +fail:
> +    if (!is_vm_running) {
> +        vm_start();
> +    }
> +}
> diff --git a/migration-unix.c b/migration-unix.c
> index 94b7022..a21262a 100644
> --- a/migration-unix.c
> +++ b/migration-unix.c
> @@ -49,6 +49,19 @@ void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **
>      unix_nonblocking_connect(path, unix_wait_for_connect, s, errp);
>  }
>  
> +void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp)
> +{
> +    s->fd = unix_connect(path, errp);
> +    if (s->fd < 0) {
> +        s->file = NULL;
> +        /* There should be a fd_error_set function */
> +        s->state = MIG_STATE_ERROR;
> +    } else {
> +        s->file = qemu_fopen_socket(s->fd, "wb");
> +        migrate_fd_connect_local(s);
> +    }
> +}
> +

Skipping most of the above as it's a work-in-progress.

>  static void unix_accept_incoming_migration(void *opaque)
>  {
>      struct sockaddr_un addr;
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 5ad6894..2890f70 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -648,6 +648,20 @@
>  { 'command': 'query-migrate-capabilities', 'returns':   ['MigrationCapabilityStatus']}
>  
>  ##
> +# @localhost-migrate
> +#
> +# Migrates the current running guest to another localhost VM.
> +#
> +# @uri: the Uniform Resource Identifier of the destination VM
> +#
> +# Returns: nothing on success
> +#
> +# Since: 1.7
> +##
> +{ 'command': 'localhost-migrate',
> +  'data': {'uri': 'str'} }

I suspect we should just do this as a flag to the existing migration
command.  Perhaps we could even just use a capability.

Regards,

Anthony Liguori

> +
> +##
>  # @MouseInfo:
>  #
>  # Information about a mouse device.
> diff --git a/qmp-commands.hx b/qmp-commands.hx
> index 8cea5e5..bf6d042 100644
> --- a/qmp-commands.hx
> +++ b/qmp-commands.hx
> @@ -787,6 +787,28 @@ Example:
>  EQMP
>  
>      {
> +        .name      = "localhost-migrate",
> +        .args_type = "uri:s",
> +        .mhandler.cmd_new = qmp_marshal_input_localhost_migrate,
> +    },
> +
> +SQMP
> +localhost-migrate
> +
> +Migrate VM in localhost.
> +
> +Arguments:
> +
> +- "uri": Destination URI (json-string)
> +
> +Example:
> +
> +-> { "execute": "localhost-migrate", "arguments": { "uri": "/tmp/foo" }
> +<- { "return": {} }
> +
> +EQMP
> +
> +    {
>          .name       = "dump-guest-memory",
>          .args_type  = "paging:b,protocol:s,begin:i?,end:i?",
>          .params     = "-p protocol [begin] [length]",
> -- 
> 1.7.7.6

Re: [Qemu-devel] [PATCH 4/7] migration-local: implementation of outgoing part

[Lei Li]

On 06/17/2013 12:07 AM, Anthony Liguori wrote:
> Lei Li <lilei@linux.vnet.ibm.com> writes:
>
>> Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
>> ---
>>   Makefile.objs                 |    1 +
>>   include/migration/migration.h |   15 +++++
>>   migration-local.c             |  116 +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   13 +++++
>>   qapi-schema.json              |   14 +++++
>>   qmp-commands.hx               |   22 ++++++++
>>   6 files changed, 181 insertions(+), 0 deletions(-)
>>   create mode 100644 migration-local.c
>>
>> diff --git a/Makefile.objs b/Makefile.objs
>> index 5b288ba..2a3d9a5 100644
>> --- a/Makefile.objs
>> +++ b/Makefile.objs
>> @@ -53,6 +53,7 @@ common-obj-$(CONFIG_LINUX) += fsdev/
>>   common-obj-y += migration.o migration-tcp.o
>>   common-obj-y += qemu-char.o #aio.o
>>   common-obj-y += block-migration.o
>> +common-obj-y += migration-local.o
>>   common-obj-y += page_cache.o xbzrle.o
>>   
>>   common-obj-$(CONFIG_POSIX) += migration-exec.o migration-unix.o migration-fd.o
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index 9cc5285..12c75cf 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -58,6 +58,17 @@ struct MigrationState
>>       int64_t xbzrle_cache_size;
>>   };
>>   
>> +
>> +typedef struct LocalMigState LocalMigState;
>> +
>> +struct LocalMigState
>> +{
>> +    int fd;
>> +    int state;
>> +    QEMUFile *file;
>> +    QemuThread thread;
>> +};
>> +
>>   void process_incoming_migration(QEMUFile *f);
>>   
>>   void qemu_start_incoming_migration(const char *uri, Error **errp);
>> @@ -80,6 +91,8 @@ void unix_start_incoming_migration(const char *path, Error **errp);
>>   
>>   void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
>>   
>> +void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp);
>> +
>>   void fd_start_incoming_migration(const char *path, Error **errp);
>>   
>>   void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **errp);
>> @@ -90,6 +103,8 @@ void migrate_fd_connect(MigrationState *s);
>>   
>>   int migrate_fd_close(MigrationState *s);
>>   
>> +void migrate_fd_connect_local(LocalMigState *s);
>> +
>>   void add_migration_state_change_notifier(Notifier *notify);
>>   void remove_migration_state_change_notifier(Notifier *notify);
>>   bool migration_is_active(MigrationState *);
>> diff --git a/migration-local.c b/migration-local.c
>> new file mode 100644
>> index 0000000..a97dd3a
>> --- /dev/null
>> +++ b/migration-local.c
>> @@ -0,0 +1,116 @@
>> +/*
>> + * QEMU localhost live migration
>> + *
>> + * Copyright IBM, Corp. 2013
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2.  See
>> + * the COPYING file in the top-level directory.
>> + *
>> + * Contributions are licensed under the terms of the GNU GPL, version 2
>> + * or (at your option) any later version.
> You can just add the "or later" clause to the first paragraph.

Sure.

>
>> + */
>> +
>> +#include "qemu-common.h"
>> +#include "migration/migration.h"
>> +#include "monitor/monitor.h"
>> +#include "migration/qemu-file.h"
>> +#include "sysemu/sysemu.h"
>> +#include "block/block.h"
>> +#include "qemu/sockets.h"
>> +#include "migration/block.h"
>> +#include "qemu/thread.h"
>> +#include "qmp-commands.h"
>> +#include "exec/memory.h"
>> +#include "trace.h"
>> +#include "qemu/osdep.h"
>> +
>> +//#define DEBUG_MIGRATION_LOCAL
>> +
>> +#ifdef DEBUG_MIGRATION_LOCAL
>> +#define DPRINTF(fmt, ...) \
>> +    do { printf("migration-local: " fmt, ## __VA_ARGS__); } while (0)
>> +#else
>> +#define DPRINTF(fmt, ...) \
>> +    do { } while (0)
>> +#endif
>> +
>> +
>> +/************************************************************************
>> + * Outgoing part
>> + */
>> +
>> +static void *migration_local_thread(void *opaque)
>> +{
>> +    LocalMigState *s = opaque;
>> +    int ret;
>> +    ram_addr_t size;
>> +
>> +    DPRINTF("Beginning savevm\n");
>> +
>> +    /* This will be replaced by new function qemu_save_local_state(). */
>> +    ret = qemu_save_device_state(s->file);
>> +    qemu_fclose(s->file);
>> +
>> +    if (ret < 0) {
>> +        s->state = MIG_STATE_ERROR;
>> +        return NULL;
>> +    } else {
>> +        /* XXX: The logic will be changed, need more work here. */
>> +        s->state = MIG_STATE_COMPLETED;
>> +
>> +        size = ram_bytes_total();
>> +        ram_madvise_free(size);
>> +    }
>> +
>> +    return NULL;
>> +}
>> +
>> +void migrate_fd_connect_local(LocalMigState *s)
>> +{
>> +    s->state = MIG_STATE_ACTIVE;
>> +    trace_migrate_set_state(MIG_STATE_ACTIVE);
>> +
>> +    qemu_thread_create(&s->thread, migration_local_thread, s,
>> +                       QEMU_THREAD_JOINABLE);
>> +}
>> +
>> +void qmp_localhost_migrate(const char *uri, Error **errp)
>> +{
>> +    const char *path;
>> +    Error *local_err = NULL;
>> +    int is_vm_running;
>> +    LocalMigState *s;
>> +
>> +    is_vm_running = runstate_is_running();
>> +
>> +    /* Stop the VM first */
>> +    if (is_vm_running) {
>> +        vm_stop(RUN_STATE_SAVE_VM);
>> +    }
>> +
>> +    bdrv_flush_all();
>> +
>> +    s = g_malloc0(sizeof(LocalMigState));
>> +
>> +    /* Start outgoing migration by unix socket. */
>> +    if (strstart(uri, "unix:", &path)) {
>> +        /* XXX. Creat a new unix_start_outgoing_migration_* is not necessary,
>> +         * just for the first step. This will be replaced by vmsplice
>> +         * mechanism. */
>> +        unix_start_local_outgoing_migration(s, path, &local_err);
>> +    } else {
>> +        error_set(errp, QERR_INVALID_PARAMETER_VALUE, "uri", "a valid migration protocol");
>> +        goto fail;
>> +    }
>> +
>> +    if (local_err) {
>> +        s->state = MIG_STATE_ERROR;
>> +        error_propagate(errp, local_err);
>> +        goto fail;
>> +    }
>> +
>> +fail:
>> +    if (!is_vm_running) {
>> +        vm_start();
>> +    }
>> +}
>> diff --git a/migration-unix.c b/migration-unix.c
>> index 94b7022..a21262a 100644
>> --- a/migration-unix.c
>> +++ b/migration-unix.c
>> @@ -49,6 +49,19 @@ void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **
>>       unix_nonblocking_connect(path, unix_wait_for_connect, s, errp);
>>   }
>>   
>> +void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp)
>> +{
>> +    s->fd = unix_connect(path, errp);
>> +    if (s->fd < 0) {
>> +        s->file = NULL;
>> +        /* There should be a fd_error_set function */
>> +        s->state = MIG_STATE_ERROR;
>> +    } else {
>> +        s->file = qemu_fopen_socket(s->fd, "wb");
>> +        migrate_fd_connect_local(s);
>> +    }
>> +}
>> +
> Skipping most of the above as it's a work-in-progress.
>
>>   static void unix_accept_incoming_migration(void *opaque)
>>   {
>>       struct sockaddr_un addr;
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index 5ad6894..2890f70 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -648,6 +648,20 @@
>>   { 'command': 'query-migrate-capabilities', 'returns':   ['MigrationCapabilityStatus']}
>>   
>>   ##
>> +# @localhost-migrate
>> +#
>> +# Migrates the current running guest to another localhost VM.
>> +#
>> +# @uri: the Uniform Resource Identifier of the destination VM
>> +#
>> +# Returns: nothing on success
>> +#
>> +# Since: 1.7
>> +##
>> +{ 'command': 'localhost-migrate',
>> +  'data': {'uri': 'str'} }
> I suspect we should just do this as a flag to the existing migration
> command.  Perhaps we could even just use a capability.

Yeah, I had thought about this by adding a flag to the existing migration command like:

migrate -l for localhost migration.

This is just for a easy start.

>
> Regards,
>
> Anthony Liguori
>
>> +
>> +##
>>   # @MouseInfo:
>>   #
>>   # Information about a mouse device.
>> diff --git a/qmp-commands.hx b/qmp-commands.hx
>> index 8cea5e5..bf6d042 100644
>> --- a/qmp-commands.hx
>> +++ b/qmp-commands.hx
>> @@ -787,6 +787,28 @@ Example:
>>   EQMP
>>   
>>       {
>> +        .name      = "localhost-migrate",
>> +        .args_type = "uri:s",
>> +        .mhandler.cmd_new = qmp_marshal_input_localhost_migrate,
>> +    },
>> +
>> +SQMP
>> +localhost-migrate
>> +
>> +Migrate VM in localhost.
>> +
>> +Arguments:
>> +
>> +- "uri": Destination URI (json-string)
>> +
>> +Example:
>> +
>> +-> { "execute": "localhost-migrate", "arguments": { "uri": "/tmp/foo" }
>> +<- { "return": {} }
>> +
>> +EQMP
>> +
>> +    {
>>           .name       = "dump-guest-memory",
>>           .args_type  = "paging:b,protocol:s,begin:i?,end:i?",
>>           .params     = "-p protocol [begin] [length]",
>> -- 
>> 1.7.7.6


-- 
Lei

[Qemu-devel] [PATCH 5/7] migration-local: implementation of incoming part

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 include/migration/migration.h |    6 +++++
 migration-local.c             |   39 ++++++++++++++++++++++++++++++++++
 migration-unix.c              |   47 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 0 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 12c75cf..e0693c2 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -73,6 +73,10 @@ void process_incoming_migration(QEMUFile *f);
 
 void qemu_start_incoming_migration(const char *uri, Error **errp);
 
+void start_local_incoming_migration(QEMUFile *f);
+
+void qemu_start_local_incoming_migration(const char *uri, Error **errp);
+
 uint64_t migrate_max_downtime(void);
 
 void do_info_migrate_print(Monitor *mon, const QObject *data);
@@ -89,6 +93,8 @@ void tcp_start_outgoing_migration(MigrationState *s, const char *host_port, Erro
 
 void unix_start_incoming_migration(const char *path, Error **errp);
 
+void unix_start_local_incoming_migration(const char *path, Error **errp);
+
 void unix_start_outgoing_migration(MigrationState *s, const char *path, Error **errp);
 
 void unix_start_local_outgoing_migration(LocalMigState *s, const char *path, Error **errp);
diff --git a/migration-local.c b/migration-local.c
index a97dd3a..61da76f 100644
--- a/migration-local.c
+++ b/migration-local.c
@@ -114,3 +114,42 @@ fail:
         vm_start();
     }
 }
+
+/**********************************************************************
+ * Incoming part
+ */
+
+void qemu_start_local_incoming_migration(const char *uri, Error **errp)
+{
+    const char *p;
+
+    if (strstart(uri, "unix:", &p)) {
+        unix_start_local_incoming_migration(p, errp);
+    } else {
+        error_setg(errp, "unknown migration protocol: %s", uri);
+    }
+}
+
+void start_local_incoming_migration(QEMUFile *f)
+{
+    int ret;
+
+    ret = qemu_loadvm_state(f);
+    if (ret < 0) {
+        fprintf(stderr, "load of migration failed\n");
+        exit(EXIT_FAILURE);
+    }
+    qemu_announce_self();
+
+    DPRINTF("successfully loaded vm state\n");
+
+    bdrv_clear_incoming_migration_all();
+    /* Make sure all file formats flush their mutable metadata */
+    bdrv_invalidate_cache_all();
+
+    if (autostart) {
+        vm_start();
+    } else {
+        runstate_set(RUN_STATE_PAUSED);
+    }
+}
diff --git a/migration-unix.c b/migration-unix.c
index a21262a..ba7c0d0 100644
--- a/migration-unix.c
+++ b/migration-unix.c
@@ -108,3 +108,50 @@ void unix_start_incoming_migration(const char *path, Error **errp)
     qemu_set_fd_handler2(s, NULL, unix_accept_incoming_migration, NULL,
                          (void *)(intptr_t)s);
 }
+
+static void unix_accept_local_incoming_migration(void *opaque)
+{
+    struct sockaddr_un addr;
+    socklen_t addrlen = sizeof(addr);
+    int s = (intptr_t)opaque;
+    QEMUFile *f;
+    int c;
+
+    do {
+        c = qemu_accept(s, (struct sockaddr *)&addr, &addrlen);
+    } while (c == -1 && errno == EINTR);
+    qemu_set_fd_handler2(s, NULL, NULL, NULL, NULL);
+    close(s);
+
+    DPRINTF("accepted migration\n");
+
+    if (c == -1) {
+        fprintf(stderr, "could not accept migration connection\n");
+        goto out;
+    }
+
+    f = qemu_fopen_socket(c, "rb");
+    if (f == NULL) {
+        fprintf(stderr, "could not qemu_fopen socket\n");
+        goto out;
+    }
+
+    start_local_incoming_migration(f);
+    return;
+
+out:
+    close(c);
+}
+
+void unix_start_local_incoming_migration(const char *path, Error **errp)
+{
+    int ret;
+
+    ret = unix_listen(path, NULL, 0, errp);
+    if (ret < 0) {
+        return;
+    }
+
+    qemu_set_fd_handler2(ret, NULL, unix_accept_local_incoming_migration, NULL,
+                         (void *)(intptr_t)ret);
+}
-- 
1.7.7.6

[Qemu-devel] [PATCH 6/7] migration-local: add option to command line for -incoming-local

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 qemu-options.hx |    9 +++++++++
 vl.c            |   14 ++++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/qemu-options.hx b/qemu-options.hx
index bf94862..bb1a88a 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -2913,6 +2913,15 @@ STEXI
 Prepare for incoming migration, listen on @var{port}.
 ETEXI
 
+DEF("incoming-local", HAS_ARG, QEMU_OPTION_incoming_local, \
+    "-incoming-local p prepare for localhost incoming migration, listen on domain unix socket p\n",
+    QEMU_ARCH_ALL)
+STEXI
+@item -incoming-local @var{port}
+@findex -incoming-local
+Prepare for localhost incoming migration, listen on @var{port}
+ETEXI
+
 DEF("nodefaults", 0, QEMU_OPTION_nodefaults, \
     "-nodefaults     don't create default devices\n", QEMU_ARCH_ALL)
 STEXI
diff --git a/vl.c b/vl.c
index 9f8fd6e..496a264 100644
--- a/vl.c
+++ b/vl.c
@@ -2856,6 +2856,7 @@ int main(int argc, char **argv, char **envp)
     const char *vga_model = "none";
     const char *pid_file = NULL;
     const char *incoming = NULL;
+    const char *incoming_local = NULL;
 #ifdef CONFIG_VNC
     int show_vnc_port = 0;
 #endif
@@ -3755,6 +3756,10 @@ int main(int argc, char **argv, char **envp)
                 incoming = optarg;
                 runstate_set(RUN_STATE_INMIGRATE);
                 break;
+            case QEMU_OPTION_incoming_local:
+                incoming_local = optarg;
+                runstate_set(RUN_STATE_INMIGRATE);
+                break;
             case QEMU_OPTION_nodefaults:
                 default_serial = 0;
                 default_parallel = 0;
@@ -4418,6 +4423,15 @@ int main(int argc, char **argv, char **envp)
             error_free(local_err);
             exit(1);
         }
+    } else if (incoming_local) {
+        Error *local_err = NULL;
+        qemu_start_local_incoming_migration(incoming_local, &local_err);
+        if (local_err) {
+            fprintf(stderr, "-incoming_local %s: %s\n", incoming_local,
+                    error_get_pretty(local_err));
+            error_free(local_err);
+            exit(1);
+        }
     } else if (autostart) {
         vm_start();
     }
-- 
1.7.7.6

[Qemu-devel] [PATCH 7/7] hmp: add hmp_localhost_migration interface

[Lei Li]

Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
---
 hmp-commands.hx |   17 +++++++++++++++++
 hmp.c           |   13 +++++++++++++
 hmp.h           |    1 +
 3 files changed, 31 insertions(+), 0 deletions(-)

diff --git a/hmp-commands.hx b/hmp-commands.hx
index 915b0d1..ed21970 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -989,6 +989,23 @@ STEXI
 Set the spice/vnc connection info for the migration target.  The spice/vnc
 server will ask the spice/vnc client to automatically reconnect using the
 new parameters (if specified) once the vm migration finished successfully.
+
+ETEXI
+
+    {
+        .name       = "localhost_migrate",
+        .args_type  = "uri:s",
+        .params     = "uri",
+        .help       = "migrate to domain socket URI on localhost",
+        .mhandler.cmd = hmp_localhost_migrate,
+    },
+
+
+STEXI
+@item localhost_migrate @var{uri}
+@findex localhost_migrate
+Migrate to @var{uri}.
+
 ETEXI
 
     {
diff --git a/hmp.c b/hmp.c
index 494a9aa..a7e779f 100644
--- a/hmp.c
+++ b/hmp.c
@@ -1191,6 +1191,19 @@ void hmp_migrate(Monitor *mon, const QDict *qdict)
     }
 }
 
+void hmp_localhost_migrate(Monitor *mon, const QDict *qdict)
+{
+    const char *uri = qdict_get_str(qdict, "uri");
+    Error *errp = NULL;
+
+    qmp_localhost_migrate(uri, &errp);
+    if (errp) {
+        monitor_printf(mon, "local_migrate: %s\n", error_get_pretty(errp));
+        error_free(errp);
+        return;
+    }
+}
+
 void hmp_device_del(Monitor *mon, const QDict *qdict)
 {
     const char *id = qdict_get_str(qdict, "id");
diff --git a/hmp.h b/hmp.h
index 56d2e92..878bd0c 100644
--- a/hmp.h
+++ b/hmp.h
@@ -72,6 +72,7 @@ void hmp_block_job_pause(Monitor *mon, const QDict *qdict);
 void hmp_block_job_resume(Monitor *mon, const QDict *qdict);
 void hmp_block_job_complete(Monitor *mon, const QDict *qdict);
 void hmp_migrate(Monitor *mon, const QDict *qdict);
+void hmp_localhost_migrate(Monitor *mon, const QDict *qdict);
 void hmp_device_del(Monitor *mon, const QDict *qdict);
 void hmp_dump_guest_memory(Monitor *mon, const QDict *qdict);
 void hmp_netdev_add(Monitor *mon, const QDict *qdict);
-- 
1.7.7.6

Re: [Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Anthony Liguori]

Lei Li <lilei@linux.vnet.ibm.com> writes:

> Hi,
>
> This patch series tries to add localhost live migration support
> to Qemu. It is just a draft version which is far from completing
> and I am still working on, send it out to have your suggestions
> and make sure it is the way should be headed. Your comments are
> very appreciated!

Thanks for posting Lei!

For the benefit of the other reviewers, the goal here is to allow "live
upgrade" of a running QEMU instance.  The work flow would look like
this:

1) Guests are running QEMU release 1.6.1
2) Admin installs QEMU release 1.6.2 via RPM or deb
3) Admin does localhost migration with page flipping to use new version
   of QEMU.

Page flipping is used in order to avoid requiring that there is enough
free memory to fit an additional copy of the largest guest which is the
requirement today with localhost migration.

> When doing localhost live migration, the host memory will balloon
> up during the period, might consume double memories for some time.
> So we want to add a new live migration mechanism localhost live 
> migration.
>
> The plan is:
>
> 1) Add new command to do localhost migration.
>
>    The qmp interface introduced like:
>
>    { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
>
> 2) Use different mechanism than current live migration.
>
>    The very basic work flow like:
>
>        qemu on the source (the source and destination are both on localhost)
>               |
>               V
>            Stop VM
>               |
>               V   
>           Create threads
>               |
>               V
>        Page flipping through vmspice
>               |
>               V
>        MADV_DONTNEED the ram pages which are already flipped

vmsplice() is a weird system call.

You would think that SPLICE_F_GIFT would imply MADV_DONTNEED but it
doesn't appear today.  So to start with, we're doing it explicitly.

We're planning on also looking at the kernel side of this as today the
combination of vmsplice() to a pipe plus vmsplice() from a pipe() will
not actually flip the page but instead copy it.  But I think this is a
fixable problem.

It would also be handy to flip large pages when THP is active but that's
future work.

Regards,

Anthony Liguori

>               |
>               V
>        Migration completes
>
>    As stopping VM first, we expect/resume the page flipping through vmspice
>    is fast enough to meet *live migration (low downtime).
>
> Notes:
> Currently the work flow is not exactly the same as description
> above. For the first step, the work flow we plan to implement is:
> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED 
> ram pages that already copied. After that, will replace to vmsplice
> mechanism instead of copying pages.
>
> TODO:
> - Working on the completing of ram copy and MADV_DONTNEED part.
> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
> - benchmark/evaluation.
>
> Lei Li (7):
>   migration: export MIG_STATE_xxx flags
>   savevm: export qemu_save_device_state()
>   arch_init: add ram_madvise_free()
>   migration-local: implementation of outgoing part
>   migration-local: impelementation of incoming part
>   migration-local: add option to command line for local incoming
>   hmp:add hmp_localhost_migration interface
>
>
>  Makefile.objs                 |    1 +
>  arch_init.c                   |   13 ++++
>  hmp-commands.hx               |   17 +++++
>  hmp.c                         |   13 ++++
>  hmp.h                         |    1 +
>  include/migration/migration.h |   32 +++++++++
>  include/sysemu/sysemu.h       |    1 +
>  migration-local.c             |  155 +++++++++++++++++++++++++++++++++++++++++
>  migration-unix.c              |   60 ++++++++++++++++
>  migration.c                   |    8 --
>  qapi-schema.json              |   14 ++++
>  qemu-options.hx               |    9 +++
>  qmp-commands.hx               |   22 ++++++
>  savevm.c                      |    2 +-
>  vl.c                          |   14 ++++
>  15 files changed, 353 insertions(+), 9 deletions(-)
>  create mode 100644 migration-local.c

Re: [Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Lei Li]

On 06/17/2013 12:01 AM, Anthony Liguori wrote:
> Lei Li <lilei@linux.vnet.ibm.com> writes:
>
>> Hi,
>>
>> This patch series tries to add localhost live migration support
>> to Qemu. It is just a draft version which is far from completing
>> and I am still working on, send it out to have your suggestions
>> and make sure it is the way should be headed. Your comments are
>> very appreciated!
> Thanks for posting Lei!
>
> For the benefit of the other reviewers, the goal here is to allow "live
> upgrade" of a running QEMU instance.  The work flow would look like
> this:
>
> 1) Guests are running QEMU release 1.6.1
> 2) Admin installs QEMU release 1.6.2 via RPM or deb
> 3) Admin does localhost migration with page flipping to use new version
>     of QEMU.
>
> Page flipping is used in order to avoid requiring that there is enough
> free memory to fit an additional copy of the largest guest which is the
> requirement today with localhost migration.
>
>> When doing localhost live migration, the host memory will balloon
>> up during the period, might consume double memories for some time.
>> So we want to add a new live migration mechanism localhost live
>> migration.
>>
>> The plan is:
>>
>> 1) Add new command to do localhost migration.
>>
>>     The qmp interface introduced like:
>>
>>     { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
>>
>> 2) Use different mechanism than current live migration.
>>
>>     The very basic work flow like:
>>
>>         qemu on the source (the source and destination are both on localhost)
>>                |
>>                V
>>             Stop VM
>>                |
>>                V
>>            Create threads
>>                |
>>                V
>>         Page flipping through vmspice
>>                |
>>                V
>>         MADV_DONTNEED the ram pages which are already flipped
> vmsplice() is a weird system call.
>
> You would think that SPLICE_F_GIFT would imply MADV_DONTNEED but it
> doesn't appear today.  So to start with, we're doing it explicitly.

I see, thanks for your explanation!

>
> We're planning on also looking at the kernel side of this as today the
> combination of vmsplice() to a pipe plus vmsplice() from a pipe() will
> not actually flip the page but instead copy it.  But I think this is a
> fixable problem.
>
> It would also be handy to flip large pages when THP is active but that's
> future work.
>
> Regards,
>
> Anthony Liguori
>
>>                |
>>                V
>>         Migration completes
>>
>>     As stopping VM first, we expect/resume the page flipping through vmspice
>>     is fast enough to meet *live migration (low downtime).
>>
>> Notes:
>> Currently the work flow is not exactly the same as description
>> above. For the first step, the work flow we plan to implement is:
>> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
>> ram pages that already copied. After that, will replace to vmsplice
>> mechanism instead of copying pages.
>>
>> TODO:
>> - Working on the completing of ram copy and MADV_DONTNEED part.
>> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
>> - benchmark/evaluation.
>>
>> Lei Li (7):
>>    migration: export MIG_STATE_xxx flags
>>    savevm: export qemu_save_device_state()
>>    arch_init: add ram_madvise_free()
>>    migration-local: implementation of outgoing part
>>    migration-local: impelementation of incoming part
>>    migration-local: add option to command line for local incoming
>>    hmp:add hmp_localhost_migration interface
>>
>>
>>   Makefile.objs                 |    1 +
>>   arch_init.c                   |   13 ++++
>>   hmp-commands.hx               |   17 +++++
>>   hmp.c                         |   13 ++++
>>   hmp.h                         |    1 +
>>   include/migration/migration.h |   32 +++++++++
>>   include/sysemu/sysemu.h       |    1 +
>>   migration-local.c             |  155 +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   60 ++++++++++++++++
>>   migration.c                   |    8 --
>>   qapi-schema.json              |   14 ++++
>>   qemu-options.hx               |    9 +++
>>   qmp-commands.hx               |   22 ++++++
>>   savevm.c                      |    2 +-
>>   vl.c                          |   14 ++++
>>   15 files changed, 353 insertions(+), 9 deletions(-)
>>   create mode 100644 migration-local.c
>


-- 
Lei

Re: [Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Stefan Hajnoczi]

On Sun, Jun 16, 2013 at 11:37:26PM +0800, Lei Li wrote:
> 2) Use different mechanism than current live migration.
> 
>    The very basic work flow like:
> 
>        qemu on the source (the source and destination are both on localhost)
>               |
>               V
>            Stop VM
>               |
>               V   
>           Create threads
>               |
>               V
>        Page flipping through vmspice
>               |
>               V
>        MADV_DONTNEED the ram pages which are already flipped
>               |
>               V
>        Migration completes
> 
>    As stopping VM first, we expect/resume the page flipping through vmspice
>    is fast enough to meet *live migration (low downtime).

Nice idea, that would allow low downtime QEMU upgrades.

Stefan

Re: [Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Lei Li]

On 06/17/2013 08:57 PM, Stefan Hajnoczi wrote:
> On Sun, Jun 16, 2013 at 11:37:26PM +0800, Lei Li wrote:
>> 2) Use different mechanism than current live migration.
>>
>>     The very basic work flow like:
>>
>>         qemu on the source (the source and destination are both on localhost)
>>                |
>>                V
>>             Stop VM
>>                |
>>                V
>>            Create threads
>>                |
>>                V
>>         Page flipping through vmspice
>>                |
>>                V
>>         MADV_DONTNEED the ram pages which are already flipped
>>                |
>>                V
>>         Migration completes
>>
>>     As stopping VM first, we expect/resume the page flipping through vmspice
>>     is fast enough to meet *live migration (low downtime).
> Nice idea, that would allow low downtime QEMU upgrades.

Well, the idea is from Anthony Liguori and Michael Roth :)

>
> Stefan
>


-- 
Lei