Re: [Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Anthony Liguori <aliguori@us.ibm.com>]

Lei Li <lilei@linux.vnet.ibm.com> writes:

> Hi,
>
> This patch series tries to add localhost live migration support
> to Qemu. It is just a draft version which is far from completing
> and I am still working on, send it out to have your suggestions
> and make sure it is the way should be headed. Your comments are
> very appreciated!

Thanks for posting Lei!

For the benefit of the other reviewers, the goal here is to allow "live
upgrade" of a running QEMU instance.  The work flow would look like
this:

1) Guests are running QEMU release 1.6.1
2) Admin installs QEMU release 1.6.2 via RPM or deb
3) Admin does localhost migration with page flipping to use new version
   of QEMU.

Page flipping is used in order to avoid requiring that there is enough
free memory to fit an additional copy of the largest guest which is the
requirement today with localhost migration.

> When doing localhost live migration, the host memory will balloon
> up during the period, might consume double memories for some time.
> So we want to add a new live migration mechanism localhost live 
> migration.
>
> The plan is:
>
> 1) Add new command to do localhost migration.
>
>    The qmp interface introduced like:
>
>    { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
>
> 2) Use different mechanism than current live migration.
>
>    The very basic work flow like:
>
>        qemu on the source (the source and destination are both on localhost)
>               |
>               V
>            Stop VM
>               |
>               V   
>           Create threads
>               |
>               V
>        Page flipping through vmspice
>               |
>               V
>        MADV_DONTNEED the ram pages which are already flipped

vmsplice() is a weird system call.

You would think that SPLICE_F_GIFT would imply MADV_DONTNEED but it
doesn't appear today.  So to start with, we're doing it explicitly.

We're planning on also looking at the kernel side of this as today the
combination of vmsplice() to a pipe plus vmsplice() from a pipe() will
not actually flip the page but instead copy it.  But I think this is a
fixable problem.

It would also be handy to flip large pages when THP is active but that's
future work.

Regards,

Anthony Liguori

>               |
>               V
>        Migration completes
>
>    As stopping VM first, we expect/resume the page flipping through vmspice
>    is fast enough to meet *live migration (low downtime).
>
> Notes:
> Currently the work flow is not exactly the same as description
> above. For the first step, the work flow we plan to implement is:
> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED 
> ram pages that already copied. After that, will replace to vmsplice
> mechanism instead of copying pages.
>
> TODO:
> - Working on the completing of ram copy and MADV_DONTNEED part.
> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
> - benchmark/evaluation.
>
> Lei Li (7):
>   migration: export MIG_STATE_xxx flags
>   savevm: export qemu_save_device_state()
>   arch_init: add ram_madvise_free()
>   migration-local: implementation of outgoing part
>   migration-local: impelementation of incoming part
>   migration-local: add option to command line for local incoming
>   hmp:add hmp_localhost_migration interface
>
>
>  Makefile.objs                 |    1 +
>  arch_init.c                   |   13 ++++
>  hmp-commands.hx               |   17 +++++
>  hmp.c                         |   13 ++++
>  hmp.h                         |    1 +
>  include/migration/migration.h |   32 +++++++++
>  include/sysemu/sysemu.h       |    1 +
>  migration-local.c             |  155 +++++++++++++++++++++++++++++++++++++++++
>  migration-unix.c              |   60 ++++++++++++++++
>  migration.c                   |    8 --
>  qapi-schema.json              |   14 ++++
>  qemu-options.hx               |    9 +++
>  qmp-commands.hx               |   22 ++++++
>  savevm.c                      |    2 +-
>  vl.c                          |   14 ++++
>  15 files changed, 353 insertions(+), 9 deletions(-)
>  create mode 100644 migration-local.c