Re: [Qemu-devel] [PATCH 0/7 RFC] Localhost live migration

[Lei Li <lilei@linux.vnet.ibm.com>]

On 06/17/2013 12:01 AM, Anthony Liguori wrote:
> Lei Li <lilei@linux.vnet.ibm.com> writes:
>
>> Hi,
>>
>> This patch series tries to add localhost live migration support
>> to Qemu. It is just a draft version which is far from completing
>> and I am still working on, send it out to have your suggestions
>> and make sure it is the way should be headed. Your comments are
>> very appreciated!
> Thanks for posting Lei!
>
> For the benefit of the other reviewers, the goal here is to allow "live
> upgrade" of a running QEMU instance.  The work flow would look like
> this:
>
> 1) Guests are running QEMU release 1.6.1
> 2) Admin installs QEMU release 1.6.2 via RPM or deb
> 3) Admin does localhost migration with page flipping to use new version
>     of QEMU.
>
> Page flipping is used in order to avoid requiring that there is enough
> free memory to fit an additional copy of the largest guest which is the
> requirement today with localhost migration.
>
>> When doing localhost live migration, the host memory will balloon
>> up during the period, might consume double memories for some time.
>> So we want to add a new live migration mechanism localhost live
>> migration.
>>
>> The plan is:
>>
>> 1) Add new command to do localhost migration.
>>
>>     The qmp interface introduced like:
>>
>>     { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
>>
>> 2) Use different mechanism than current live migration.
>>
>>     The very basic work flow like:
>>
>>         qemu on the source (the source and destination are both on localhost)
>>                |
>>                V
>>             Stop VM
>>                |
>>                V
>>            Create threads
>>                |
>>                V
>>         Page flipping through vmspice
>>                |
>>                V
>>         MADV_DONTNEED the ram pages which are already flipped
> vmsplice() is a weird system call.
>
> You would think that SPLICE_F_GIFT would imply MADV_DONTNEED but it
> doesn't appear today.  So to start with, we're doing it explicitly.

I see, thanks for your explanation!

>
> We're planning on also looking at the kernel side of this as today the
> combination of vmsplice() to a pipe plus vmsplice() from a pipe() will
> not actually flip the page but instead copy it.  But I think this is a
> fixable problem.
>
> It would also be handy to flip large pages when THP is active but that's
> future work.
>
> Regards,
>
> Anthony Liguori
>
>>                |
>>                V
>>         Migration completes
>>
>>     As stopping VM first, we expect/resume the page flipping through vmspice
>>     is fast enough to meet *live migration (low downtime).
>>
>> Notes:
>> Currently the work flow is not exactly the same as description
>> above. For the first step, the work flow we plan to implement is:
>> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
>> ram pages that already copied. After that, will replace to vmsplice
>> mechanism instead of copying pages.
>>
>> TODO:
>> - Working on the completing of ram copy and MADV_DONTNEED part.
>> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
>> - benchmark/evaluation.
>>
>> Lei Li (7):
>>    migration: export MIG_STATE_xxx flags
>>    savevm: export qemu_save_device_state()
>>    arch_init: add ram_madvise_free()
>>    migration-local: implementation of outgoing part
>>    migration-local: impelementation of incoming part
>>    migration-local: add option to command line for local incoming
>>    hmp:add hmp_localhost_migration interface
>>
>>
>>   Makefile.objs                 |    1 +
>>   arch_init.c                   |   13 ++++
>>   hmp-commands.hx               |   17 +++++
>>   hmp.c                         |   13 ++++
>>   hmp.h                         |    1 +
>>   include/migration/migration.h |   32 +++++++++
>>   include/sysemu/sysemu.h       |    1 +
>>   migration-local.c             |  155 +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   60 ++++++++++++++++
>>   migration.c                   |    8 --
>>   qapi-schema.json              |   14 ++++
>>   qemu-options.hx               |    9 +++
>>   qmp-commands.hx               |   22 ++++++
>>   savevm.c                      |    2 +-
>>   vl.c                          |   14 ++++
>>   15 files changed, 353 insertions(+), 9 deletions(-)
>>   create mode 100644 migration-local.c
>


-- 
Lei