Re: [Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration

[Lei Li <lilei@linux.vnet.ibm.com>]

On 07/26/2013 05:41 PM, Paolo Bonzini wrote:
> Il 25/07/2013 22:18, Lei Li ha scritto:
>> Hi,
>>
>> This patch series tries to add localhost migration support to
>> Qemu.
>>
>> When doing localhost migration, the host memory will balloon
>> up during the period, might consume double memories for some time.
>> So we want to add a new live migration mechanism localhost
>> migration.
>>
>> Following I copied from last version that Anthony added for the
>> benefit of the other reviewers:
>>
>> The goal here is to allow "live upgrade" of a running QEMU instance.
>> The work flow would look like this:
>>
>> 1) Guests are running QEMU release 1.6.1
>> 2) Admin installs QEMU release 1.6.2 via RPM or deb
>> 3) Admin does localhost migration with page flipping to use new version
>>     of QEMU.
>>
>> Page flipping is used in order to avoid requiring that there is enough
>> free memory to fit an additional copy of the largest guest which is the
>> requirement today with localhost migration.
>>
>> You can also read from the link below:
>> http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html
>>
>> The plan is:
>>
>> 1) Add new command to do localhost migration.
>>
>>     The qmp interface introduced like:
>>
>>     { 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
>>
>> 2) Use different mechanism than current live migration.
>>
>>     The very basic work flow like:
>>
>>         qemu on the source (the source and destination are both on localhost)
>>                |
>>                V
>>             Stop VM
>>                |
>>                V
>>            Create threads
>>                |
>>                V
>>         Page flipping through vmspice
>>                |
>>                V
>>         MADV_DONTNEED the ram pages which are already flipped
>>                |
>>                V
>>         Migration completes
>>
>>     As stopping VM first, we expect/resume the page flipping through vmspice
>>     is fast enough to meet *live migration (low downtime).
>>
>> Notes:
>> Currently the work flow is not exactly the same as description
>> above. For the first step, the work flow we implemented is:
>> stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
>> ram pages that already copied. After that, will replace to vmsplice
>> mechanism instead of copying pages.
>>
>> Now it's still a draft version, and as it is implemented separately
>> to the current migration code for a easy start, the next step will
>> be trying to integrate it into the current migration implementation
>> closely. To make sure we are on the right direction that should be
>> headed, please let me know your suggestions on this.
>>
>> For the interface, as Anthony has suggested using a flag or a
>> capability, which one would you prefer or any ideas?
> Using a capability on the source makes sense; I don't think anything
> special is needed on the destination.  The destination just sees a
> special packet telling it that a pipe is available via SCM_RIGHTS; then
> it fetches the file descriptor and uses it for the new protocol.
>
> It looks like this could reuse a lot of the RAM copying hooks that we
> introduced for RDMA.  You shouldn't need to change anything in savevm.c
> or arch_init.c

Hi Paolo,

Thanks for your suggestions!

And sorry for the late reply, I am looking into RDMA implementation and
trying to figure out the way to integrate with its approach as you suggested.

For the interface on the source, I am OK with using a capability.
For the destination, currently we are copying ram pages via unix
domain socket as first step, will replace to pipe later. How about
add a new URI prefix as Michael R.Hines suggested?



> Paolo
>
>> Your comments are very welcome!
>>
>> TODO:
>> - Integrate to the current implement of migration closely?
>> - Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
>> - benchmark/evaluation.
>>
>> Lei Li (12):
>>    migration: export MIG_STATE_xxx flags
>>    savevm: export qemu_save_device_state()
>>    rename is_active to is_block_active
>>    arch_init: introduce ram_page_save()
>>    arch_init: introduce ram_save_local()
>>    arch_init: add save_local_setup to savevm_ram_handlers
>>    savevm: introduce qemu_savevm_local()
>>    savevm: adjust is_ram check in register_savevm_live()
>>    migration-local: implementation of outgoing part
>>    migration-local: implementation of incoming part
>>    migration-local: add option to command line for local incoming
>>    hmp:add hmp_localhost_migration interface
>>
>>   Makefile.objs                 |    1 +
>>   arch_init.c                   |   110 +++++++++++++++++++
>>   block-migration.c             |    2 +-
>>   hmp-commands.hx               |   17 ++++
>>   hmp.c                         |   13 +++
>>   hmp.h                         |    1 +
>>   include/migration/migration.h |   32 +++++++
>>   include/sysemu/sysemu.h       |    1 +
>>   migration-local.c             |  228 +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   60 ++++++++++++
>>   migration.c                   |    8 --
>>   qapi-schema.json              |   14 +++
>>   qemu-options.hx               |    9 ++
>>   qmp-commands.hx               |   22 +++++
>>   savevm.c                      |   100 +++++++++++++++--
>>   vl.c                          |   14 +++
>>   16 files changed, 613 insertions(+), 19 deletions(-)
>>   create mode 100644 migration-local.c
>>
>


-- 
Lei