Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li <lilei@linux.vnet.ibm.com>]

On 09/25/2013 11:02 PM, Paolo Bonzini wrote:
> Il 25/09/2013 16:32, Lei Li ha scritto:
>> This RFC patch series tries to introduce a mechanism using side
>> channel pipe for RAM via SCM_RIGHTS with unix domain socket
>> protocol migration.
>>
>> This side channel will be used for the page flipping by vmsplice,
>> which will be the internal mechanism for localhost migration that
>> we are trying to add. The previous patch series for localhost migration
>> as link,
>>
>> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html
>>
>> After this series, will adjust the process of current migration for
>> the localhost migration and involve the vmsplice based on the previous
>> patch set as link above.
>>
>> Please let me know if it is the proper way for it or there is anything
>> need to be improved. Your suggestions and comments are very welcome, and
>> thanks for Paolo for his review and useful suggestions.
>>
>>
>> Lei Li (8):
>>    migration-local: add pipe protocol for QEMUFileOps
>>    migration-local: add qemu_fopen_pipe()
>>    migration-local: add send_pipefd()
>>    migration-local: add recv_pipefd()
>>    QAPI: introduce magration capability unix_page_flipping
>>    migration: add migrate_unix_page_flipping()
>>    migration-unix: side channel support on unix outgoing
>>    migration-unix: side channel support on unix incoming
>>
>>   Makefile.target               |    1 +
>>   include/migration/migration.h |    3 +
>>   include/migration/qemu-file.h |    4 +
>>   migration-local.c             |  247 +++++++++++++++++++++++++++++++++++++++++
>>   migration-unix.c              |   48 +++++++-
>>   migration.c                   |    9 ++
>>   qapi-schema.json              |    8 +-
>>   7 files changed, 315 insertions(+), 5 deletions(-)
>>   create mode 100644 migration-local.c
>>
> Yes, this is much closer!
>
> There are two problems to be fixed, but it is getting there.
>
> First, it breaks migration from old QEMU to new QEMU, and also migration
> where the source uses "unix:" and the destination uses "fd:" migration
> (this should work as long as page flipping is disabled).  The problem is
> that recv_pipefd() "eats" one byte, and old versions of QEMU do not send
> that byte.

Hi Paolo,

I didn't consider this, thanks for pointing it out!

> The second problem is that you are not really using a side channel; you
> are still using the QEMUFile and relying on the normal migration code to
> send pages on the pipe.  This will not be possible when you use vmsplice.

Yes, you are right, and I am trying to involve the vmsplice.

>
> Both problems can be addressed with a single change in your approach:
> always use the Unix socket QEMUFile but, if page flipping is enabled,
> only transmit page addresses on the socket; page data will be on the
> pipe.  You can use hooks such as before_ram_iterate, save_page and
> hook_ram_load to do all your customizations: send the pipe file
> descriptor, read the pipe file descriptor, and use the pipe as a side
> channel.
>
> To fix the first problem, you can use the before_ram_iterate callback to
> send the fd, and the hook_ram_load callback to receive it.  The
> before_ram_iterate callback can write a special 8-byte record (with the
> RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by
> send_pipefd().  The load_hook callback is called after the first 8-byte
> record is sent, and can just do recv_pipefd().
>
> To fix the second problem, and really use the pipe as a side channel,
> you can use the save_page QEMUFile callback on the send side.  This
> callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is
> disabled.  If it is enabled, it should write another 8-byte record with
> the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on
> the Unix socket; then write the page data on the pipe, and return 0.  On
> the receive side, the 8-byte page address will once more cause the
> load_hook callback to be called.  This time you already have a file
> descriptor, so you do not need to call recv_pipefd(): you just extract
> the page address from the 8-byte record and read the page data from the
> pipe.

Thanks for your comprehensive suggestions, really nice ideas!

>
> The basis of your code will still be the socket-based QEMUFile, but
> you'll need your own QEMUFile since you're adding Unix-specific
> functionality.  For this it is not a problem to have two copies the
> QEMUFile code for sockets, one in savevm.c and one in migration-unix.c.

Have two copies of the QEMUFile code for sockets, do you mean in my own
QEMUFile, say QEMUFilePipe, includes both the copy of QEMUFileSocket
code (like get_fd, get_buffer, writev_buffer..) and the Unix-specific
functionality code that override these three hooks like your suggestions
above?

I guess 'migration-unix.c' you typed is 'migration-local.c', right?

>   It's a very small amount of code.
>
> Paolo
>


-- 
Lei