Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM

[Lei Li <lilei@linux.vnet.ibm.com>]

On 09/26/2013 08:44 PM, Lei Li wrote:
> On 09/25/2013 11:02 PM, Paolo Bonzini wrote:
>> Il 25/09/2013 16:32, Lei Li ha scritto:
>>> This RFC patch series tries to introduce a mechanism using side
>>> channel pipe for RAM via SCM_RIGHTS with unix domain socket
>>> protocol migration.
>>>
>>> This side channel will be used for the page flipping by vmsplice,
>>> which will be the internal mechanism for localhost migration that
>>> we are trying to add. The previous patch series for localhost migration
>>> as link,
>>>
>>> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html
>>>
>>> After this series, will adjust the process of current migration for
>>> the localhost migration and involve the vmsplice based on the previous
>>> patch set as link above.
>>>
>>> Please let me know if it is the proper way for it or there is anything
>>> need to be improved. Your suggestions and comments are very welcome, 
>>> and
>>> thanks for Paolo for his review and useful suggestions.
>>>
>>>
>>> Lei Li (8):
>>>    migration-local: add pipe protocol for QEMUFileOps
>>>    migration-local: add qemu_fopen_pipe()
>>>    migration-local: add send_pipefd()
>>>    migration-local: add recv_pipefd()
>>>    QAPI: introduce magration capability unix_page_flipping
>>>    migration: add migrate_unix_page_flipping()
>>>    migration-unix: side channel support on unix outgoing
>>>    migration-unix: side channel support on unix incoming
>>>
>>>   Makefile.target               |    1 +
>>>   include/migration/migration.h |    3 +
>>>   include/migration/qemu-file.h |    4 +
>>>   migration-local.c             |  247 
>>> +++++++++++++++++++++++++++++++++++++++++
>>>   migration-unix.c              |   48 +++++++-
>>>   migration.c                   |    9 ++
>>>   qapi-schema.json              |    8 +-
>>>   7 files changed, 315 insertions(+), 5 deletions(-)
>>>   create mode 100644 migration-local.c
>>>
>> Yes, this is much closer!
>>
>> There are two problems to be fixed, but it is getting there.
>>
>> First, it breaks migration from old QEMU to new QEMU, and also migration
>> where the source uses "unix:" and the destination uses "fd:" migration
>> (this should work as long as page flipping is disabled).  The problem is
>> that recv_pipefd() "eats" one byte, and old versions of QEMU do not send
>> that byte.
>
> Hi Paolo,
>
> I didn't consider this, thanks for pointing it out!
>
>> The second problem is that you are not really using a side channel; you
>> are still using the QEMUFile and relying on the normal migration code to
>> send pages on the pipe.  This will not be possible when you use 
>> vmsplice.
>
> Yes, you are right, and I am trying to involve the vmsplice.
>
>>
>> Both problems can be addressed with a single change in your approach:
>> always use the Unix socket QEMUFile but, if page flipping is enabled,
>> only transmit page addresses on the socket; page data will be on the
>> pipe.  You can use hooks such as before_ram_iterate, save_page and
>> hook_ram_load to do all your customizations: send the pipe file
>> descriptor, read the pipe file descriptor, and use the pipe as a side
>> channel.
>>
>> To fix the first problem, you can use the before_ram_iterate callback to
>> send the fd, and the hook_ram_load callback to receive it.  The
>> before_ram_iterate callback can write a special 8-byte record (with the
>> RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by
>> send_pipefd().  The load_hook callback is called after the first 8-byte
>> record is sent, and can just do recv_pipefd().

Hi Paolo,

When debugging the code, I realized that this problem might still exist. In the
incoming part, it will qemu_fopen_pipe() in unix_accept_incoming_migration
first to enable the load_hook callback, the check action of this RAM_SAVE_FLAG_HOOK
flags would lead to 8 bytes taken. Turns out, it will break normal unix migration
(without unix-page-flipping), because no matter normal unix migration or
unix-page-flipping migration, the incoming side has to check this 8-byes flags
first to decide whether the load_hook is called, and normal unix migration
did not send this 8-byte flags.

I wonder if I didn't understand your suggestion correctly?

>>
>> To fix the second problem, and really use the pipe as a side channel,
>> you can use the save_page QEMUFile callback on the send side. This
>> callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is
>> disabled.  If it is enabled, it should write another 8-byte record with
>> the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on
>> the Unix socket; then write the page data on the pipe, and return 0.  On
>> the receive side, the 8-byte page address will once more cause the
>> load_hook callback to be called.  This time you already have a file
>> descriptor, so you do not need to call recv_pipefd(): you just extract
>> the page address from the 8-byte record and read the page data from the
>> pipe.
>
> Thanks for your comprehensive suggestions, really nice ideas!
>
>>
>> The basis of your code will still be the socket-based QEMUFile, but
>> you'll need your own QEMUFile since you're adding Unix-specific
>> functionality.  For this it is not a problem to have two copies the
>> QEMUFile code for sockets, one in savevm.c and one in migration-unix.c.
>
> Have two copies of the QEMUFile code for sockets, do you mean in my own
> QEMUFile, say QEMUFilePipe, includes both the copy of QEMUFileSocket
> code (like get_fd, get_buffer, writev_buffer..) and the Unix-specific
> functionality code that override these three hooks like your suggestions
> above?
>
> I guess 'migration-unix.c' you typed is 'migration-local.c', right?
>
>>   It's a very small amount of code.
>>
>> Paolo
>>
>
>


-- 
Lei