Re: [PATCH 1/2] migration: Add some documentation for multifd

[Fabiano Rosas <farosas@suse.de>]

Prasad Pandit <ppandit@redhat.com> writes:

> Hello Fabiano,
>
> * First big thank you for starting/writing this document. It is a
> great resource.
>
> On Fri, 7 Mar 2025 at 19:13, Fabiano Rosas <farosas@suse.de> wrote:
>> +++ b/docs/devel/migration/multifd.rst
>> @@ -0,0 +1,254 @@
>> +Multifd
>> +Multifd is the name given for the migration capability that enables
>> +data transfer using multiple threads. Multifd supports all the
>> +transport types currently in use with migration (inet, unix, vsock,
>> +fd, file).
>
> * Multifd is Multiple File Descriptors, right? Ie. Does it work with
> one thread but multiple file descriptors? OR one thread per file
> descriptor is always the case? I have not used/tried 'multifd +
> file://' migration, but I imagined there one thread might be able to
> read/write to multiple file descriptors at a time.
>

Technically both can happen. But that would just be the case of
file:fdset migration which requires an extra fd for O_DIRECT. So
"multiple" in the usual sense of "more is better" is only
fd-per-thread. IOW, using multiple fds is an implementation detail IMO,
what people really care about is medium saturation, which we can only
get (with multifd) via parallelization.

>> +Usage
>> +-----
>> +
>> +On both source and destination, enable the ``multifd`` capability:
>> +
>> +    ``migrate_set_capability multifd on``
>> +
>> +Define a number of channels to use (default is 2, but 8 usually
>> +provides best performance).
>> +
>> +    ``migrate_set_parameter multifd-channels 8``
>> +
>
> * I get that this is a QEMU documentation, but for users/reader's
> convenience it'll help to point to libvirt:virsh migrate usage here ->
> https://www.libvirt.org/manpages/virsh.html#migrate , just as an
> alternative.

AFAIK, we tend to not do that in QEMU docs.

> Because doing migration via QMP commands is not as
> straightforward, I wonder who might do that and why.
>

All of QEMU developers, libvirt developers, cloud software developers,
kernel developers etc.

>
>> +Restrictions
>> +------------
>> +
>> +For migration to a file, support is conditional on the presence of the
>> +mapped-ram capability, see `mapped-ram`.
>> +
>> +Snapshots are currently not supported.
>
> * Maybe: Sanpshot using multiple threads (multifd) is not supported.
>
>> +`postcopy` migration is currently not supported.
>
> * Maybe - 'postcopy' migration using multiple threads (multifd) is not
> supported. ie. 'postcopy' uses a single thread to transfer migration
> data.
>
> * Reason for these suggestions: as a writer it is easy to think
> everything written in this page is to be taken with multifd context,
> but readers may not do that, they may take sentences in isolation.
> (just sharing thoughts)
>

Sure, I can expand on those.

>> +Multifd consists of:
>> +
>> +- A client that produces the data on the migration source side and
>> +  consumes it on the destination. Currently the main client code is
>> +  ram.c, which selects the RAM pages for migration;
>
> * So multifd mechanism can be used to transfer non-ram data as well? I
> thought it's only used for RAM migration. Are device/gpu states etc
> bits also transferred via multifd threads?
>

device state migration with multifd has been merged for 10.0

<rant>
If it were up to me, we'd have a pool of multifd threads that transmit
everything migration-related. Unfortunately, that's not so
straight-forward to implement without rewriting a lot of code, multifd
requires too much entanglement from the data producer. We're constantly
dealing with details of data transmission getting in the way of data
production/consumption (e.g. try to change ram.c to produce multiple
pages at once and watch everyting explode).

I've been experimenting with a MultiFDIov payload type to allow
separation between the data type handling details and multifd inner
workings. However in order for that to be useful we'd need to have a
sync that doesn't depend on control data on the main migration
thread. That's why I've been asking about a multifd-only sync with Peter
in the other thread.

There's a bunch of other issues as well:

- no clear distinction between what should go in the header and what
  should go in the packet.

- the header taking up one slot in the iov, which should in theory be
  responsibility of the client

- the whole multifd_ops situation which doesn't allow a clear interface
  between multifd and client

- the lack of uniformity between send/recv in regards to doing I/O from
  multifd code or from client code

- the recv having two different modes of operation, socket and file

the list goes on...
</rant>

>> +- A packet which is the final result of all the data aggregation
>> +  and/or transformation. The packet contains: a *header* with magic and
>> +  version numbers and flags that inform of special processing needed
>> +  on the destination; a *payload-specific header* with metadata referent
>> +  to the packet's data portion, e.g. page counts; and a variable-size
>> +  *data portion* which contains the actual opaque payload data.
>
> * It'll help to define the exact packet format here. Like they do in RFCs.

I'll try to produce some ascii art.

>
> Thank you for writing this.
> ---
>   - Prasad