Re: [PATCH V4 19/19] migration: cpr-transfer documentation

[Steven Sistare <steven.sistare@oracle.com>]

On 12/19/2024 12:02 PM, Peter Xu wrote:
> On Mon, Dec 02, 2024 at 05:20:11AM -0800, Steve Sistare wrote:
>> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
> 
> I think this one is already good enough, so:
> 
> Reviewed-by: Peter Xu <peterx@redhat.com>
> 
> But still, a few comments inline.
> 
>> ---
>>   docs/devel/migration/CPR.rst | 176 ++++++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 174 insertions(+), 2 deletions(-)
>>
>> diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
>> index 63c3647..a8a57c0 100644
>> --- a/docs/devel/migration/CPR.rst
>> +++ b/docs/devel/migration/CPR.rst
>> @@ -5,7 +5,7 @@ CPR is the umbrella name for a set of migration modes in which the
>>   VM is migrated to a new QEMU instance on the same host.  It is
>>   intended for use when the goal is to update host software components
>>   that run the VM, such as QEMU or even the host kernel.  At this time,
>> -cpr-reboot is the only available mode.
>> +the cpr-reboot and cpr-transfer modes are available.
>>   
>>   Because QEMU is restarted on the same host, with access to the same
>>   local devices, CPR is allowed in certain cases where normal migration
>> @@ -53,7 +53,7 @@ RAM is copied to the migration URI.
>>   Outgoing:
>>     * Set the migration mode parameter to ``cpr-reboot``.
>>     * Set the ``x-ignore-shared`` capability if desired.
>> -  * Issue the ``migrate`` command.  It is recommended the the URI be a
>> +  * Issue the ``migrate`` command.  It is recommended the URI be a
>>       ``file`` type, but one can use other types such as ``exec``,
>>       provided the command captures all the data from the outgoing side,
>>       and provides all the data to the incoming side.
>> @@ -145,3 +145,175 @@ Caveats
>>   
>>   cpr-reboot mode may not be used with postcopy, background-snapshot,
>>   or COLO.
>> +
>> +cpr-transfer mode
>> +-----------------
>> +
>> +This mode allows the user to transfer a guest to a new QEMU instance
>> +on the same host with minimal guest pause time, by preserving guest
>> +RAM in place, albeit with new virtual addresses in new QEMU.
> 
> IMHO it's important to have some words discussing the diff
> v.s. ignore-shared, because from above it's undistinguishable from it.
> 
> The important bit could be that cpr-transfer allows pinning.  If that's too
> internal a concept, we could mention "unlike an ignore-shared migration,
> cpr-transfer will start to enable local migrations to work seamlessly with
> all kinds of device assignments like VFIO, vDPA and so on".

OK, I'll add something like that.

>> +
>> +The user starts new QEMU on the same host as old QEMU, with the
>> +same arguments as old QEMU, plus the ``-incoming option``.  The user
>> +issues the migrate command to old QEMU, which stops the VM, saves
>> +state to the migration channels, and enters the postmigrate state.
>> +Execution resumes in new QEMU.
>> +
>> +This mode requires a second migration channel type "cpr" in the
>> +channel arguments on the outgoing side.  The channel must be a type,
>> +such as unix socket, that supports SCM_RIGHTS.  However, the cpr
>> +channel cannot be added to the list of channels for a migrate-incoming
>> +command, because it must be read before new QEMU opens a monitor.
>> +Instead, the user passes the channel as a second -incoming
>> +command-line argument to new QEMU using JSON syntax.
>> +
>> +Usage
>> +^^^^^
>> +
>> +Memory backend objects must have the ``share=on`` attribute.
>> +
>> +The VM must be started with the ``-machine aux-ram-share=on``
>> +option.  This causes implicit RAM blocks (those not described by
>> +a memory-backend object) to be allocated by mmap'ing a memfd.
>> +Examples include VGA and ROM.
>> +
>> +Outgoing:
>> +  * Set the migration mode parameter to ``cpr-transfer``.
>> +  * Issue the ``migrate`` command, containing a main channel and
>> +    a cpr channel.
>> +
>> +Incoming:
>> +  * Start new QEMU with two ``-incoming`` options.
>> +  * If the VM was running when the outgoing ``migrate`` command was
>> +    issued, then QEMU automatically resumes VM execution.
>> +
>> +Caveats
>> +^^^^^^^
>> +
>> +cpr-transfer mode may not be used with postcopy, background-snapshot,
>> +or COLO.
> 
> Maybe we can even remove this line.  It's not like someone could think
> about supporting any of above; they just don't apply in cpr context.

The same caveat is listed for cpr-reboot mode.  We should either delete both
or keep both.  But IMO many aspects of migration are confusing for beginners,
and more detail rather than less would be helpful.

- Steve

>> +
>> +memory-backend-epc is not supported.
>> +
>> +The main incoming migration channel cannot be a file type.
>> +
>> +If the main incoming migration channel is a tcp type, then the port
>> +cannot be 0 (meaning dynamically choose a port).
>> +
>> +When using ``-incoming defer``, you must issue the migrate command to
>> +old QEMU before issuing any monitor commands to new QEMU, because new
>> +QEMU blocks waiting to read from the cpr channel before starting its
>> +monitor, and old QEMU does not write to the channel until the migrate
>> +command is issued.  However, new QEMU does not open and read the
>> +main migration channel until you issue the migrate incoming command.
>> +
>> +Example 1: incoming channel
>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> +
>> +In these examples, we simply restart the same version of QEMU, but
>> +in a real scenario one would start new QEMU on the incoming side.
>> +Note that new QEMU does not print the monitor prompt until old QEMU
>> +has issued the migrate command.  The outgoing side uses QMP because
>> +HMP cannot specify a CPR channel.  Some QMP responses are omitted for
>> +brevity.
>> +
>> +::
>> +
>> +  Outgoing:                             Incoming:
>> +
>> +  # qemu-kvm -qmp stdio
>> +  -object memory-backend-file,id=ram0,size=4G,
>> +  mem-path=/dev/shm/ram0,share=on -m 4G
>> +  -machine aux-ram-share=on
>> +  ...
>> +                                        # qemu-kvm -monitor stdio
>> +                                        -incoming tcp:0:44444
>> +                                        -incoming '{"channel-type": "cpr",
>> +                                          "addr": { "transport": "socket",
>> +                                          "type": "unix", "path": "cpr.sock"}}'
>> +                                        ...
>> +  {"execute":"qmp_capabilities"}
>> +
>> +  {"execute": "query-status"}
>> +  {"return": {"status": "running",
>> +              "running": true}}
>> +
>> +  {"execute":"migrate-set-parameters",
>> +   "arguments":{"mode":"cpr-transfer"}}
>> +
>> +  {"execute": "migrate", "arguments": { "channels": [
>> +    {"channel-type": "main",
>> +     "addr": { "transport": "socket", "type": "inet",
>> +               "host": "0", "port": "44444" }},
>> +    {"channel-type": "cpr",
>> +     "addr": { "transport": "socket", "type": "unix",
>> +               "path": "cpr.sock" }}]}}
>> +
>> +                                        QEMU 10.0.50 monitor
>> +                                        (qemu) info status
>> +                                        VM status: running
>> +
>> +  {"execute": "query-status"}
>> +  {"return": {"status": "postmigrate",
>> +              "running": false}}
>> +
>> +Example 2: incoming defer
>> +^^^^^^^^^^^^^^^^^^^^^^^^^
>> +
>> +This example uses ``-incoming defer`` to hot plug a device before
>> +accepting the main migration channel.  Again note you must issue the
>> +migrate command to old QEMU before you can issue any monitor
>> +commands to new QEMU.
>> +
>> +
>> +::
>> +
>> +  Outgoing:                             Incoming:
>> +
>> +  # qemu-kvm -monitor stdio
>> +  -object memory-backend-file,id=ram0,size=4G,
>> +  mem-path=/dev/shm/ram0,share=on -m 4G
>> +  -machine aux-ram-share=on
>> +  ...
>> +                                        # qemu-kvm -monitor stdio
>> +                                        -incoming defer
>> +                                        -incoming '{"channel-type": "cpr",
>> +                                          "addr": { "transport": "socket",
>> +                                          "type": "unix", "path": "cpr.sock"}}'
>> +                                        ...
>> +  {"execute":"qmp_capabilities"}
>> +
>> +  {"execute": "device_add",
>> +   "arguments": {"driver": "pcie-root-port"}}
>> +
>> +  {"execute":"migrate-set-parameters",
>> +   "arguments":{"mode":"cpr-transfer"}}
>> +
>> +  {"execute": "migrate", "arguments": { "channels": [
>> +    {"channel-type": "main",
>> +     "addr": { "transport": "socket", "type": "inet",
>> +               "host": "0", "port": "44444" }},
>> +    {"channel-type": "cpr",
>> +     "addr": { "transport": "socket", "type": "unix",
>> +               "path": "cpr.sock" }}]}}
>> +
>> +                                        QEMU 10.0.50 monitor
>> +                                        (qemu) info status
>> +                                        VM status: paused (inmigrate)
>> +                                        (qemu) device_add pcie-root-port
>> +                                        (qemu) migrate_incoming tcp:0:44444
>> +                                        (qemu) info status
>> +                                        VM status: running
>> +
>> +  {"execute": "query-status"}
>> +  {"return": {"status": "postmigrate",
>> +              "running": false}}
>> +
>> +Futures
>> +^^^^^^^
>> +
>> +cpr-transfer mode is based on a capability to transfer open file
>> +descriptors from old to new QEMU.  In the future, descriptors for
>> +vfio, iommufd, vhost, and char devices could be transferred,
>> +preserving those devices and their kernel state without interruption,
>> +even if they do not explicitly support live migration.
>> -- 
>> 1.8.3.1
>>
>