Re: [PATCH V7 24/24] migration: cpr-transfer documentation

[Steven Sistare <steven.sistare@oracle.com>]

On 1/17/2025 9:42 AM, Fabiano Rosas wrote:
> Steve Sistare <steven.sistare@oracle.com> writes:
> 
>> Add documentation for the cpr-transfer migration mode.
>>
>> Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
>> Reviewed-by: Peter Xu <peterx@redhat.com>
>> ---
>>   docs/devel/migration/CPR.rst | 182 ++++++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 180 insertions(+), 2 deletions(-)
>>
>> diff --git a/docs/devel/migration/CPR.rst b/docs/devel/migration/CPR.rst
>> index 63c3647..d6021d5 100644
>> --- a/docs/devel/migration/CPR.rst
>> +++ b/docs/devel/migration/CPR.rst
>> @@ -5,7 +5,7 @@ CPR is the umbrella name for a set of migration modes in which the
>>   VM is migrated to a new QEMU instance on the same host.  It is
>>   intended for use when the goal is to update host software components
>>   that run the VM, such as QEMU or even the host kernel.  At this time,
>> -cpr-reboot is the only available mode.
>> +the cpr-reboot and cpr-transfer modes are available.
>>   
>>   Because QEMU is restarted on the same host, with access to the same
>>   local devices, CPR is allowed in certain cases where normal migration
>> @@ -53,7 +53,7 @@ RAM is copied to the migration URI.
>>   Outgoing:
>>     * Set the migration mode parameter to ``cpr-reboot``.
>>     * Set the ``x-ignore-shared`` capability if desired.
>> -  * Issue the ``migrate`` command.  It is recommended the the URI be a
>> +  * Issue the ``migrate`` command.  It is recommended the URI be a
>>       ``file`` type, but one can use other types such as ``exec``,
>>       provided the command captures all the data from the outgoing side,
>>       and provides all the data to the incoming side.
>> @@ -145,3 +145,181 @@ Caveats
>>   
>>   cpr-reboot mode may not be used with postcopy, background-snapshot,
>>   or COLO.
>> +
>> +cpr-transfer mode
>> +-----------------
>> +
>> +This mode allows the user to transfer a guest to a new QEMU instance
>> +on the same host with minimal guest pause time, by preserving guest
>> +RAM in place, albeit with new virtual addresses in new QEMU.  Devices
>> +and their pinned memory pages will also be preserved in a future QEMU
>> +release.
>> +
>> +The user starts new QEMU on the same host as old QEMU, with command-
>> +line arguments to create the same machine, plus the ``-incoming``
>> +option for the main migration channel, like normal live migration.
>> +In addition, the user adds a second -incoming option with channel
>> +type ``cpr``.  This CPR channel must support file descriptor transfer
>> +with SCM_RIGHTS, i.e. it must be a UNIX domain socket.
>> +
>> +To initiate CPR, the user issues a migrate command to old QEMU,
>> +adding a second migration channel of type ``cpr`` in the channels
>> +argument.  Old QEMU stops the VM, saves state to the migration
>> +channels, and enters the postmigrate state.  Execution resumes in
>> +new QEMU.
>> +
>> +New QEMU reads the CPR channel before opening a monitor, hence
>> +the CPR channel cannot be specified in the list of channels for a
>> +migrate-incoming command.  It may only be specified on the command
>> +line.
>> +
>> +Usage
>> +^^^^^
>> +
>> +Memory backend objects must have the ``share=on`` attribute.
>> +
>> +The VM must be started with the ``-machine aux-ram-share=on``
>> +option.  This causes implicit RAM blocks (those not described by
>> +a memory-backend object) to be allocated by mmap'ing a memfd.
>> +Examples include VGA and ROM.
>> +
>> +Outgoing:
>> +  * Set the migration mode parameter to ``cpr-transfer``.
>> +  * Issue the ``migrate`` command, containing a main channel and
>> +    a cpr channel.
>> +
>> +Incoming:
>> +  * Start new QEMU with two ``-incoming`` options.
>> +  * If the VM was running when the outgoing ``migrate`` command was
>> +    issued, then QEMU automatically resumes VM execution.
>> +
>> +Caveats
>> +^^^^^^^
>> +
>> +cpr-transfer mode may not be used with postcopy, background-snapshot,
>> +or COLO.
>> +
>> +memory-backend-epc is not supported.
>> +
>> +The main incoming migration channel address cannot be a file type.
>> +
>> +If the main incoming channel address is an inet socket, then the port
>> +cannot be 0 (meaning dynamically choose a port).
>> +
>> +When using ``-incoming defer``, you must issue the migrate command to
>> +old QEMU before issuing any monitor commands to new QEMU, because new
>> +QEMU blocks waiting to read from the cpr channel before starting its
>> +monitor, and old QEMU does not write to the channel until the migrate
>> +command is issued.  However, new QEMU does not open and read the
>> +main migration channel until you issue the migrate incoming command.
>> +
>> +Example 1: incoming channel
>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> +
>> +In these examples, we simply restart the same version of QEMU, but
>> +in a real scenario one would start new QEMU on the incoming side.
>> +Note that new QEMU does not print the monitor prompt until old QEMU
>> +has issued the migrate command.  The outgoing side uses QMP because
>> +HMP cannot specify a CPR channel.  Some QMP responses are omitted for
>> +brevity.
>> +
>> +::
>> +
>> +  Outgoing:                             Incoming:
>> +
>> +  # qemu-kvm -qmp stdio
>> +  -object memory-backend-file,id=ram0,size=4G,
>> +  mem-path=/dev/shm/ram0,share=on -m 4G
>> +  -machine aux-ram-share=on
>> +  ...
>> +                                        # qemu-kvm -monitor stdio
>> +                                        -incoming tcp:0:44444
>> +                                        -incoming '{"channel-type": "cpr",
>> +                                          "addr": { "transport": "socket",
>> +                                          "type": "unix", "path": "cpr.sock"}}'
>> +                                        ...
> 
> I'm attempting this and not having much success. Surely I'm missing
> something:
> 
> 
> $ qemu-system-x86_64 -cpu host -smp 16 -machine pc,accel=kvm \
>    -drive id=drive0,if=none,format=qcow2,file=img.qcow2 \
>    -device virtio-blk-pci,id=image1,drive=drive0,bootindex=0 \
>    -qmp unix:./dst-qmp.sock,server,wait=off \
>    -nographic -serial mon:stdio \
>    -object memory-backend-file,id=ram0,size=4G,mem-path=/dev/shm/ram0,share=on \
>    -m 4G -machine aux-ram-share=on \
> 
>    -incoming tcp:0:44444 \
>    -incoming '{"channel-type": "cpr", "addr": { "transport": "socket", "type": "unix", "path": "cpr.sock"}}' \
> 
>    -trace loadvm_* -trace cpr_* -trace migration_* -trace migrate_* -trace qemu_loadvm_*
> 
> cpr_transfer_input cpr.sock
> cpr_state_load cpr-transfer mode
> cpr_find_fd pc.bios, id 0 returns 15
> cpr_find_fd pc.rom, id 0 returns 14
> cpr_find_fd 0000:00:02.0/vga.vram, id 0 returns 13
> cpr_find_fd 0000:00:02.0/vga.rom, id 0 returns 12
> cpr_find_fd 0000:00:03.0/e1000.rom, id 0 returns 11
> cpr_find_fd /rom@etc/acpi/tables, id 0 returns 10
> cpr_find_fd /rom@etc/table-loader, id 0 returns 8
> cpr_find_fd /rom@etc/acpi/rsdp, id 0 returns 3
> migrate_set_state new state setup
> migration_socket_incoming_accepted
> migration_set_incoming_channel ioc=0x564dc31e7000 ioctype=qio-channel-socket
> migrate_set_state new state active
> loadvm_state_setup
> qemu_loadvm_state_section 1
> qemu_loadvm_state_section_startfull 2(ram) 0 4
> qemu_loadvm_state_section 3
> qemu_loadvm_state_section_partend 2
> qemu_loadvm_state_section 4
> qemu_loadvm_state_section_startfull 0(timer) 0 2
> qemu_loadvm_state_section 4
> qemu_loadvm_state_section_startfull 1(slirp) 0 4
> qemu_loadvm_state_section 4
> qemu_loadvm_state_section_startfull 4(cpu_common) 0 1
> qemu_loadvm_state_section 4
> qemu_loadvm_state_section_startfull 5(cpu) 0 12
> qemu_loadvm_state_section 4
> qemu_loadvm_state_section_startfull 6(kvm-tpr-opt) 0 1
> qemu-system-x86_64: error while loading state for instance 0x0 of device 'kvm-tpr-opt'
> qemu_loadvm_state_post_main -1
> migrate_set_state new state failed
> migrate_error error=load of migration failed: Operation not permitted
> loadvm_state_cleanup
> qemu-system-x86_64: load of migration failed: Operation not permitted

Check for a mismatch between the qemu args on the source vs dest.  Maybe -cpu.

- Steve