All of lore.kernel.org
 help / color / mirror / Atom feed
From: Fabiano Rosas <farosas@suse.de>
To: Dongli Zhang <dongli.zhang@oracle.com>, qemu-devel@nongnu.org
Cc: david@kernel.org, imammedo@redhat.com,
	maciej.szmigiero@oracle.com, peterx@redhat.com,
	mark.kanda@oracle.com, bchaney@akamai.com,
	pierrick.bouvier@oss.qualcomm.com, alex@shazbot.org,
	clg@redhat.com, yi.l.liu@intel.com, eric.auger@redhat.com,
	zhenzhong.duan@intel.com, mst@redhat.com, cohuck@redhat.com,
	pbonzini@redhat.com, armbru@redhat.com, eblake@redhat.com
Subject: Re: [PATCH 00/21] migration/cpr: introduce cpr-luo mode to support KHO/LUO
Date: Tue, 17 Nov 2026 16:01:45 -0300	[thread overview]
Message-ID: <875wxvs2na.fsf@suse.de> (raw)
In-Reply-To: <95d4d77e-7b30-463e-96a1-6add4495e451@oracle.com>

Dongli Zhang <dongli.zhang@oracle.com> writes:

> I am rebasing the patchset on top of the most recent QEMU commit, especially
> "migration/cpr: support vhost-vsock devices".
>
> Meanwhile, any feedback on the proposal is appreciated.
>

Sorry for the delay, we're all swimming in slop these days. The only
feedback I have at the moment is that this feature needs tests. What's
your plan in this regard? Can this run with nested virtualization? Or
perhaps is there a way to mock the kernel interfaces and ensure QEMU
does the right thing even without them? We don't need much, but the code
definitely needs to be exercised somehow.

> So far, the main idea is to reuse as much code as possible from cpr-exec and
> cpr-transfer.
>
> 1. Introduce the new cpr-luo mode.
> 2. Use a UUID to identify a LUO session.
> 3. Introduce APIs to create, preserve, and retrieve sessions and fds.
> 4. As LUO uses the name, id, and token instead of an fd, create an independent
> VMStateDescription rather than reuse the one for cpr-exec and cpr-transfer.
> 5. Compute the token when transforming entries from the hash table into the list.
> 6. Preserve fds on the source. Follow the approach used in cpr-exec and
> cpr-transfer.
> 7. Retrieve fds on the target. Follow the approach used in cpr-exec and
> cpr-transfer. The tricky part is where to set the cpr incoming mode.
> 8. There is no need to modify save_fd() or load_fd(), as we only need to migrate
> the corresponding name and token, not the fds.
> 9. Handle corner-case errors.
> 10. Block VFIO.
>
> Thank you very much!
>
> Dongli Zhang
>
> On Mon, Oct 12, 2026 4:55:56PM -0700, Dongli Zhang wrote:
>> This series introduces cpr-luo, a CPR migration mode that uses the Linux Live
>> Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host kernel
>> updates through kexec. On the outgoing side, QEMU saves CPR state in a memfd and
>> preserves
>> 
>> This series introduces cpr-luo, a CPR migration mode that uses the Linux
>> Live Update Orchestrator (LUO) and Kexec Handover (KHO) to support KVM host
>> kernel updates through kexec.
>> 
>> On the outgoing side, QEMU saves CPR state in a memfd and preserves that fd
>> together with the RAMBlock backing fds in LUO. KHO carries the preserved LUO
>> state into the new kernel. The incoming QEMU retrieves the CPR state and
>> RAMBlock backing fds from LUO after kexec.
>> 
>> The regular migration stream carries device vmstate.
>> 
>> Although VFIO is currently blocked, I have an offline implementation that
>> skips cpr-luo mode in cpr_is_incoming(), skips precopy, and carefully handles
>> monitor_fd_param() although I never use it, so that VFIO is always
>> reconstructed. It works with both iommufd and legacy for mlx5_vfio_pci by
>> saving the VFIO and vendor specific state in vmstate as well.
>> 
>> The result is that guest RAM remains available across a kexec-based host
>> kernel update without copying its contents through the migration stream.
>> 
>> Here is the main idea of the patchset:
>> 
>> 1. Introduce the new cpr-luo mode.
>> 2. Use a UUID to identify a LUO session.
>> 3. Introduce APIs to create, preserve, and retrieve sessions and fds.
>> 4. As LUO uses the name, id, and token instead of an fd, create an
>> independent VMStateDescription rather than reuse the one for cpr-exec and
>> cpr-transfer.
>> 5. Compute the token when transforming entries from the hash table into the
>> list.
>> 6. Preserve fds on the source. Follow the approach used in cpr-exec and
>> cpr-transfer.
>> 7. Retrieve fds on the target. Follow the approach used in cpr-exec and
>> cpr-transfer. The tricky part is where to set the cpr incoming mode.
>> 8. There is no need to modify save_fd() or load_fd(), as we only need to
>> migrate the corresponding name and token, not the fds.
>> 9. Handle corner-case errors.
>> 10. Block VFIO.
>> 
>> I may need to rebase the current patchset on top of the patches below,
>> from which I learned quite a lot in each iteration. While working on this
>> patchset, I identified some cleanup and code issues in cpr-exec and
>> cpr-transfer. I may send fixes for them separately.
>> 
>> [PATCH v16 00/22] migration/cpr: support vhost-vsock devices
>> https://urldefense.com/v3/__https://lore.kernel.org/all/20261012-up-vsock-vhost-
>> v5-v16-0-36c38bd15ffc@virtuozzo.com/__;!!ACWV5N9M2RV99hQ!N58C4d74-
>> zpeHeXtxKWUAIRgf18-Xzk7_V7p49rzvh29deg5g2oBeAn0R5b89vJzLBPv6rkzjTo-8F0Iy16Z0oRy$ <https://urldefense.com/v3/__https://lore.kernel.org/all/20261012-up-vsock-vhost-v5-v16-0-36c38bd15ffc@virtuozzo.com/__;!!ACWV5N9M2RV99hQ!N58C4d74-zpeHeXtxKWUAIRgf18-Xzk7_V7p49rzvh29deg5g2oBeAn0R5b89vJzLBPv6rkzjTo-8F0Iy16Z0oRy$>
>> 
>> Example:
>> 
>> Add "kho=on liveupdate=on" to host kernel cmdline.
>> 
>> Create QEMU instance with memfd.
>> 
>> qemu-system-x86_64 \
>> -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \
>> -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \
>> -accel kvm -cpu host \
>> -name debug-threads=on \
>> -m 8G -smp 4 \
>> -object memory-backend-memfd,id=ram0,size=8G,share=on \
>> -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \
>> -drive file=boot.qcow2,if=none,id=drive0 \
>> -netdev user,id=net0,hostfwd=tcp::5028-:22 \
>> -device virtio-net-pci,netdev=net0 \
>> -vnc :8 -monitor stdio
>> 
>> Don't terminate QEMU process after cpr-luo migration.
>> 
>> (qemu) migrate_set_parameter mode cpr-luo
>> (qemu) migrate -d file:cpr-luo.state
>> (qemu) info status
>> VM status: paused (postmigrate)
>> 
>> Use kexec to reboot the KVM host.
>> 
>> host# kexec -s -l /boot/vmlinuz-$(uname -r) --initrd=/boot/initramfs-$(uname -r).img --reuse-cmdline
>> host# sync
>> host# kexec -e
>> 
>> Resume QEMU instance after kexec reboot.
>> 
>> qemu-system-x86_64 \
>> -uuid 6f1b9f5a-6b92-4a5d-ae57-7a0ddabc1234 \
>> -machine q35,dump-guest-core=off,memory-backend=ram0,aux-ram-share=on \
>> -accel kvm -cpu host \
>> -name debug-threads=on \
>> -m 8G -smp 4 \
>> -object memory-backend-memfd,id=ram0,size=8G,share=on \
>> -device virtio-blk-pci,drive=drive0,id=virtblk0,num-queues=4 \
>> -drive file=boot.qcow2,if=none,id=drive0 \
>> -netdev user,id=net0,hostfwd=tcp::5028-:22 \
>> -device virtio-net-pci,netdev=net0 \
>> -vnc :8 -monitor stdio \
>> -incoming cpr-luo:defer
>> 
>> (qemu) migrate_incoming file:cpr-luo.state
>> (qemu) info status
>> VM status: running
>> 
>> 
>> Dongli Zhang (21):
>>   qapi/migration: add cpr-luo migration mode
>>   migration/cpr: move CprFd definition to header file
>>   scripts/update-linux-headers: Add liveupdate.h
>>   linux-headers: import liveupdate UAPI header
>>   migration/cpr-luo: require a non-null machine UUID
>>   migration/cpr-luo: add LUO session management
>>   migration/cpr-luo: add LUO fd management helpers
>>   migration/cpr-luo: track LUO fds
>>   migration/cpr-luo: add LUO fd VMStateDescription
>>   migration/cpr-luo: add tokens for LUO fds
>>   migration/cpr: move QEMUFile fd helpers to common code
>>   migration/cpr-luo: add state fd helpers
>>   migration/cpr: add more arguments for cpr_walk_fd()
>>   migration/cpr-luo: add helpers to preserve and retrieve all LUO fds
>>   migration/cpr-luo: enable cpr-luo mode for source side preservation
>>   migration/cpr-luo: enable cpr-luo mode for target side retrieval
>>   migration/cpr-luo: abort session when migration fails
>>   system/physmem: add cpr-luo blocker
>>   monitor/qmp: close LUO session when resuming source QEMU
>>   vfio: block cpr-luo migration
>>   migration: cpr-luo documentation
>> 
>>  backends/hostmem-memfd.c         |   1 +
>>  backends/hostmem-shm.c           |   1 +
>>  docs/devel/migration/CPR.rst     |  64 +++++++
>>  hw/vfio/cpr-iommufd.c            |  17 +-
>>  hw/vfio/cpr-legacy.c             |   8 +
>>  include/hw/vfio/vfio-cpr.h       |   1 +
>>  include/migration/cpr.h          |  37 +++-
>>  include/system/iommufd.h         |   1 +
>>  include/system/system.h          |   2 +
>>  linux-headers/linux/liveupdate.h | 239 +++++++++++++++++++++++++
>>  migration/cpr-exec.c             |  32 +---
>>  migration/cpr-luo-stub.c         |  78 ++++++++
>>  migration/cpr-luo.c              | 328 ++++++++++++++++++++++++++++++++++
>>  migration/cpr.c                  | 126 +++++++++++--
>>  migration/meson.build            |   6 +
>>  migration/migration.c            |  12 +-
>>  migration/options.c              |   6 +
>>  migration/ram.c                  |   1 +
>>  monitor/qmp-cmds.c               |   7 +
>>  qapi/migration.json              |  11 +-
>>  scripts/update-linux-headers.sh  |   2 +-
>>  system/globals.c                 |   5 +
>>  system/physmem.c                 |   9 +-
>>  system/vl.c                      |   9 +
>>  24 files changed, 950 insertions(+), 53 deletions(-)
>> 
>> base-commit: 3377670ee96b5a68361016680a95b9013709d4b7
>> 
>> Thank you very much!
>> 
>> Dongli Zhang
>> 
>> 


  reply	other threads:[~2026-11-17 19:02 UTC|newest]

Thread overview: 33+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-12 23:55 [PATCH 00/21] migration/cpr: introduce cpr-luo mode to support KHO/LUO Dongli Zhang
2026-10-12 23:55 ` [PATCH 01/21] qapi/migration: add cpr-luo migration mode Dongli Zhang
2026-10-14  7:42   ` Markus Armbruster
2026-10-14 15:52     ` Dongli Zhang
2026-10-15  5:06       ` Markus Armbruster
2026-10-15 20:37         ` Dongli Zhang
2026-10-16  5:04           ` Markus Armbruster
2026-10-12 23:55 ` [PATCH 02/21] migration/cpr: move CprFd definition to header file Dongli Zhang
2026-10-12 23:55 ` [PATCH 03/21] scripts/update-linux-headers: Add liveupdate.h Dongli Zhang
2026-10-12 23:56 ` [PATCH 04/21] linux-headers: import liveupdate UAPI header Dongli Zhang
2026-10-13  6:35   ` Cornelia Huck
2026-10-13 21:09     ` Dongli Zhang
2026-10-12 23:56 ` [PATCH 05/21] migration/cpr-luo: require a non-null machine UUID Dongli Zhang
2026-10-12 23:56 ` [PATCH 06/21] migration/cpr-luo: add LUO session management Dongli Zhang
2026-10-12 23:56 ` [PATCH 07/21] migration/cpr-luo: add LUO fd management helpers Dongli Zhang
2026-10-12 23:56 ` [PATCH 08/21] migration/cpr-luo: track LUO fds Dongli Zhang
2026-10-12 23:56 ` [PATCH 09/21] migration/cpr-luo: add LUO fd VMStateDescription Dongli Zhang
2026-10-12 23:56 ` [PATCH 10/21] migration/cpr-luo: add tokens for LUO fds Dongli Zhang
2026-10-12 23:56 ` [PATCH 11/21] migration/cpr: move QEMUFile fd helpers to common code Dongli Zhang
2026-10-12 23:56 ` [PATCH 12/21] migration/cpr-luo: add state fd helpers Dongli Zhang
2026-10-12 23:56 ` [PATCH 13/21] migration/cpr: add more arguments for cpr_walk_fd() Dongli Zhang
2026-10-12 23:56 ` [PATCH 14/21] migration/cpr-luo: add helpers to preserve and retrieve all LUO fds Dongli Zhang
2026-10-12 23:56 ` [PATCH 15/21] migration/cpr-luo: enable cpr-luo mode for source side preservation Dongli Zhang
2026-10-12 23:56 ` [PATCH 16/21] migration/cpr-luo: enable cpr-luo mode for target side retrieval Dongli Zhang
2026-10-12 23:56 ` [PATCH 17/21] migration/cpr-luo: abort session when migration fails Dongli Zhang
2026-10-12 23:56 ` [PATCH 18/21] system/physmem: add cpr-luo blocker Dongli Zhang
2026-10-12 23:56 ` [PATCH 19/21] monitor/qmp: close LUO session when resuming source QEMU Dongli Zhang
2026-10-12 23:56 ` [PATCH 20/21] vfio: block cpr-luo migration Dongli Zhang
2026-10-12 23:56 ` [PATCH 21/21] migration: cpr-luo documentation Dongli Zhang
2026-11-09 10:46 ` [PATCH 00/21] migration/cpr: introduce cpr-luo mode to support KHO/LUO Dongli Zhang
2026-11-17 19:01   ` Fabiano Rosas [this message]
2026-11-17 19:25     ` Peter Xu
2026-11-18  2:02       ` Dongli Zhang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=875wxvs2na.fsf@suse.de \
    --to=farosas@suse.de \
    --cc=alex@shazbot.org \
    --cc=armbru@redhat.com \
    --cc=bchaney@akamai.com \
    --cc=clg@redhat.com \
    --cc=cohuck@redhat.com \
    --cc=david@kernel.org \
    --cc=dongli.zhang@oracle.com \
    --cc=eblake@redhat.com \
    --cc=eric.auger@redhat.com \
    --cc=imammedo@redhat.com \
    --cc=maciej.szmigiero@oracle.com \
    --cc=mark.kanda@oracle.com \
    --cc=mst@redhat.com \
    --cc=pbonzini@redhat.com \
    --cc=peterx@redhat.com \
    --cc=pierrick.bouvier@oss.qualcomm.com \
    --cc=qemu-devel@nongnu.org \
    --cc=yi.l.liu@intel.com \
    --cc=zhenzhong.duan@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.