Re: [PATCH V9 00/46] Live Update

[Michael Galaxy <mgalaxy@akamai.com>]

[-- Attachment #1: Type: text/plain, Size: 17181 bytes --]

Hi Steve,

In the current design you have, we have to specify both the command line 
parameter "-migrate-mode-enable cpr-reboot"
*and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".

Is it possible to opt-in to the CPR mode just once over the monitor 
instead of having to specify it twice on the command line?
This would also match the live migration model: You do not need to 
necessarily "opt in" to live migration mode through
a command line parameter, you simply request it when you need to. Can 
CPR behave the same way?

This would also make switching over to a CPR-capable version of QEMU 
much simpler and would even make it work for
existing libvirt-managed guests as their command line parameters would 
no longer need to change. This would allow us to
simply power-off and power-on existing VMs to make them CPR-capable and 
then work on a libvirt patch later when
we're ready to do so.


Comments?

- Michael


On 12/7/22 09:48, Steven Sistare wrote:
> This series desperately needs review in its intersection with live migration.
> The code in other areas has been reviewed and revised multiple times -- thank you!
>
> David, Juan, can you spare some time to review this?  I have done my best to order
> the patches logically (see the labelled groups in this email), and to provide
> complete and clear cover letter and commit messages. Can I do anything to facilitate,
> like doing a code walk through via zoom?
>
> And of course, I welcome anyone's feedback.
>
> Here is the original posting.
>
> https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$  
>
> - Steve
>
> On 7/26/2022 12:09 PM, Steve Sistare wrote:
>> This version of the live update patch series integrates live update into the
>> live migration framework.  The new interfaces are:
>>    * mode (migration parameter)
>>    * cpr-exec-args (migration parameter)
>>    * file (migration URI)
>>    * migrate-mode-enable (command-line argument)
>>    * only-cpr-capable (command-line argument)
>>
>> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
>> save and restore VM state, with minimal guest pause time, so that qemu may be
>> updated to a new version in between.  The caller sets the mode parameter
>> before invoking the migrate or migrate-incoming commands.
>>
>> In cpr-reboot mode, the migrate command saves state to a file, allowing
>> one to quit qemu, reboot to an updated kernel, start an updated version of
>> qemu, and resume via the migrate-incoming command.  The caller must specify
>> a migration URI that writes to and reads from a file.  Unlike normal mode,
>> the use of certain local storage options does not block the migration, but
>> the caller must not modify guest block devices between the quit and restart.
>> The guest RAM memory-backend must be shared, and the @x-ignore-shared
>> migration capability must be set, to avoid saving it to the file.  Guest RAM
>> must be non-volatile across reboot, which can be achieved by backing it with
>> a dax device, or /dev/shm PKRAM as proposed in
>> https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$  
>> but this is not enforced.  The restarted qemu arguments must match those used
>> to initially start qemu, plus the -incoming option.
>>
>> The reboot mode supports vfio devices if the caller first suspends the guest,
>> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
>> drivers' suspend methods flush outstanding requests and re-initialize the
>> devices, and thus there is no device state to save and restore.  After
>> issuing migrate-incoming, the caller must issue a system_wakeup command to
>> resume.
>>
>> In cpr-exec mode, the migrate command saves state to a file and directly
>> exec's a new version of qemu on the same host, replacing the original process
>> while retaining its PID.  The caller must specify a migration URI that writes
>> to and reads from a file, and resumes execution via the migrate-incoming
>> command.  Arguments for the new qemu process are taken from the cpr-exec-args
>> migration parameter, and must include the -incoming option.
>>
>> Guest RAM must be backed by a memory backend with share=on, but cannot be
>> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
>> ram is efficiently preserved in place, albeit with new virtual addresses.
>> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
>> causes secondary guest ram blocks (those not specified on the command line)
>> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
>> their values are saved in special cpr state which is retrieved after exec,
>> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
>> are not migrated, the caller must disable all capabilities related to page
>> and block copy.  The implementation ignores all related parameters.
>>
>> The exec mode supports vfio devices by preserving the vfio container, group,
>> device, and event descriptors across the qemu re-exec, and by updating DMA
>> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
>> VFIO_DMA_MAP_FLAG_VADDR as defined in
>>    https://urldefense.com/v3/__https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3IK0BrTn$  
>> and integrated in Linux kernel 5.12.
>>
>> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
>> The software update is performed while the guest is running to minimize
>> downtime.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ...                         |
>>    -migrate-mode-enable cpr-exec                 |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update qemu
>> (qemu) migrate_set_parameter mode cpr-exec      |
>> (qemu) migrate_set_parameter cpr-exec-args      |
>>    qemu-system-$arch ... -incoming defer         |
>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>>
>> Here is an example of updating the host kernel using reboot mode.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>>    -migrate-mode-enable cpr-reboot               |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update kernel-uek
>> (qemu) migrate_set_parameter mode cpr-reboot    |
>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>> (qemu) quit                                     |
>>                                                  |
>> # systemctl kexec                               |
>> kexec_core: Starting new kernel                 |
>> ...                                             |
>>                                                  |
>> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>>    -incoming defer                               |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>> Changes from V8 to V9:
>>    vfio:
>>      - free all cpr state during unwind in vfio_connect_container
>>      - change cpr_resave_fd to return void, and avoid new unwind cases
>>      - delete incorrect .unmigratable=1 in vmstate handlers
>>      - add route batching in vfio_claim_vectors
>>      - simplified vfio intx cpr code
>>      - fix commit message for 'recover from unmap-all-vaddr failure'
>>      - verify suspended runstate for cpr-reboot mode
>>    Other:
>>      - delete cpr-save, cpr-exec, cpr-load
>>      - delete ram block vmstate handlers that were added in V8
>>      - rename cpr-enable option to migrate-mode-enable
>>      - add file URI for migration
>>      - add mode and cpr-exec-args migration parameters
>>      - add per-mode migration blockers
>>      - add mode checks in migration notifiers
>>      - fix suspended runstate during migration
>>      - replace RAM_ANON flag with RAM_NAMED_FILE
>>      - support memory-backend-epc
>>
>> Steve Sistare (44):
>>    migration: fix populate_vfio_info                  ---  reboot mode  ---
>>    memory: RAM_NAMED_FILE flag
>>    migration: file URI
>>    migration: mode parameter
>>    migration: migrate-enable-mode option
>>    migration: simplify blockers
>>    migration: per-mode blockers
>>    cpr: relax some blockers
>>    cpr: reboot mode
>>
>>    qdev-properties: strList                           ---  exec mode ---
>>    qapi: strList_from_string
>>    qapi: QAPI_LIST_LENGTH
>>    qapi: strv_from_strList
>>    qapi: strList unit tests
>>    migration: cpr-exec-args parameter
>>    migration: simplify notifiers
>>    migration: check mode in notifiers
>>    memory: flat section iterator
>>    oslib: qemu_clear_cloexec
>>    vl: helper to request re-exec
>>    cpr: preserve extra state
>>    cpr: exec mode
>>    cpr: add exec-mode blockers
>>    cpr: ram block blockers
>>    cpr: only-cpr-capable
>>    cpr: Mismatched GPAs fix
>>    hostmem-memfd: cpr support
>>    hostmem-epc: cpr support
>>
>>    pci: export msix_is_pending                       --- vfio for exec ---
>>    vfio-pci: refactor for cpr
>>    vfio-pci: cpr part 1 (fd and dma)
>>    vfio-pci: cpr part 2 (msi)
>>    vfio-pci: cpr part 3 (intx)
>>    vfio-pci: recover from unmap-all-vaddr failure
>>
>>    chardev: cpr framework                            --- misc for exec ---
>>    chardev: cpr for simple devices
>>    chardev: cpr for pty
>>    python/machine: QEMUMachine full_args
>>    python/machine: QEMUMachine reopen_qmp_connection
>>    tests/avocado: add cpr regression test
>>
>>    vl: start on wakeup request                       --- vfio for reboot ---
>>    migration: fix suspended runstate
>>    migration: notifier error reporting
>>    vfio: allow cpr-reboot migration if suspended
>>
>> Mark Kanda, Steve Sistare (2):
>>    vhost: reset vhost devices for cpr
>>    chardev: cpr for sockets
>>
>>   MAINTAINERS                         |  14 ++
>>   accel/xen/xen-all.c                 |   3 +
>>   backends/hostmem-epc.c              |  18 +-
>>   backends/hostmem-file.c             |   1 +
>>   backends/hostmem-memfd.c            |  22 ++-
>>   backends/tpm/tpm_emulator.c         |  11 +-
>>   block/parallels.c                   |   7 +-
>>   block/qcow.c                        |   7 +-
>>   block/vdi.c                         |   7 +-
>>   block/vhdx.c                        |   7 +-
>>   block/vmdk.c                        |   7 +-
>>   block/vpc.c                         |   7 +-
>>   block/vvfat.c                       |   7 +-
>>   chardev/char-mux.c                  |   1 +
>>   chardev/char-null.c                 |   1 +
>>   chardev/char-pty.c                  |  16 +-
>>   chardev/char-serial.c               |   1 +
>>   chardev/char-socket.c               |  48 +++++
>>   chardev/char-stdio.c                |  31 +++
>>   chardev/char.c                      |  49 ++++-
>>   dump/dump.c                         |   4 +-
>>   gdbstub.c                           |   1 +
>>   hmp-commands.hx                     |   2 +-
>>   hw/9pfs/9p.c                        |  11 +-
>>   hw/core/qdev-properties-system.c    |  12 ++
>>   hw/core/qdev-properties.c           |  44 +++++
>>   hw/display/virtio-gpu-base.c        |   8 +-
>>   hw/intc/arm_gic_kvm.c               |   3 +-
>>   hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>>   hw/intc/arm_gicv3_kvm.c             |   3 +-
>>   hw/misc/ivshmem.c                   |   8 +-
>>   hw/net/virtio-net.c                 |  10 +-
>>   hw/pci/msix.c                       |   2 +-
>>   hw/pci/pci.c                        |  12 ++
>>   hw/ppc/pef.c                        |   2 +-
>>   hw/ppc/spapr.c                      |   2 +-
>>   hw/ppc/spapr_events.c               |   2 +-
>>   hw/ppc/spapr_rtas.c                 |   2 +-
>>   hw/remote/proxy.c                   |   7 +-
>>   hw/s390x/s390-virtio-ccw.c          |   9 +-
>>   hw/scsi/vhost-scsi.c                |   9 +-
>>   hw/vfio/common.c                    | 235 +++++++++++++++++++----
>>   hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>>   hw/vfio/meson.build                 |   1 +
>>   hw/vfio/migration.c                 |  23 +--
>>   hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>>   hw/vfio/trace-events                |   1 +
>>   hw/virtio/vhost-vdpa.c              |   6 +-
>>   hw/virtio/vhost.c                   |  32 +++-
>>   include/chardev/char-socket.h       |   1 +
>>   include/chardev/char.h              |   5 +
>>   include/exec/memory.h               |  48 +++++
>>   include/exec/ram_addr.h             |   1 +
>>   include/exec/ramblock.h             |   1 +
>>   include/hw/pci/msix.h               |   1 +
>>   include/hw/qdev-properties-system.h |   4 +
>>   include/hw/qdev-properties.h        |   3 +
>>   include/hw/vfio/vfio-common.h       |  12 ++
>>   include/hw/virtio/vhost.h           |   1 +
>>   include/migration/blocker.h         |  69 ++++++-
>>   include/migration/cpr-state.h       |  30 +++
>>   include/migration/cpr.h             |  20 ++
>>   include/migration/misc.h            |  13 +-
>>   include/migration/vmstate.h         |   2 +
>>   include/qapi/util.h                 |  28 +++
>>   include/qemu/osdep.h                |   9 +
>>   include/sysemu/runstate.h           |   2 +
>>   migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>>   migration/cpr.c                     |  85 +++++++++
>>   migration/file.c                    |  62 ++++++
>>   migration/file.h                    |  14 ++
>>   migration/meson.build               |   3 +
>>   migration/migration.c               | 268 +++++++++++++++++++++++---
>>   migration/ram.c                     |  24 ++-
>>   migration/target.c                  |   1 +
>>   migration/trace-events              |  12 ++
>>   monitor/hmp-cmds.c                  |  59 +++---
>>   monitor/hmp.c                       |   3 +
>>   monitor/qmp.c                       |   4 +
>>   python/qemu/machine/machine.py      |  14 ++
>>   qapi/char.json                      |   7 +-
>>   qapi/migration.json                 |  68 ++++++-
>>   qapi/qapi-util.c                    |  37 ++++
>>   qemu-options.hx                     |  50 ++++-
>>   replay/replay.c                     |   4 +
>>   softmmu/memory.c                    |  31 ++-
>>   softmmu/physmem.c                   | 100 +++++++++-
>>   softmmu/runstate.c                  |  42 ++++-
>>   softmmu/vl.c                        |  10 +
>>   stubs/cpr-state.c                   |  26 +++
>>   stubs/meson.build                   |   2 +
>>   stubs/migr-blocker.c                |   9 +-
>>   stubs/migration.c                   |  33 ++++
>>   target/i386/kvm/kvm.c               |   8 +-
>>   target/i386/nvmm/nvmm-all.c         |   4 +-
>>   target/i386/sev.c                   |   2 +-
>>   target/i386/whpx/whpx-all.c         |   3 +-
>>   tests/avocado/cpr.py                | 176 ++++++++++++++++++
>>   tests/unit/meson.build              |   1 +
>>   tests/unit/test-strlist.c           |  81 ++++++++
>>   trace-events                        |   1 +
>>   ui/spice-core.c                     |   5 +-
>>   ui/vdagent.c                        |   5 +-
>>   util/oslib-posix.c                  |   9 +
>>   util/oslib-win32.c                  |   4 +
>>   105 files changed, 2781 insertions(+), 330 deletions(-)
>>   create mode 100644 hw/vfio/cpr.c
>>   create mode 100644 include/migration/cpr-state.h
>>   create mode 100644 include/migration/cpr.h
>>   create mode 100644 migration/cpr-state.c
>>   create mode 100644 migration/cpr.c
>>   create mode 100644 migration/file.c
>>   create mode 100644 migration/file.h
>>   create mode 100644 stubs/cpr-state.c
>>   create mode 100644 stubs/migration.c
>>   create mode 100644 tests/avocado/cpr.py
>>   create mode 100644 tests/unit/test-strlist.c
>>

[-- Attachment #2: Type: text/html, Size: 18093 bytes --]