[RFC PATCH 00/26] Confidential guest live migration

[Dov Murik <dovmurik@linux.vnet.ibm.com>]

This is an RFC series for fast migration of confidential guests using an
in-guest migration helper that lives in OVMF.  QEMU VM live migration
needs to read source VM's RAM and write it in the target VM; this
mechanism doesn't work when the guest memory is encrypted or QEMU is
prevented from reading it in another way.  In order to support live
migration in such scenarios, we introduce an in-guest migration helper
which can securely extract RAM content from the guest in order to send
it to the target.  The migration helper is implemented as part of the
VM's firmware in OVMF.

We've implemented and tested this on AMD SEV, but expect most of the
processes can be used with other technologies that prevent direct access
of hypervisor to the guest's memory.  Specifically, we don't use SEV's
PSP migration commands (SEV_SEND_START, SEV_RECEIVE_START, etc) at all.

Corresponding RFC patches for OVMF have been posted by my colleague
Tobin Feldman-Fitzthum on edk2-devel.  Those include the crux of the
migration helper: a mailbox protocol over a shared memory page which
allows communication between QEMU and the migration helper.  In the
source VM this is used to read a page and encrypt it for transport; in
the target it is used to decrypt the incoming page and storing the
content in the correct address in the guest memory.  All encryption and
decryption operations occur inside the trusted context in the VM, and
therefore the VM's memory plaintext content is never accessible to the
hosts participating in the migration.

In order to allow OVMF to run the migration helper in parallel to the
guest OS, we introduce the notion of auxiliary vcpus, which are usable
for OVMF but are hidden from the guest OS.  These might have other
future uses for in-guest operations/agents.

In the target VM we need the migration handler running to receive
incoming RAM pages; to achieve that, we boot the VM into OVMF with a
special fw_cfg value that causes OVMF to not boot the guest OS; we then
allow QEMU to receive an incoming migration by issuing a new
start-migrate-incoming QMP command.

The confidential RAM migration requires checking whether a given guest
RAM page is encrypted or not.  This is currently achieved using AMD's
patches which track the encryption status of guest pages in KVM, using
hypercalls from OVMF and guest Linux to report changes of such status.
The QEMU side of these patches is included as the first two patches in
this series.  The concrete implementation of this page encryption tracking
is currently in flux in the KVM mailing list, but the underlying
implementation doesn't affect our confidential RAM migration as long as
it can check whether a given guest address is encrypted.

List of patches in this series:

1-2: reposting AMD encrypted page bitmap support.
3-11: introduce the notion of auxiliary vcpus.
12-21: introduce the migration specifics.
22-23: fix devices issues when loading state into a live VM
24: introduce the start-migrate-incoming QMP command to switch the
target into accepting the incoming migration.
25: remove SEV migration blocker
26: add documentation


Brijesh Singh (1):
  kvm: add support to sync the page encryption state bitmap

Dov Murik (21):
  linux-headers: Add definitions of KVM page encryption bitmap ioctls
  machine: Add auxcpus=N suboption to -smp
  hw/boards: Add aux flag to CPUArchId
  hw/i386: Mark auxiliary vcpus in possible_cpus
  cpu: Add boolean aux field to CPUState
  hw/i386: Set CPUState.aux=true for auxiliary vcpus
  softmmu: Don't sync aux vcpus in pre_loadvm
  softmmu: Add cpu_synchronize_without_aux_post_init
  migration: Add helpers to save confidential RAM
  migration: Add helpers to load confidential RAM
  migration: Introduce gpa_inside_migration_helper_shared_area
  migration: Save confidential guest RAM using migration helper
  migration: Load confidential guest RAM using migration helper
  migration: Stop VM after loading confidential RAM
  migration: Don't sync vcpus when migrating confidential guests
  migration: When starting target, don't sync auxiliary vcpus
  hw/isa/lpc_ich9: Allow updating an already-running VM
  target/i386: Re-sync kvm-clock after confidential guest migration
  migration: Add start-migrate-incoming QMP command
  target/i386: SEV: Allow migration unless there are no aux vcpus
  docs: Add confidential guest live migration documentation

Tobin Feldman-Fitzthum (4):
  hw/acpi: Don't include auxiliary vcpus in ACPI tables
  softmmu: Add pause_all_vcpus_except_aux
  migration: Stop non-aux vcpus before copying the last pages
  migration: Call migration handler cleanup routines

 docs/confidential-guest-live-migration.rst | 142 ++++++++++++
 docs/confidential-guest-support.txt        |   5 +
 docs/index.rst                             |   1 +
 qapi/migration.json                        |  26 +++
 include/exec/ram_addr.h                    | 197 ++++++++++++++++
 include/exec/ramblock.h                    |   3 +
 include/exec/ramlist.h                     |   3 +-
 include/hw/boards.h                        |   3 +
 include/hw/core/cpu.h                      |   2 +
 include/hw/i386/x86.h                      |   2 +-
 include/sysemu/cpus.h                      |   2 +
 linux-headers/linux/kvm.h                  |  13 ++
 migration/confidential-ram.h               |  23 ++
 accel/kvm/kvm-all.c                        |  43 ++++
 hw/acpi/cpu.c                              |  10 +
 hw/core/cpu.c                              |   1 +
 hw/core/machine.c                          |   7 +
 hw/i386/acpi-build.c                       |   5 +
 hw/i386/acpi-common.c                      |   5 +
 hw/i386/pc.c                               |   7 +
 hw/i386/x86.c                              |  10 +-
 hw/isa/lpc_ich9.c                          |   3 +-
 migration/confidential-ram.c               | 258 +++++++++++++++++++++
 migration/migration.c                      |  18 +-
 migration/ram.c                            | 135 ++++++++++-
 migration/savevm.c                         |  13 +-
 softmmu/cpus.c                             |  68 +++++-
 softmmu/runstate.c                         |   1 +
 softmmu/vl.c                               |   3 +
 target/i386/machine.c                      |   9 +
 target/i386/sev.c                          |  25 +-
 migration/meson.build                      |   6 +-
 migration/trace-events                     |   4 +
 33 files changed, 1027 insertions(+), 26 deletions(-)
 create mode 100644 docs/confidential-guest-live-migration.rst
 create mode 100644 migration/confidential-ram.h
 create mode 100644 migration/confidential-ram.c


base-commit: 00d8ba9e0d62ea1c7459c25aeabf9c8bb7659462
-- 
2.20.1