[PATCH v4 00/11] migration: fast snapshot load

[Aadeshveer Singh <aadeshveer07@gmail.com>]

This series implements a "fast snapshot load" mechanism to
significantly reduce the perceived resume time of a VM from a snapshot
file.

Currently, resuming a VM from a snapshot file requires loading all RAM
pages into the QEMU instance before execution begins. This extension
allows the user to run the VM nearly instantly by loading only the
required device states up front and loading RAM pages lazily, by
trapping access to pages that have not yet been loaded.

Using the Linux userfaultfd syscall, a fault thread catches all page
faults caused by the guest and loads in the pages required to keep
the VM running. Concurrently, an eager background thread iteratively
loads all remaining pages into RAM so the guest does not have to
depend on the fault thread indefinitely.

Much of code is reused from postcopy for fault handling and precopy
for reading mapped ram file. Implementation revolves around two
threads named the fault thread and eager load thread. Fault thread as
name suggests catches page faults by the guest and serves them using
userfaultfd. Postcopy fault thread is reused but instead of requesting
source for a page it loads the page directly by reading from file. In
order to remove the dependency of guest on fault thread indefinitely
the eager load thread loads in the entire RAM sequentially, and after
iterating through the entire RAM signals fault thread to exit and
calls cleanup.

In order to prevent the case of a page being loaded twice(in the
case when eager load thread is loading it and fault thread also
tries to serve fault on same page) a bitmap called pending_bmap is
used to track pages which are pending and not being loaded by any
thread. Atomic operations on this bitmap allows coordination between
threads to prevent any unwanted behaviours

This patch series was tested on a single machine:
host OS   : Fedora 44
host RAM  : 32GB DDR5(4KB pagesize)
host CPU  : Intel Ultra9 185H(22 cores, x86_64)

Guests tested:
- KVM enabled
  guest OS  : Fedora 44
  guest RAM : 16GB(4KB pagesize)
  guest CPU : 4 cores(x86_64)
- emulated power PC
  guest OS  : Debian 10
  guest RAM : 2GB(64KB pagesize)
  guest CPU : 2 cores(ppc)
- host using 2MB hugepages
  guest OS  : Debian 13
  guest RAM : 16GB(16KB pagesize)
  guest CPU : 4 cores(x86_64)

Future direction:
- Add support for multifd
- Add support for vhost-user

---
v3 -> v4

- Set errp in qemu_get_buffer_at on failing if file is in failure
  state to ensure errp is set if a function returns failure
- Update logic for postcopy_mapped_ram_load_page in patch 7 to support
  all possibilities of guest and host pagesizes as pointed by Peter
- This also improves on correctness by not relying on mapped ram
  filling zeropages with zero values as pointed by Peter
- Simplified postcopy_ram_eager_load_thread logic in patch 8 for exit
  as suggested by Peter
- Use a more generalized language in documentation overview as
  suggested by Peter
- Added a check against vhost user in case of fast snapshot load in
  options.c(Patch 9) as suggested by Peter

Aadeshveer Singh (11):
  migration: Propagate error in postcopy setup functions
  migration: Extract blocktime marking helper
  migration: Rename postcopy_listen_thread_bh
  migration: Use file_bmap for RAMBlock during incoming file load
  migration: Make qemu_get_buffer_at() thread-safe
  migration: add RAMBlock field and helper for fast snapshot load
  migration: add support for fault thread to load pages from disk
  migration: add eager load thread and setup for fast snapshot load
  migration: update capability conflict test for postcopy-ram+mapped-ram
  migration/tests: Add test for fast snapshot load
  docs/migration: Add documentation for fast snapshot load feature

 docs/devel/migration/fast-snapshot-load.rst |  82 +++++
 docs/devel/migration/features.rst           |   1 +
 include/qemu/notify.h                       |   2 +
 include/system/ramblock.h                   |   6 +
 migration/migration.c                       |  60 ++--
 migration/migration.h                       |   5 +
 migration/options.c                         |  20 +-
 migration/postcopy-ram.c                    | 324 +++++++++++++++++---
 migration/postcopy-ram.h                    |   8 +-
 migration/qemu-file.c                       |  11 +-
 migration/qemu-file.h                       |   4 +-
 migration/ram.c                             |  94 +++++-
 migration/savevm.c                          |  16 +
 migration/savevm.h                          |   2 +
 migration/trace-events                      |   2 +
 tests/qtest/migration/file-tests.c          |  24 ++
 tests/qtest/migration/misc-tests.c          |  52 ----
 util/notify.c                               |   5 +
 18 files changed, 575 insertions(+), 143 deletions(-)
 create mode 100644 docs/devel/migration/fast-snapshot-load.rst

-- 
2.55.0