[RFC PATCH v2 0/7] migration: fast snapshot load

[Aadeshveer Singh <aadeshveer07@gmail.com>]

This RFC implements a "fast snapshot load" mechanism to significantly
reduce the perceived resume time of a VM from a snapshot file.

Currently, resuming a VM from a snapshot file requires loading all RAM
pages into the QEMU instance before execution begins. This extension
allows the user to run the VM nearly instantly by loading only the
required device states up front and loading RAM pages lazily, by
trapping access to pages that have not yet been loaded.

Using the Linux userfaultfd syscall, a fault thread catches all page
faults caused by the guest and loads in the pages required to keep
the VM running. Concurrently, an eager background thread iteratively
loads all remaining pages into RAM so the guest does not have to
depend on the fault thread indefinitely.

Much of code is reused from postcopy for fault handling and precopy
for reading mapped ram file. Implementation revolves around two
threads named the fault thread and eager load thread. Fault thread as
name suggests catches page faults by the guest and serves them using
userfaultfd. Postcopy fault thread is reused but instead of requesting
source for a page it loads the page directly by reading from file. In
order to remove the dependency of guest on fault thread indefinitely
the eager load thread loads in the entire RAM sequentially, and after
iterating through the entire RAM signals fault thread to exit and
calls cleanup.

In order to prevent the case of a page being loaded twice(in the
case when eager load thread is loading it and fault thread also
tries to serve fault on same page) a bitmap called pending_bmap is
used to track pages which are pending and not being loaded by any
thread. Atomic operations on this bitmap allows coordination between
threads to prevent any unwanted behaviours

This patch was tested using a Debian 13 bare minimum system and Fedora
44 KDE, snapshots for both are loaded successfully with no error.

Next Steps:
- Add testing framework, in qtest and unit tests
- Update documentation

Future direction:
- Add support for hugepages
- Add support for multifd
- Add support for vhost-user

---
v1 -> v2

- Structure Change: Expanded from 5 to 7 patches
- Move test removal patch from last to first as suggested by Peter
- Patch 2 to 4 are meant to be non functional changes meant to prepare
  for actual feature Implementation
- Patch 3 and 4 from last series were squashed together into the
  final patch
- Improve error handling in postcopy_ram_incoming_setup() and
  postcopy_temp_pages_setup() for easier error handling
- Use pending_bmap for loading eliminating need of nonzeropages map
  as suggested by Peter
- Keep exit from qemu_get_buffer_at() in case the file has error as
  pointed by Peter
- Eliminate migrate_fast_snapshot_load() to prevent complication of
  capability matrix as pointed by Fabiano
- Add function ram_should_load_postcopy_pages() to improve readability
  as suggested by Peter
- Fixed comment for postcopy_mapped_ram_load_page() to align with
  auto documentation as pointed by peter
- Modify postcopy_mapped_ram_load_page() to return boolean for success
  and use errp for modular error handling
- Replaced use of TARGET_PAGE_SIZE with qemu_ram_pagesize(rb) for
  modularity and ease of adding hugepages support
- Add blocktime support directly and fixed a race condition on
  received map
- Add migration_incoming_has_postcopy_thread() to improve readability
  as suggested by Peter
- Moved the setup and run logic for fast snapshot from
  qemu_loadvm_state() to process_incoming_migration_co() for cleanliness
  as suggested by Peter
- Rename eager load thread to snapshot_load as suggested by Peter
- Use RAM_CHANNEL_PRECOPY/RAM_CHANNEL_POSTCOPY instead of hardcoded
  values for eager thread and fault thread load channels as suggested
  by Peter
- Reused postcopy_listen_thread_bh(and rename it to
  postcopy_incoming_complete_bh) as suggested by Peter
- Removed the bypass on process_incoming_migration_bh scheduling as
  pointed by Peter and Fabiano

Aadeshveer Singh (7):
  migration/tests: remove capability conflict test
    postcopy-ram+mapped-ram
  migration: Propagate error in postcopy setup functions
  migration: Use file_bmap for RAMBlock during incoming file load
  migration: Make qemu_get_buffer_at() thread-safe
  migration: add RAMBlock field and helper for fast snapshot load
  migration: add support for fault thread to load pages from disk
  migration: add eager load thread and setup for fast snapshot load

 include/system/ramblock.h          |   6 +
 migration/migration.c              |  33 +++-
 migration/migration.h              |   5 +
 migration/options.c                |   6 -
 migration/postcopy-ram.c           | 236 +++++++++++++++++++++++------
 migration/postcopy-ram.h           |   4 +-
 migration/qemu-file.c              |   5 +-
 migration/ram.c                    |  88 +++++++++--
 migration/savevm.c                 |  26 ++++
 migration/savevm.h                 |   2 +
 migration/trace-events             |   2 +
 tests/qtest/migration/misc-tests.c |  52 -------
 12 files changed, 347 insertions(+), 118 deletions(-)

-- 
2.54.0