[PATCH RFC v2 0/8] Live migration for VMs with QEMU backed local storage

[Bruno Alvisio <bruno.alvisio@gmail.com>]

I am reviving this thread about the migration of VMs with local storage. I have worked on a solution to be able to migrate VMs that use QEMU as the backend disk driver. I have adapted the migration flow and piggybacked on the “drive-mirroring” capability already provided by QEMU.

Overview
1. The “xl migrate” command has an additional “-q” flag. When provided the local storage of the VM is mirrored to the destination during the migration process.
2. Internally, the modification consists on adding a new libxl__stream_read_state struct to the libxl__domain_create_state structure and libxl__stream_read_state structure to the libxl__domain_save_state struct.
3. Migration flow can now be divided into three phases:
   a. Phase One: Copies the necessary state to start a QEMU process on the destination. It is started with the “-incoming defer” option.
   b. Phase Two: Disk is mirrored using the QEMU embedded NBD server.
   c. Phase Three: Once the disk is completely mirrored, virtual RAM of the domain is live migrated to the destination. This phase most closely resembles to the current migration flow.
4. If the “-q” option is not provided the migration is equivalent to the current migration flow.

The new migration flow has follows the following major sequence of steps:
1. 1st stream copies the QEMU devices RAM from source to destination.
2. QEMU process is started on the destination with the option “-incoming defer”. (This creates the QEMU process but it doesn’t start running the main loop until “migrate incoming” command is executed)
3. “drive mirror” QMP command is executed so that the disk is mirrored to the destination node.
4. An event listener waits for the QMP BLOCK_JOB_READY event sent by QEMU which signals that the "disk mirror job" is complete.
5. 2nd Stream copies the virtual RAM from source to destination including QEMU state. At this point, the VM is suspended on source.
6. “migrate incoming” QMP command is executed on destination.
7. VM is restored in destination.

This is sample configuration file that I have used to test my branch:

name="tinycore"
disk=['/home/balvisio/tinycore.img,raw,xvda,w']
memory=128
builder='hvm'
vcpus=1
vfb = ['type=vnc']
vif= ['bridge=xenbr0']
boot='b'
acpi=1
device_model_version='qemu-xen'
serial='pty'
vnc=1
xen_platform_pci=0

Notes

1. Note that the configuration file uses "xen_platform_pci=0”. This is necessary so that the block device is seen by QEMU. Further modification should be made for the case "xen_platform_pci=1” if we still want to use NBD mirroring capability provided by QEMU.
2. The current branch has still many hardcoded values. Many of the can be easily removed:
	a. Port used for disk mirroring (11000)
    b. Name of the block devices. (ide0-hd0) Currently the branch only supports VM with on disk drive.
    c. Live migration memory transfer: The pages transferred by libxc is hardcoded. Only a VM with 128 MB of memory is supported.

Here is a link to the branch in Github: 
https://github.com/balvisio/xen/tree/feature/local_storage_migration

Any feedback/suggestion is appreciated.

Cheers,

Bruno

Signed-off-by: Bruno Alvisio <bruno.alvisio@gmail.com>

---
 tools/libxc/include/xenguest.h       |   6 +-
 tools/libxc/xc_nomigrate.c           |   6 +-
 tools/libxc/xc_sr_common.h           |   3 + 
 tools/libxc/xc_sr_restore.c          |  14 +-
 tools/libxc/xc_sr_save.c             | 117 +++++++++++++++-
 tools/libxc/xc_sr_save_x86_hvm.c     |   7 +-
 tools/libxc/xc_sr_stream_format.h    |   4 + 
 tools/libxl/libxl.h                  |  11 +-
 tools/libxl/libxl_create.c           | 189 +++++++++++++++++++++++--
 tools/libxl/libxl_dm.c               |  20 ++- 
 tools/libxl/libxl_dom_save.c         |  66 ++++++++-
 tools/libxl/libxl_domain.c           |   4 +-
 tools/libxl/libxl_internal.h         |  62 ++++++++-
 tools/libxl/libxl_qmp.c              | 261 +++++++++++++++++++++++++++++++++++
 tools/libxl/libxl_save_callout.c     |  30 ++--
 tools/libxl/libxl_save_helper.c      |  13 +-
 tools/libxl/libxl_stream_read.c      |  17 ++- 
 tools/libxl/libxl_stream_write.c     |  28 +++-
 tools/libxl/libxl_types.idl          |   6 + 
 tools/ocaml/libs/xl/xenlight_stubs.c |   4 +-
 tools/xl/xl.h                        |   1 + 
 tools/xl/xl_cmdtable.c               |   3 +-
 tools/xl/xl_migrate.c                |  36 +++--
 tools/xl/xl_saverestore.c            |   2 +-
 tools/xl/xl_vmcontrol.c              |   5 +-
 25 files changed, 845 insertions(+), 70 deletions(-)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel