[Qemu-devel] [PATCH v9 00/14] rdma: migration support

[mrhines@linux.vnet.ibm.com]

From: "Michael R. Hines" <mrhines@us.ibm.com>

Changes since v8:
    For very large virtual machines, pinning can take a long time.
    While this does not affect the migration's *actual* time itself,
    it is still important for the user to know what's going on and
    to know what component of the total time is actual taken up by
    pinning.

    For example, using a 14GB virtual machine, pinning can take as
    long as 5 seconds, for which the user would not otherwise know
    what was happening.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Wiki: http://wiki.qemu.org/Features/RDMALiveMigration
Github: git@github.com:hinesmr/qemu.git

Here is a brief summary of total migration time and downtime using RDMA:

Using a 40gbps infiniband link performing a worst-case stress test,
using an 8GB RAM virtual machine:
Using the following command:

$ apt-get install stress
$ stress --vm-bytes 7500M --vm 1 --vm-keep

RESULTS:

1. Migration throughput: 26 gigabits/second.
2. Downtime (stop time) varies between 15 and 100 milliseconds.

EFFECTS of memory registration on bulk phase round:

For example, in the same 8GB RAM example with all 8GB of memory in 
active use and the VM itself is completely idle using the same 40 gbps 
infiniband link:

1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps

These numbers would of course scale up to whatever size virtual machine
you have to migrate using RDMA.

Enabling this feature does *not* have any measurable affect on 
migration *downtime*. This is because, without this feature, all of the 
memory will have already been registered already in advance during
the bulk round and does not need to be re-registered during the successive
iteration rounds.

The following changes since commit f3aa844bbb2922a5b8393d17620eca7d7e921ab3:

  build: include config-{, all-}devices.mak after defining CONFIG_SOFTMMU and CONFIG_USER_ONLY (2013-04-24 12:18:41 -0500)

are available in the git repository at:

  git@github.com:hinesmr/qemu.git rdma_patch_v9

for you to fetch changes up to 75e6fac1f642885b93cefe6e1874d648e9850f8f:

  rdma: send pc.ram (2013-04-24 14:55:01 -0400)

----------------------------------------------------------------
Michael R. Hines (14):
  rdma: add documentation
  rdma: introduce qemu_update_position()
  rdma: export yield_until_fd_readable()
  rdma: export throughput w/ MigrationStats QMP
  rdma: introduce qemu_file_mode_is_not_valid()
  rdma: export qemu_fflush()
  rdma: introduce ram_handle_compressed()
  rdma: introduce qemu_ram_foreach_block()
  rdma: new QEMUFileOps hooks
  rdma: introduce capability x-rdma-pin-all
  rdma: core logic
  rdma: send pc.ram
  rdma: fix mlock() freezes and accounting
  rdma: add pin-all accounting timestamp to QMP statistics

 Makefile.objs                 |    1 +
 arch_init.c                   |   69 +-
 configure                     |   29 +
 docs/rdma.txt                 |  415 ++++++
 exec.c                        |    9 +
 hmp.c                         |    6 +
 include/block/coroutine.h     |    6 +
 include/exec/cpu-common.h     |    5 +
 include/migration/migration.h |   32 +
 include/migration/qemu-file.h |   32 +
 migration-rdma.c              | 2831 +++++++++++++++++++++++++++++++++++++++++
 migration.c                   |   36 +-
 qapi-schema.json              |   15 +-
 qemu-coroutine-io.c           |   23 +
 savevm.c                      |  114 +-
 15 files changed, 3574 insertions(+), 49 deletions(-)
 create mode 100644 docs/rdma.txt
 create mode 100644 migration-rdma.c

-- 
1.7.10.4