[Qemu-devel] [PATCH v8 00/13] rdma: migration support

[mrhines@linux.vnet.ibm.com]

From: "Michael R. Hines" <mrhines@us.ibm.com>

Please pull.

Changes since v7:
    This fixes the problems experienced by others when 
    the x-rdma-pin-all feature appeared to freeze the VM. 
    
    By moving this operation out of the connection setup
    time and instead moving it to ram_save_setup() code, we no longer
    execute pinning inside the BQL and thus the pinning is parallelized
    with the VM execution and also properly accounted for inside the
    QMP migrate total time statistics.

Wiki: http://wiki.qemu.org/Features/RDMALiveMigration
Github: git@github.com:hinesmr/qemu.git

Here is a brief summary of total migration time and downtime using RDMA:

Using a 40gbps infiniband link performing a worst-case stress test,
using an 8GB RAM virtual machine:
Using the following command:

$ apt-get install stress
$ stress --vm-bytes 7500M --vm 1 --vm-keep

RESULTS:

1. Migration throughput: 26 gigabits/second.
2. Downtime (stop time) varies between 15 and 100 milliseconds.

EFFECTS of memory registration on bulk phase round:

For example, in the same 8GB RAM example with all 8GB of memory in 
active use and the VM itself is completely idle using the same 40 gbps 
infiniband link:

1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps

These numbers would of course scale up to whatever size virtual machine
you have to migrate using RDMA.

Enabling this feature does *not* have any measurable affect on 
migration *downtime*. This is because, without this feature, all of the 
memory will have already been registered already in advance during
the bulk round and does not need to be re-registered during the successive
iteration rounds.

The following changes since commit f3aa844bbb2922a5b8393d17620eca7d7e921ab3:

  build: include config-{, all-}devices.mak after defining CONFIG_SOFTMMU and CONFIG_USER_ONLY (2013-04-24 12:18:41 -0500)

are available in the git repository at:

  git@github.com:hinesmr/qemu.git rdma_patch_v6

for you to fetch changes up to 75e6fac1f642885b93cefe6e1874d648e9850f8f:

  rdma: send pc.ram (2013-04-24 14:55:01 -0400)

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

----------------------------------------------------------------
Michael R. Hines (13):
  rdma: add documentation
  rdma: introduce qemu_update_position()
  rdma: export yield_until_fd_readable()
  rdma: export throughput w/ MigrationStats QMP
  rdma: introduce qemu_file_mode_is_not_valid()
  rdma: export qemu_fflush()
  rdma: introduce ram_handle_compressed()
  rdma: introduce qemu_ram_foreach_block()
  rdma: new QEMUFileOps hooks
  rdma: introduce capability x-rdma-pin-all
  rdma: core logic
  rdma: send pc.ram
  rdma: fix mlock() freezes and accounting

 Makefile.objs                 |    1 +
 arch_init.c                   |   69 +-
 configure                     |   29 +
 docs/rdma.txt                 |  415 ++++++
 exec.c                        |    9 +
 hmp.c                         |    2 +
 include/block/coroutine.h     |    6 +
 include/exec/cpu-common.h     |    5 +
 include/migration/migration.h |   31 +
 include/migration/qemu-file.h |   32 +
 migration-rdma.c              | 2812 +++++++++++++++++++++++++++++++++++++++++
 migration.c                   |   23 +
 qapi-schema.json              |   12 +-
 qemu-coroutine-io.c           |   23 +
 savevm.c                      |  114 +-
 15 files changed, 3536 insertions(+), 47 deletions(-)
 create mode 100644 docs/rdma.txt
 create mode 100644 migration-rdma.c

-- 
1.7.10.4