From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51174) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnNQK-0007uS-FR for qemu-devel@nongnu.org; Fri, 14 Jun 2013 02:23:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UnNQH-0000sO-IP for qemu-devel@nongnu.org; Fri, 14 Jun 2013 02:23:48 -0400 Received: from e35.co.us.ibm.com ([32.97.110.153]:49508) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnNQH-0000sH-BP for qemu-devel@nongnu.org; Fri, 14 Jun 2013 02:23:45 -0400 Received: from /spool/local by e35.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 14 Jun 2013 00:23:44 -0600 Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id B76313E4003F for ; Fri, 14 Jun 2013 00:23:23 -0600 (MDT) Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by d03relay04.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r5E6NfVE314990 for ; Fri, 14 Jun 2013 00:23:41 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r5E6NflG032528 for ; Fri, 14 Jun 2013 00:23:41 -0600 From: mrhines@linux.vnet.ibm.com Date: Fri, 14 Jun 2013 02:23:12 -0400 Message-Id: <1371191005-9349-1-git-send-email-mrhines@linux.vnet.ibm.com> Subject: [Qemu-devel] [PATCH v8 00/13] rdma: migration support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: aliguori@us.ibm.com, quintela@redhat.com, knoel@redhat.com, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com, chegu_vinod@hp.com From: "Michael R. Hines" Please pull. Changes since v7: This fixes the problems experienced by others when the x-rdma-pin-all feature appeared to freeze the VM. By moving this operation out of the connection setup time and instead moving it to ram_save_setup() code, we no longer execute pinning inside the BQL and thus the pinning is parallelized with the VM execution and also properly accounted for inside the QMP migrate total time statistics. Wiki: http://wiki.qemu.org/Features/RDMALiveMigration Github: git@github.com:hinesmr/qemu.git Here is a brief summary of total migration time and downtime using RDMA: Using a 40gbps infiniband link performing a worst-case stress test, using an 8GB RAM virtual machine: Using the following command: $ apt-get install stress $ stress --vm-bytes 7500M --vm 1 --vm-keep RESULTS: 1. Migration throughput: 26 gigabits/second. 2. Downtime (stop time) varies between 15 and 100 milliseconds. EFFECTS of memory registration on bulk phase round: For example, in the same 8GB RAM example with all 8GB of memory in active use and the VM itself is completely idle using the same 40 gbps infiniband link: 1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps 2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps These numbers would of course scale up to whatever size virtual machine you have to migrate using RDMA. Enabling this feature does *not* have any measurable affect on migration *downtime*. This is because, without this feature, all of the memory will have already been registered already in advance during the bulk round and does not need to be re-registered during the successive iteration rounds. The following changes since commit f3aa844bbb2922a5b8393d17620eca7d7e921ab3: build: include config-{, all-}devices.mak after defining CONFIG_SOFTMMU and CONFIG_USER_ONLY (2013-04-24 12:18:41 -0500) are available in the git repository at: git@github.com:hinesmr/qemu.git rdma_patch_v6 for you to fetch changes up to 75e6fac1f642885b93cefe6e1874d648e9850f8f: rdma: send pc.ram (2013-04-24 14:55:01 -0400) Reviewed-by: Eric Blake Reviewed-by: Paolo Bonzini ---------------------------------------------------------------- Michael R. Hines (13): rdma: add documentation rdma: introduce qemu_update_position() rdma: export yield_until_fd_readable() rdma: export throughput w/ MigrationStats QMP rdma: introduce qemu_file_mode_is_not_valid() rdma: export qemu_fflush() rdma: introduce ram_handle_compressed() rdma: introduce qemu_ram_foreach_block() rdma: new QEMUFileOps hooks rdma: introduce capability x-rdma-pin-all rdma: core logic rdma: send pc.ram rdma: fix mlock() freezes and accounting Makefile.objs | 1 + arch_init.c | 69 +- configure | 29 + docs/rdma.txt | 415 ++++++ exec.c | 9 + hmp.c | 2 + include/block/coroutine.h | 6 + include/exec/cpu-common.h | 5 + include/migration/migration.h | 31 + include/migration/qemu-file.h | 32 + migration-rdma.c | 2812 +++++++++++++++++++++++++++++++++++++++++ migration.c | 23 + qapi-schema.json | 12 +- qemu-coroutine-io.c | 23 + savevm.c | 114 +- 15 files changed, 3536 insertions(+), 47 deletions(-) create mode 100644 docs/rdma.txt create mode 100644 migration-rdma.c -- 1.7.10.4