From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53421) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnzK1-0001uw-ER for qemu-devel@nongnu.org; Sat, 15 Jun 2013 18:51:50 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UnzJz-0003F4-Dj for qemu-devel@nongnu.org; Sat, 15 Jun 2013 18:51:49 -0400 Received: from g1t0026.austin.hp.com ([15.216.28.33]:39640) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnzJz-0003Dt-7X for qemu-devel@nongnu.org; Sat, 15 Jun 2013 18:51:47 -0400 Message-ID: <51BCF001.7080204@hp.com> Date: Sat, 15 Jun 2013 15:51:45 -0700 From: Chegu Vinod MIME-Version: 1.0 References: <1371242153-11262-1-git-send-email-mrhines@linux.vnet.ibm.com> In-Reply-To: <1371242153-11262-1-git-send-email-mrhines@linux.vnet.ibm.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v9 00/14] rdma: migration support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: mrhines@linux.vnet.ibm.com Cc: aliguori@us.ibm.com, quintela@redhat.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com, knoel@redhat.com On 6/14/2013 1:35 PM, mrhines@linux.vnet.ibm.com wrote: > From: "Michael R. Hines" > > Changes since v8: > For very large virtual machines, pinning can take a long time. > While this does not affect the migration's *actual* time itself, > it is still important for the user to know what's going on and > to know what component of the total time is actual taken up by > pinning. > > For example, using a 14GB virtual machine, pinning can take as > long as 5 seconds, for which the user would not otherwise know > what was happening. > > Reviewed-by: Eric Blake > Reviewed-by: Paolo Bonzini Reviewed-by: Chegu Vinod Tested-by: Chegu Vinod Thx Vinod > > Wiki: http://wiki.qemu.org/Features/RDMALiveMigration > Github: git@github.com:hinesmr/qemu.git > > Here is a brief summary of total migration time and downtime using RDMA: > > Using a 40gbps infiniband link performing a worst-case stress test, > using an 8GB RAM virtual machine: > Using the following command: > > $ apt-get install stress > $ stress --vm-bytes 7500M --vm 1 --vm-keep > > RESULTS: > > 1. Migration throughput: 26 gigabits/second. > 2. Downtime (stop time) varies between 15 and 100 milliseconds. > > EFFECTS of memory registration on bulk phase round: > > For example, in the same 8GB RAM example with all 8GB of memory in > active use and the VM itself is completely idle using the same 40 gbps > infiniband link: > > 1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps > 2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps > > These numbers would of course scale up to whatever size virtual machine > you have to migrate using RDMA. > > Enabling this feature does *not* have any measurable affect on > migration *downtime*. This is because, without this feature, all of the > memory will have already been registered already in advance during > the bulk round and does not need to be re-registered during the successive > iteration rounds. > > The following changes since commit f3aa844bbb2922a5b8393d17620eca7d7e921ab3: > > build: include config-{, all-}devices.mak after defining CONFIG_SOFTMMU and CONFIG_USER_ONLY (2013-04-24 12:18:41 -0500) > > are available in the git repository at: > > git@github.com:hinesmr/qemu.git rdma_patch_v9 > > for you to fetch changes up to 75e6fac1f642885b93cefe6e1874d648e9850f8f: > > rdma: send pc.ram (2013-04-24 14:55:01 -0400) > > ---------------------------------------------------------------- > Michael R. Hines (14): > rdma: add documentation > rdma: introduce qemu_update_position() > rdma: export yield_until_fd_readable() > rdma: export throughput w/ MigrationStats QMP > rdma: introduce qemu_file_mode_is_not_valid() > rdma: export qemu_fflush() > rdma: introduce ram_handle_compressed() > rdma: introduce qemu_ram_foreach_block() > rdma: new QEMUFileOps hooks > rdma: introduce capability x-rdma-pin-all > rdma: core logic > rdma: send pc.ram > rdma: fix mlock() freezes and accounting > rdma: add pin-all accounting timestamp to QMP statistics > > Makefile.objs | 1 + > arch_init.c | 69 +- > configure | 29 + > docs/rdma.txt | 415 ++++++ > exec.c | 9 + > hmp.c | 6 + > include/block/coroutine.h | 6 + > include/exec/cpu-common.h | 5 + > include/migration/migration.h | 32 + > include/migration/qemu-file.h | 32 + > migration-rdma.c | 2831 +++++++++++++++++++++++++++++++++++++++++ > migration.c | 36 +- > qapi-schema.json | 15 +- > qemu-coroutine-io.c | 23 + > savevm.c | 114 +- > 15 files changed, 3574 insertions(+), 49 deletions(-) > create mode 100644 docs/rdma.txt > create mode 100644 migration-rdma.c >