From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:36816) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uo4FU-0007r1-9O for qemu-devel@nongnu.org; Sun, 16 Jun 2013 00:07:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Uo4FQ-000456-B1 for qemu-devel@nongnu.org; Sun, 16 Jun 2013 00:07:28 -0400 Received: from e34.co.us.ibm.com ([32.97.110.152]:53309) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uo4FQ-000451-4t for qemu-devel@nongnu.org; Sun, 16 Jun 2013 00:07:24 -0400 Received: from /spool/local by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 15 Jun 2013 22:07:23 -0600 Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id 8FD2E3E40039 for ; Sat, 15 Jun 2013 22:07:03 -0600 (MDT) Received: from d03av02.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.195.168]) by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r5G47LDn140730 for ; Sat, 15 Jun 2013 22:07:21 -0600 Received: from d03av02.boulder.ibm.com (loopback [127.0.0.1]) by d03av02.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r5G47LVP021757 for ; Sat, 15 Jun 2013 22:07:21 -0600 Message-ID: <51BD39F8.5090807@linux.vnet.ibm.com> Date: Sun, 16 Jun 2013 00:07:20 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <1371242153-11262-1-git-send-email-mrhines@linux.vnet.ibm.com> <51BCF001.7080204@hp.com> In-Reply-To: <51BCF001.7080204@hp.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v9 00/14] rdma: migration support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Chegu Vinod Cc: aliguori@us.ibm.com, quintela@redhat.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com, knoel@redhat.com On 06/15/2013 06:51 PM, Chegu Vinod wrote: > On 6/14/2013 1:35 PM, mrhines@linux.vnet.ibm.com wrote: >> From: "Michael R. Hines" >> >> Changes since v8: >> For very large virtual machines, pinning can take a long time. >> While this does not affect the migration's *actual* time itself, >> it is still important for the user to know what's going on and >> to know what component of the total time is actual taken up by >> pinning. >> >> For example, using a 14GB virtual machine, pinning can take as >> long as 5 seconds, for which the user would not otherwise know >> what was happening. >> >> Reviewed-by: Eric Blake >> Reviewed-by: Paolo Bonzini > > Reviewed-by: Chegu Vinod > Tested-by: Chegu Vinod > Thanks... > > Thx > Vinod >> >> Wiki: http://wiki.qemu.org/Features/RDMALiveMigration >> Github: git@github.com:hinesmr/qemu.git >> >> Here is a brief summary of total migration time and downtime using RDMA: >> >> Using a 40gbps infiniband link performing a worst-case stress test, >> using an 8GB RAM virtual machine: >> Using the following command: >> >> $ apt-get install stress >> $ stress --vm-bytes 7500M --vm 1 --vm-keep >> >> RESULTS: >> >> 1. Migration throughput: 26 gigabits/second. >> 2. Downtime (stop time) varies between 15 and 100 milliseconds. >> >> EFFECTS of memory registration on bulk phase round: >> >> For example, in the same 8GB RAM example with all 8GB of memory in >> active use and the VM itself is completely idle using the same 40 gbps >> infiniband link: >> >> 1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ >> 9.5 Gbps >> 2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps >> >> These numbers would of course scale up to whatever size virtual machine >> you have to migrate using RDMA. >> >> Enabling this feature does *not* have any measurable affect on >> migration *downtime*. This is because, without this feature, all of the >> memory will have already been registered already in advance during >> the bulk round and does not need to be re-registered during the >> successive >> iteration rounds. >> >> The following changes since commit >> f3aa844bbb2922a5b8393d17620eca7d7e921ab3: >> >> build: include config-{, all-}devices.mak after defining >> CONFIG_SOFTMMU and CONFIG_USER_ONLY (2013-04-24 12:18:41 -0500) >> >> are available in the git repository at: >> >> git@github.com:hinesmr/qemu.git rdma_patch_v9 >> >> for you to fetch changes up to 75e6fac1f642885b93cefe6e1874d648e9850f8f: >> >> rdma: send pc.ram (2013-04-24 14:55:01 -0400) >> >> ---------------------------------------------------------------- >> Michael R. Hines (14): >> rdma: add documentation >> rdma: introduce qemu_update_position() >> rdma: export yield_until_fd_readable() >> rdma: export throughput w/ MigrationStats QMP >> rdma: introduce qemu_file_mode_is_not_valid() >> rdma: export qemu_fflush() >> rdma: introduce ram_handle_compressed() >> rdma: introduce qemu_ram_foreach_block() >> rdma: new QEMUFileOps hooks >> rdma: introduce capability x-rdma-pin-all >> rdma: core logic >> rdma: send pc.ram >> rdma: fix mlock() freezes and accounting >> rdma: add pin-all accounting timestamp to QMP statistics >> >> Makefile.objs | 1 + >> arch_init.c | 69 +- >> configure | 29 + >> docs/rdma.txt | 415 ++++++ >> exec.c | 9 + >> hmp.c | 6 + >> include/block/coroutine.h | 6 + >> include/exec/cpu-common.h | 5 + >> include/migration/migration.h | 32 + >> include/migration/qemu-file.h | 32 + >> migration-rdma.c | 2831 >> +++++++++++++++++++++++++++++++++++++++++ >> migration.c | 36 +- >> qapi-schema.json | 15 +- >> qemu-coroutine-io.c | 23 + >> savevm.c | 114 +- >> 15 files changed, 3574 insertions(+), 49 deletions(-) >> create mode 100644 docs/rdma.txt >> create mode 100644 migration-rdma.c >> > >