From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:44886) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnBOs-0006KR-Kz for qemu-devel@nongnu.org; Thu, 13 Jun 2013 13:33:34 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UnBOp-0000wR-No for qemu-devel@nongnu.org; Thu, 13 Jun 2013 13:33:30 -0400 Received: from e32.co.us.ibm.com ([32.97.110.150]:39845) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UnBOp-0000iU-Hx for qemu-devel@nongnu.org; Thu, 13 Jun 2013 13:33:27 -0400 Received: from /spool/local by e32.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 13 Jun 2013 08:56:17 -0600 Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by d03dlp01.boulder.ibm.com (Postfix) with ESMTP id 27DB01FF0087 for ; Thu, 13 Jun 2013 08:50:40 -0600 (MDT) Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170]) by d03relay04.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r5DEtc5N289480 for ; Thu, 13 Jun 2013 08:55:39 -0600 Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1]) by d03av04.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r5DEtZvb002512 for ; Thu, 13 Jun 2013 08:55:35 -0600 Message-ID: <51B9DD5C.1030409@linux.vnet.ibm.com> Date: Thu, 13 Jun 2013 10:55:24 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <1370880226-2208-1-git-send-email-mrhines@linux.vnet.ibm.com> <4168C988EBDF2141B4E0B6475B6A73D10CE2AAC1@G6W2488.americas.hpqcorp.net> <51B60ABA.2070401@linux.vnet.ibm.com> <4168C988EBDF2141B4E0B6475B6A73D10CE2BAE7@G6W2488.americas.hpqcorp.net> <51B7B652.3070905@linux.vnet.ibm.com> <51B85EE5.1050702@hp.com> <51B868B3.9090607@linux.vnet.ibm.com> <51B9A614.2050101@hp.com> <51B9C2D6.30000@linux.vnet.ibm.com> <51B9D6A8.9070007@hp.com> In-Reply-To: <51B9D6A8.9070007@hp.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v7 00/12] rdma: migration support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Chegu Vinod Cc: Juan Jose Quintela Carreira , "qemu-devel@nongnu.org" , Bulent Abali , "mrhines@us.ibm.com" , Anthony Liguori , Paolo Bonzini On 06/13/2013 10:26 AM, Chegu Vinod wrote: >> >> 1. start QEMU with the lock option *first* >> 2. Then enable x-rdma-pin-all >> 3. Then perform the migration >> >> What happens here? Does pinning "in advance" help you? > > Yes it does help by avoiding the freeze time at the start of the > pin-all migration. > > I already mentioned about this in in my earlier responses as an option > to consider for larger guests > (https://lists.gnu.org/archive/html/qemu-devel/2013-05/msg00435.html). > > But pinning all of guest memory has a few drawbacks...as you may > already know. > > Just to be sure I just double checked it again with your v7 bits . > Started a 64GB/10VCPU guest (started qemu with the"-realtime > mlock=on" option) and as expected the guest startup took about 20 > seconds longer (i.e. time taken to mlock the 64GB of guest RAM) but > the pin-all migration started fine...i.e. didn't observe any freezes > at the start of the migration > > (CC-ing qemu-devel). OK, that's good to know. This means that we need to bringup the mlock() problem as a "larger" issue in the linux community instead of the QEMU community. In the meantime, how about I make update to the RDMA patch which does the following: 1. Solution #1: If user requests "x-rdma-pin-all", then If QEMU has enabled "-realtime mlock=on" Then, allow the capability Else Disallow the capability 2. Solution #2: Create NEW qemu monitor command which locks memory *in advance* before the migrate command occurs, to clearly indicate to the user that the cost of locking memory must be paid before the migration starts. Which solution do you prefer? Or do you have alternative idea? > https://lists.gnu.org/archive/html/qemu-devel/2013-04/msg04161.html > > Again this is a generic linux mlock/clearpage related issue and not > directly related to your changes. > Do you have any ideas on how linux can be improved to solve this? Is there any ongoing work that you know of on mlock() performance? Is there, perhaps, some way for linux to "parallelize" the mlock()/clearpage operation? - Michael