From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:37259) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHjPQ-0008AG-Dw for qemu-devel@nongnu.org; Mon, 18 Mar 2013 19:24:08 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UHjPL-0004yE-Lm for qemu-devel@nongnu.org; Mon, 18 Mar 2013 19:24:04 -0400 Received: from e8.ny.us.ibm.com ([32.97.182.138]:37625) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHjPL-0004xi-HG for qemu-devel@nongnu.org; Mon, 18 Mar 2013 19:23:59 -0400 Received: from /spool/local by e8.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 18 Mar 2013 19:23:56 -0400 Received: from d01relay05.pok.ibm.com (d01relay05.pok.ibm.com [9.56.227.237]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id 7B9F638C801A for ; Mon, 18 Mar 2013 19:23:54 -0400 (EDT) Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by d01relay05.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r2INNsKo303868 for ; Mon, 18 Mar 2013 19:23:54 -0400 Received: from d01av04.pok.ibm.com (loopback [127.0.0.1]) by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r2INNr5P024932 for ; Mon, 18 Mar 2013 19:23:53 -0400 Message-ID: <5147A209.80202@linux.vnet.ibm.com> Date: Mon, 18 Mar 2013 19:23:53 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <1363576743-6146-1-git-send-email-mrhines@linux.vnet.ibm.com> <1363576743-6146-4-git-send-email-mrhines@linux.vnet.ibm.com> <20130318104013.GE5267@redhat.com> <5147780C.1080800@linux.vnet.ibm.com> <20130318212646.GB20406@redhat.com> In-Reply-To: <20130318212646.GB20406@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v4: 03/10] more verbose documentation of the RDMA transport List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: aliguori@us.ibm.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com On 03/18/2013 05:26 PM, Michael S. Tsirkin wrote: > > Probably but I haven't mentioned ballooning at all. > > memory overcommit != ballooning Sure, then setting ballooning aside for the moment, then let's just consider regular (unused) virtual memory. In this case, what's wrong with the destination mapping and pinning all the memory if it is not being ballooned? If the guest touches all the memory during normal operation before migration begins (which would be the common case), then overcommit is irrelevant, no? > This is already handled by the RDMA connection manager (librdmacm). > > The library already has functions like listen() and accept() the same > way that TCP does. > > Once these functions return success, we have a gaurantee that both > sides of the connection have already posted the appropriate work > requests sufficient for driving the migration. > Not if you don't post anything. librdmacm does not post requests. So > everyone posts 1 buffer on RQ during connection setup? > OK though this is not what the document said, I was under the impression > this is done after connection setup. Sorry, I wasn't being clear. Here's the existing sequence that I've already coded and validated: 1. Receiver and Sender are started (command line): (The receiver has to be running before QMP migrate can connect, of course or this all falls apart.) 2. Both sides post RQ work requests (or multiple ones) 3. Receiver does listen() 4. Sender does connect() At this point both sides have already posted work requests as stated before. 5. Receiver accept() => issue first SEND message At this point the sequence of events I describe in the documentation for put_buffer() / get_buffer() all kick in and everything is normal. I'll be sure to post an extra few work requests as suggested. > > So # of buffers goes 0 -> 1 -> 0 -> 1. > What I am saying is you should have an extra buffer > so it goes 1 -> 2 -> 1 -> 2 > otherwise you keep hitting slow path in RQ processing: > each time you consume the last buffer, IIRC receiver sends > and ACK to sender saying "hey this is the last buffer, slow down". > You don't want that. No problem - I'll take care of it.......