From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:45522) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQJ0m-0004FE-R7 for qemu-devel@nongnu.org; Thu, 11 Apr 2013 11:02:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UQJ0h-0001sG-57 for qemu-devel@nongnu.org; Thu, 11 Apr 2013 11:02:04 -0400 Received: from e9.ny.us.ibm.com ([32.97.182.139]:40199) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQJ0h-0001ru-1X for qemu-devel@nongnu.org; Thu, 11 Apr 2013 11:01:59 -0400 Received: from /spool/local by e9.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 11 Apr 2013 11:01:58 -0400 Received: from d01relay02.pok.ibm.com (d01relay02.pok.ibm.com [9.56.227.234]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id DC95BC9005A for ; Thu, 11 Apr 2013 11:01:54 -0400 (EDT) Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d01relay02.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r3BF1rIN301938 for ; Thu, 11 Apr 2013 11:01:54 -0400 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r3BF1pxG003974 for ; Thu, 11 Apr 2013 09:01:51 -0600 Message-ID: <5166D05E.4000705@linux.vnet.ibm.com> Date: Thu, 11 Apr 2013 11:01:50 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <20130410052714.GB12777@redhat.com> <5165636C.1090908@linux.vnet.ibm.com> <20130410133448.GA18128@redhat.com> <51658554.2000909@linux.vnet.ibm.com> <20130410174107.GB32247@redhat.com> <5165C60E.20006@linux.vnet.ibm.com> <20130411071927.GA17063@redhat.com> <5166B6B1.2030003@linux.vnet.ibm.com> <20130411134820.GA24942@redhat.com> <5166C19A.1040402@linux.vnet.ibm.com> <20130411143718.GC24942@redhat.com> <5166CDAD.8060807@redhat.com> In-Reply-To: <5166CDAD.8060807@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v5: 03/12] comprehensive protocol documentation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: aliguori@us.ibm.com, "Michael S. Tsirkin" , qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com You cannot write data in the pipeline because you do not have the permissions to do so yet until the registrations in the pipeline have completed and been received by the primary VM. On 04/11/2013 10:50 AM, Paolo Bonzini wrote: > Il 11/04/2013 16:37, Michael S. Tsirkin ha scritto: >> pg1 -> pin -> req -> res -> rdma -> done >> pg2 -> pin -> req -> res -> rdma -> done >> pg3 -> pin -> req -> res -> rdma -> done >> pg4 -> pin -> req -> res -> rdma -> done >> pg4 -> pin -> req -> res -> rdma -> done >> >> It's like a assembly line see? So while software does the registration >> roundtrip dance, hardware is processing rdma requests for previous >> chunks. > Does this only affects the implementation, or also the wire protocol? > Does the destination have to be aware that the source is doing pipelining? > > Paolo Yes, the destination has to be aware. The destination has to acknowledge all of the registrations in the pipeline *and* the primary-VM has to block until all the registrations in the pipeline have been received.