From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:46922) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQJ4M-0007B6-Si for qemu-devel@nongnu.org; Thu, 11 Apr 2013 11:05:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UQJ4E-0003Q2-7H for qemu-devel@nongnu.org; Thu, 11 Apr 2013 11:05:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]:12182) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQJ4D-0003Po-T1 for qemu-devel@nongnu.org; Thu, 11 Apr 2013 11:05:38 -0400 Date: Thu, 11 Apr 2013 17:56:32 +0300 From: "Michael S. Tsirkin" Message-ID: <20130411145632.GA2280@redhat.com> References: <20130410133448.GA18128@redhat.com> <51658554.2000909@linux.vnet.ibm.com> <20130410174107.GB32247@redhat.com> <5165C60E.20006@linux.vnet.ibm.com> <20130411071927.GA17063@redhat.com> <5166B6B1.2030003@linux.vnet.ibm.com> <20130411134820.GA24942@redhat.com> <5166C19A.1040402@linux.vnet.ibm.com> <20130411143718.GC24942@redhat.com> <5166CDAD.8060807@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5166CDAD.8060807@redhat.com> Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v5: 03/12] comprehensive protocol documentation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: aliguori@us.ibm.com, qemu-devel@nongnu.org, "Michael R. Hines" , owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com On Thu, Apr 11, 2013 at 04:50:21PM +0200, Paolo Bonzini wrote: > Il 11/04/2013 16:37, Michael S. Tsirkin ha scritto: > > > > pg1 -> pin -> req -> res -> rdma -> done > > pg2 -> pin -> req -> res -> rdma -> done > > pg3 -> pin -> req -> res -> rdma -> done > > pg4 -> pin -> req -> res -> rdma -> done > > pg4 -> pin -> req -> res -> rdma -> done > > > > It's like a assembly line see? So while software does the registration > > roundtrip dance, hardware is processing rdma requests for previous > > chunks. > > Does this only affects the implementation, or also the wire protocol? It affects the wire protocol. > Does the destination have to be aware that the source is doing pipelining? > > Paolo Yes. At the moment the protocol assumption is there's only one outstanding command on the control queue. So destination has to prequeue multiple buffers on hardware receive queue, and keep the source updated about the number of available buffers. Preferably it should do this using existing responses, maybe a separate ready command is enough - this needs some thought, since a separate command consumes buffers itself. > > > > .... > > > > When do you have to stall? when you run out of rx buffer credits so you > > can not start a new req. Your protocol has 2 outstanding buffers, > > so you can only have one req in the air. Do more and > > you will not need to stall - possibly at all. > > > > One other minor point is that your protocol requires extra explicit > > ready commands. You can pass the number of rx buffers as extra payload > > in the traffic you are sending anyway, and reduce that overhead.