From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:42223) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQIqj-0001TB-Un for qemu-devel@nongnu.org; Thu, 11 Apr 2013 10:51:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UQIqi-0006RH-M8 for qemu-devel@nongnu.org; Thu, 11 Apr 2013 10:51:41 -0400 Received: from mx1.redhat.com ([209.132.183.28]:3073) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQIqi-0006RB-CM for qemu-devel@nongnu.org; Thu, 11 Apr 2013 10:51:40 -0400 Message-ID: <5166CDAD.8060807@redhat.com> Date: Thu, 11 Apr 2013 16:50:21 +0200 From: Paolo Bonzini MIME-Version: 1.0 References: <20130410052714.GB12777@redhat.com> <5165636C.1090908@linux.vnet.ibm.com> <20130410133448.GA18128@redhat.com> <51658554.2000909@linux.vnet.ibm.com> <20130410174107.GB32247@redhat.com> <5165C60E.20006@linux.vnet.ibm.com> <20130411071927.GA17063@redhat.com> <5166B6B1.2030003@linux.vnet.ibm.com> <20130411134820.GA24942@redhat.com> <5166C19A.1040402@linux.vnet.ibm.com> <20130411143718.GC24942@redhat.com> In-Reply-To: <20130411143718.GC24942@redhat.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v5: 03/12] comprehensive protocol documentation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: aliguori@us.ibm.com, qemu-devel@nongnu.org, "Michael R. Hines" , owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com Il 11/04/2013 16:37, Michael S. Tsirkin ha scritto: > > pg1 -> pin -> req -> res -> rdma -> done > pg2 -> pin -> req -> res -> rdma -> done > pg3 -> pin -> req -> res -> rdma -> done > pg4 -> pin -> req -> res -> rdma -> done > pg4 -> pin -> req -> res -> rdma -> done > > It's like a assembly line see? So while software does the registration > roundtrip dance, hardware is processing rdma requests for previous > chunks. Does this only affects the implementation, or also the wire protocol? Does the destination have to be aware that the source is doing pipelining? Paolo > > .... > > When do you have to stall? when you run out of rx buffer credits so you > can not start a new req. Your protocol has 2 outstanding buffers, > so you can only have one req in the air. Do more and > you will not need to stall - possibly at all. > > One other minor point is that your protocol requires extra explicit > ready commands. You can pass the number of rx buffers as extra payload > in the traffic you are sending anyway, and reduce that overhead.