From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:40940) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQLdG-0003fN-Lw for qemu-devel@nongnu.org; Thu, 11 Apr 2013 13:50:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UQLd9-0005Oy-O3 for qemu-devel@nongnu.org; Thu, 11 Apr 2013 13:49:58 -0400 Received: from e34.co.us.ibm.com ([32.97.110.152]:60885) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UQLd9-0005OY-Ha for qemu-devel@nongnu.org; Thu, 11 Apr 2013 13:49:51 -0400 Received: from /spool/local by e34.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 11 Apr 2013 11:49:50 -0600 Received: from d03relay04.boulder.ibm.com (d03relay04.boulder.ibm.com [9.17.195.106]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id C8A4519D8042 for ; Thu, 11 Apr 2013 11:49:40 -0600 (MDT) Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay04.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r3BHnffI357890 for ; Thu, 11 Apr 2013 11:49:41 -0600 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r3BHnbBe017536 for ; Thu, 11 Apr 2013 11:49:38 -0600 Message-ID: <5166F7AE.8070209@linux.vnet.ibm.com> Date: Thu, 11 Apr 2013 13:49:34 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <20130410133448.GA18128@redhat.com> <51658554.2000909@linux.vnet.ibm.com> <20130410174107.GB32247@redhat.com> <5165C60E.20006@linux.vnet.ibm.com> <20130411071927.GA17063@redhat.com> <5166B6B1.2030003@linux.vnet.ibm.com> <20130411134820.GA24942@redhat.com> <5166C19A.1040402@linux.vnet.ibm.com> <20130411143718.GC24942@redhat.com> <5166CDAD.8060807@redhat.com> <20130411145632.GA2280@redhat.com> In-Reply-To: <20130411145632.GA2280@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v5: 03/12] comprehensive protocol documentation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: aliguori@us.ibm.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, Paolo Bonzini On 04/11/2013 10:56 AM, Michael S. Tsirkin wrote: > On Thu, Apr 11, 2013 at 04:50:21PM +0200, Paolo Bonzini wrote: >> Il 11/04/2013 16:37, Michael S. Tsirkin ha scritto: >>> pg1 -> pin -> req -> res -> rdma -> done >>> pg2 -> pin -> req -> res -> rdma -> done >>> pg3 -> pin -> req -> res -> rdma -> done >>> pg4 -> pin -> req -> res -> rdma -> done >>> pg4 -> pin -> req -> res -> rdma -> done >>> >>> It's like a assembly line see? So while software does the registration >>> roundtrip dance, hardware is processing rdma requests for previous >>> chunks. >> Does this only affects the implementation, or also the wire protocol? > It affects the wire protocol. I *do* believe chunked registration was a *very* useful request by the community, and I want to thank you for convincing me to implement it. But, with all due respect, pipelining is a "solution looking for a problem". Improving the protocol does not help the behavior of any well-known workloads, because it is based on the idea the the memory footprint of a VM would *rapidly* shrink and contract up and down during the steady-state iteration rounds while the migration is taking place. This simply does not happen - workloads don't behave that way - they either grow really big or they grow really small and they settle that way for a reasonable amount of time before the load on the application changes at a future point in time. - Michael