From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:38823) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHwUA-0004T5-DL for qemu-devel@nongnu.org; Tue, 19 Mar 2013 09:21:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UHwU2-0006rd-Ff for qemu-devel@nongnu.org; Tue, 19 Mar 2013 09:21:50 -0400 Received: from e39.co.us.ibm.com ([32.97.110.160]:52738) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHwU2-0006qj-9Q for qemu-devel@nongnu.org; Tue, 19 Mar 2013 09:21:42 -0400 Received: from /spool/local by e39.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 19 Mar 2013 07:21:38 -0600 Received: from d01relay02.pok.ibm.com (d01relay02.pok.ibm.com [9.56.227.234]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id 59A5A38C8029 for ; Tue, 19 Mar 2013 09:21:27 -0400 (EDT) Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay02.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r2JDLQDn252948 for ; Tue, 19 Mar 2013 09:21:27 -0400 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r2JDLPL6002022 for ; Tue, 19 Mar 2013 10:21:25 -0300 Message-ID: <5148664E.3090207@linux.vnet.ibm.com> Date: Tue, 19 Mar 2013 09:21:18 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <1363576743-6146-1-git-send-email-mrhines@linux.vnet.ibm.com> <1363576743-6146-4-git-send-email-mrhines@linux.vnet.ibm.com> <20130318104013.GE5267@redhat.com> <5147780C.1080800@linux.vnet.ibm.com> <20130318212646.GB20406@redhat.com> <5147A209.80202@linux.vnet.ibm.com> <20130319081939.GC11259@redhat.com> In-Reply-To: <20130319081939.GC11259@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v4: 03/10] more verbose documentation of the RDMA transport List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: aliguori@us.ibm.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com On 03/19/2013 04:19 AM, Michael S. Tsirkin wrote: > We have ways (e.g. cgroups) to limit what a VM can do. If it tries to > use more RAM than we let it, it will swap, still making progress, just > slower. OTOH it looks like pinning more memory than allowed by the > cgroups limit will just get stuck forever (probably a bug, should fail > instead? but does not help your protocol which needs it all pinned at > all times). There are also per-task resource limits. If you exceed > this registration will fail, so not good either. I just don't see why > do registration by chunks on source but not on destination. Would this a hard requirement for an initial version? I do understand how and why this makes things more flexible during the long run, but it does have the potential to slow down the RDMA protocol significantly. The way its implemented now, the sender can dump bytes onto the wire at full speed (up to 30gbps last time I measured it), but if we insert a round-trip message + registration on the destination side before we're allowed to push more bytes out, we'll have to introduce more complex flow control only for the benefit of making the destination side have the flexibility that you described.