From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:51826) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UODXU-0003Vp-8R for qemu-devel@nongnu.org; Fri, 05 Apr 2013 16:47:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UODXL-00031e-LF for qemu-devel@nongnu.org; Fri, 05 Apr 2013 16:47:12 -0400 Received: from e9.ny.us.ibm.com ([32.97.182.139]:51663) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UODXL-00031a-Gt for qemu-devel@nongnu.org; Fri, 05 Apr 2013 16:47:03 -0400 Received: from /spool/local by e9.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 5 Apr 2013 16:47:03 -0400 Received: from d01relay04.pok.ibm.com (d01relay04.pok.ibm.com [9.56.227.236]) by d01dlp03.pok.ibm.com (Postfix) with ESMTP id 32FD4C9001E for ; Fri, 5 Apr 2013 16:46:59 -0400 (EDT) Received: from d03av06.boulder.ibm.com (d03av06.boulder.ibm.com [9.17.195.245]) by d01relay04.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r35KkwWf257264 for ; Fri, 5 Apr 2013 16:46:58 -0400 Received: from d03av06.boulder.ibm.com (loopback [127.0.0.1]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r35Knh3l031437 for ; Fri, 5 Apr 2013 14:49:43 -0600 Received: from [172.16.1.222] (klinux.watson.ibm.com [9.2.208.21]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r35KngXD031402 for ; Fri, 5 Apr 2013 14:49:43 -0600 Message-ID: <515F3841.70304@linux.vnet.ibm.com> Date: Fri, 05 Apr 2013 16:46:57 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <20130318212646.GB20406@redhat.com> <5147A209.80202@linux.vnet.ibm.com> <20130319081939.GC11259@redhat.com> <51487F68.2060305@linux.vnet.ibm.com> <20130319151606.GA13649@redhat.com> <51488521.4010909@linux.vnet.ibm.com> <20130319153658.GA14317@redhat.com> <51489BC3.3030504@linux.vnet.ibm.com> <51489D05.2000400@redhat.com> <5148A52E.6020208@linux.vnet.ibm.com> <20130321061159.GA28328@redhat.com> In-Reply-To: <20130321061159.GA28328@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v4: 03/10] more verbose documentation of the RDMA transport List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org FYI, I used the following redhat cgroups instructions, to test if overcommit + RDMA was working: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html - Michael On 03/21/2013 02:11 AM, Michael S. Tsirkin wrote: > On Tue, Mar 19, 2013 at 01:49:34PM -0400, Michael R. Hines wrote: >> I also did a test using RDMA + cgroup, and the kernel killed my QEMU :) >> >> So, infiniband is not smart enough to know how to avoid pinning a >> zero page, I guess. >> >> - Michael >> >> On 03/19/2013 01:14 PM, Paolo Bonzini wrote: >>> Il 19/03/2013 18:09, Michael R. Hines ha scritto: >>>> Allowing QEMU to swap due to a cgroup limit during migration is a viable >>>> overcommit option? >>>> >>>> I'm trying to keep an open mind, but that would kill the migration >>>> time..... >>> Would it swap? Doesn't the kernel back all zero pages with a single >>> copy-on-write page? If that still accounts towards cgroup limits, it >>> would be a bug. >>> >>> Old kernels do not have a shared zero hugepage, and that includes some >>> distro kernels. Perhaps that's the problem. >>> >>> Paolo >>> > I really shouldn't break COW if you don't request LOCAL_WRITE. > I think it's a kernel bug, and apparently has been there in the code since the > first version: get_user_pages parameters swapped. > > I'll send a patch. If it's applied, you should also > change your code from > > + IBV_ACCESS_LOCAL_WRITE | > + IBV_ACCESS_REMOTE_WRITE | > + IBV_ACCESS_REMOTE_READ); > > to > > + IBV_ACCESS_REMOTE_READ); > > on send side. > Then, each time we detect a page has changed we must make sure to > unregister and re-register it. Or if you want to be very > smart, check that the PFN didn't change and reregister > if it did. > > This will make overcommit work. >