From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([208.118.235.92]:48389)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mrhines@linux.vnet.ibm.com>) id 1UPcmS-0004wP-M9
	for qemu-devel@nongnu.org; Tue, 09 Apr 2013 13:56:31 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mrhines@linux.vnet.ibm.com>) id 1UPcmP-0002u0-OM
	for qemu-devel@nongnu.org; Tue, 09 Apr 2013 13:56:28 -0400
Received: from e7.ny.us.ibm.com ([32.97.182.137]:55253)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mrhines@linux.vnet.ibm.com>) id 1UPcmP-0002tR-L5
	for qemu-devel@nongnu.org; Tue, 09 Apr 2013 13:56:25 -0400
Received: from /spool/local
	by e7.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only!
	Violators will be prosecuted
	for <qemu-devel@nongnu.org> from <mrhines@linux.vnet.ibm.com>;
	Tue, 9 Apr 2013 13:56:24 -0400
Received: from d01relay03.pok.ibm.com (d01relay03.pok.ibm.com [9.56.227.235])
	by d01dlp02.pok.ibm.com (Postfix) with ESMTP id 38A866E8057
	for <qemu-devel@nongnu.org>; Tue,  9 Apr 2013 13:56:13 -0400 (EDT)
Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216])
	by d01relay03.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id
	r39HuEeo284290
	for <qemu-devel@nongnu.org>; Tue, 9 Apr 2013 13:56:15 -0400
Received: from d01av02.pok.ibm.com (loopback [127.0.0.1])
	by d01av02.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id
	r39Hu7mw005915
	for <qemu-devel@nongnu.org>; Tue, 9 Apr 2013 14:56:07 -0300
Message-ID: <51645630.3030608@linux.vnet.ibm.com>
Date: Tue, 09 Apr 2013 13:56:00 -0400
From: "Michael R. Hines" <mrhines@linux.vnet.ibm.com>
MIME-Version: 1.0
References: <20130324155153.GA8597@redhat.com>
	<515F3160.4020007@linux.vnet.ibm.com>
	<CAL1RGDU2jtrRrxi6srsH_ADiYGvJbCpCT3UvoLGTX3cvQ=s0kQ@mail.gmail.com>
	<515F3A0F.5030507@linux.vnet.ibm.com>
	<20130409163929.GA7661@redhat.com>
In-Reply-To: <20130409163929.GA7661@redhat.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [Qemu-devel] [PATCHv2] rdma: add a new IB_ACCESS_GIFT flag
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Roland Dreier <roland@kernel.org>, Christoph Lameter <cl@linux.com>, "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>, Yishai Hadas <yishaih@mellanox.com>, qemu-devel@nongnu.org, LKML <linux-kernel@vger.kernel.org>, Jason Gunthorpe <jgunthorpe@obsidianresearch.com>, Sean Hefty <sean.hefty@intel.com>, Hal Rosenstock <hal.rosenstock@gmail.com>

On 04/09/2013 12:39 PM, Michael S. Tsirkin wrote:
> On Fri, Apr 05, 2013 at 04:54:39PM -0400, Michael R. Hines wrote:
>> To be more specific, here's what I did:
>>
>> 1. apply kernel module patch - re-insert module
>> 1. QEMU does: ibv_reg_mr(........IBV_ACCESS_GIFT | IBV_ACCESS_REMOTE_READ)
>> 2. Start the RDMA migration
>> 3. Migration completes without any errors
>>
>> This test does *not* work with a cgroup swap limit, however. The
>> process gets killed. (Both with and without GIFT)
>>
>> - Michael
> Try to attach a debugger and see where it is when it gets killed?
>

It's killed by cgroups - not a CPU exception.

The same test works fine using TCP migration with cgroups - everything 
is fine there.

The memory that RDMA attempted to register hits some kind of cgroups policy
which results in a kernel message saying that the cgroup swap limit was hit
and then it goes ahead and kills the process altogether.

It's not a QEMU problem - it seems to be a kernel bug.