From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Thu, 24 Nov 2016 23:58:17 -0800 From: Christoph Hellwig Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20161125075817.GA18428@infradead.org> References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: linux-nvdimm-bounces@lists.01.org Sender: "Linux-nvdimm" To: Logan Gunthorpe Cc: Haggai Eran , "Bridgman, John , linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Kuehling, Felix , Serguei Sagalovitch" , "Blinzer, Paul , linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Sander, Ben , Jason Gunthorpe , Suthikulpanit, Suravee" , "linux-pci@vger.kernel.org" , Deucher,, "Christian , Linux-media@vger.kernel.org" List-ID: On Thu, Nov 24, 2016 at 11:11:34AM -0700, Logan Gunthorpe wrote: > * Regular DAX in the FS doesn't work at this time because the FS can > move the file you think your transfer to out from under you. Though I > understand there's been some work with XFS to solve that issue. The file system will never move anything under locked down pages, locking down pages is used exactly to protect against that. So as long as we page structures available RDMA to/from device memory _from kernel space_ is trivial, although for file systems to work properly you really want a notification to the consumer if the file systems wants to remove the mapping. We have implemented that using FL_LAYOUTS locks for NFSD, but only XFS supports it so far. Without that a long term locked down region of memory (e.g. a kernel MR) would prevent various file operations that would simply hang. _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bombadil.infradead.org ([198.137.202.9]:59519 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750726AbcKYIVb (ORCPT ); Fri, 25 Nov 2016 03:21:31 -0500 Date: Thu, 24 Nov 2016 23:58:17 -0800 From: Christoph Hellwig To: Logan Gunthorpe Cc: Jason Gunthorpe , Serguei Sagalovitch , Dan Williams , "Deucher, Alexander" , "linux-nvdimm@lists.01.org" , "linux-rdma@vger.kernel.org" , "linux-pci@vger.kernel.org" , "Kuehling, Felix" , "Bridgman, John" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Koenig, Christian" , "Sander, Ben" , "Suthikulpanit, Suravee" , "Blinzer, Paul" , "Linux-media@vger.kernel.org" , Haggai Eran Subject: Re: Enabling peer to peer device transactions for PCIe devices Message-ID: <20161125075817.GA18428@infradead.org> References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> Sender: linux-pci-owner@vger.kernel.org List-ID: On Thu, Nov 24, 2016 at 11:11:34AM -0700, Logan Gunthorpe wrote: > * Regular DAX in the FS doesn't work at this time because the FS can > move the file you think your transfer to out from under you. Though I > understand there's been some work with XFS to solve that issue. The file system will never move anything under locked down pages, locking down pages is used exactly to protect against that. So as long as we page structures available RDMA to/from device memory _from kernel space_ is trivial, although for file systems to work properly you really want a notification to the consumer if the file systems wants to remove the mapping. We have implemented that using FL_LAYOUTS locks for NFSD, but only XFS supports it so far. Without that a long term locked down region of memory (e.g. a kernel MR) would prevent various file operations that would simply hang. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christoph Hellwig Subject: Re: Enabling peer to peer device transactions for PCIe devices Date: Thu, 24 Nov 2016 23:58:17 -0800 Message-ID: <20161125075817.GA18428@infradead.org> References: <20161123190515.GA12146@obsidianresearch.com> <7bc38037-b6ab-943f-59db-6280e16901ab@amd.com> <20161123193228.GC12146@obsidianresearch.com> <20161123203332.GA15062@obsidianresearch.com> <20161123215510.GA16311@obsidianresearch.com> <91d28749-bc64-622f-56a1-26c00e6b462a@deltatee.com> <20161124164249.GD20818@obsidianresearch.com> <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <9cc22068-ede8-c1bc-5d8b-cf6224a7ce05@deltatee.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Logan Gunthorpe Cc: Haggai Eran , "linux-rdma@vger.kernel.org" , "linux-nvdimm@lists.01.org" , "Kuehling, Felix" , Serguei Sagalovitch , "Blinzer, Paul" , "linux-kernel@vger.kernel.org" , "dri-devel@lists.freedesktop.org" , "Sander, Ben" , Jason Gunthorpe , "Suthikulpanit, Suravee" , "linux-pci@vger.kernel.org" , "Deucher, Alexander" , Dan Williams , "Koenig, Christian" , "Linux-media@vger.kernel.org" List-Id: linux-rdma@vger.kernel.org T24gVGh1LCBOb3YgMjQsIDIwMTYgYXQgMTE6MTE6MzRBTSAtMDcwMCwgTG9nYW4gR3VudGhvcnBl IHdyb3RlOgo+ICogUmVndWxhciBEQVggaW4gdGhlIEZTIGRvZXNuJ3Qgd29yayBhdCB0aGlzIHRp bWUgYmVjYXVzZSB0aGUgRlMgY2FuCj4gbW92ZSB0aGUgZmlsZSB5b3UgdGhpbmsgeW91ciB0cmFu c2ZlciB0byBvdXQgZnJvbSB1bmRlciB5b3UuIFRob3VnaCBJCj4gdW5kZXJzdGFuZCB0aGVyZSdz IGJlZW4gc29tZSB3b3JrIHdpdGggWEZTIHRvIHNvbHZlIHRoYXQgaXNzdWUuCgpUaGUgZmlsZSBz eXN0ZW0gd2lsbCBuZXZlciBtb3ZlIGFueXRoaW5nIHVuZGVyIGxvY2tlZCBkb3duIHBhZ2VzLAps b2NraW5nIGRvd24gcGFnZXMgaXMgdXNlZCBleGFjdGx5IHRvIHByb3RlY3QgYWdhaW5zdCB0aGF0 LiAgU28gYXMgbG9uZwphcyB3ZSBwYWdlIHN0cnVjdHVyZXMgYXZhaWxhYmxlIFJETUEgdG8vZnJv bSBkZXZpY2UgbWVtb3J5IF9mcm9tIGtlcm5lbApzcGFjZV8gaXMgdHJpdmlhbCwgYWx0aG91Z2gg Zm9yIGZpbGUgc3lzdGVtcyB0byB3b3JrIHByb3Blcmx5IHlvdQpyZWFsbHkgd2FudCBhIG5vdGlm aWNhdGlvbiB0byB0aGUgY29uc3VtZXIgaWYgdGhlIGZpbGUgc3lzdGVtcyB3YW50cwp0byByZW1v dmUgdGhlIG1hcHBpbmcuICBXZSBoYXZlIGltcGxlbWVudGVkIHRoYXQgdXNpbmcgRkxfTEFZT1VU UyBsb2Nrcwpmb3IgTkZTRCwgYnV0IG9ubHkgWEZTIHN1cHBvcnRzIGl0IHNvIGZhci4gIFdpdGhv dXQgdGhhdCBhIGxvbmcgdGVybQpsb2NrZWQgZG93biByZWdpb24gb2YgbWVtb3J5IChlLmcuIGEg a2VybmVsIE1SKSB3b3VsZCBwcmV2ZW50IHZhcmlvdXMKZmlsZSBvcGVyYXRpb25zIHRoYXQgd291 bGQgc2ltcGx5IGhhbmcuCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCmRyaS1kZXZlbCBtYWlsaW5nIGxpc3QKZHJpLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9w Lm9yZwpodHRwczovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2RyaS1k ZXZlbAo=