From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [PATCH 2/8] PCI: Add pci_find_common_upstream_dev() Date: Mon, 2 Apr 2018 15:16:50 -0400 Message-ID: <20180402191649.GB18231@redhat.com> References: <70adc2cc-f7aa-d4b9-7d7a-71f3ae99f16c@gmail.com> <98ce6cfd-bcf3-811e-a0f1-757b60da467a@deltatee.com> <8d050848-8970-b8c4-a657-429fefd31769@amd.com> <20180330015854.GA3572@redhat.com> <0234bc5e-495e-0f68-fb0a-debb17a35761@deltatee.com> <20180330194519.GC3198@redhat.com> <31266710-f6bb-99ee-c73d-6e58afe5c38c@deltatee.com> <20180402172027.GA18231@redhat.com> <6f796779-0ba3-d056-de33-341ee55d6b38@deltatee.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <6f796779-0ba3-d056-de33-341ee55d6b38@deltatee.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: Logan Gunthorpe Cc: linaro-mm-sig@lists.linaro.org, Will Davis , amd-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, Christoph Hellwig , dri-devel@lists.freedesktop.org, Bjorn Helgaas , Christian =?iso-8859-1?Q?K=F6nig?= , linux-media@vger.kernel.org List-Id: amd-gfx.lists.freedesktop.org T24gTW9uLCBBcHIgMDIsIDIwMTggYXQgMTE6Mzc6MDdBTSAtMDYwMCwgTG9nYW4gR3VudGhvcnBl IHdyb3RlOgo+IAo+IAo+IE9uIDAyLzA0LzE4IDExOjIwIEFNLCBKZXJvbWUgR2xpc3NlIHdyb3Rl Ogo+ID4gVGhlIHBvaW50IGkgaGF2ZSBiZWVuIHRyeWluZyB0byBnZXQgYWNjcm9zcyBpcyB0aGF0 IHlvdSBkbyBoYXZlIHRoaXMKPiA+IGluZm9ybWF0aW9uIHdpdGggZG1hX21hcF9yZXNvdXJjZSgp IHlvdSBrbm93IHRoZSBkZXZpY2UgdG8gd2hpY2ggeW91Cj4gPiBhcmUgdHJ5aW5nIHRvIG1hcCAo ZGV2IGFyZ3VtZW50IHRvIGRtYV9tYXBfcmVzb3VyY2UoKSkgYW5kIHlvdSBjYW4KPiA+IGVhc2ls eSBnZXQgdGhlIGRldmljZSB0byB3aGljaCB0aGUgbWVtb3J5IGJlbG9uZ3MgYmVjYXVzZSB5b3Ug aGF2ZSB0aGUKPiA+IENQVSBwaHlzaWNhbCBhZGRyZXNzIG9mIHRoZSBtZW1vcnkgaGVuY2UgeW91 IGNhbiBsb29rdXAgdGhlIHJlc291cmNlCj4gPiBhbmQgZ2V0IHRoZSBkZXZpY2UgZnJvbSB0aGF0 Lgo+IAo+IEhvdyBkbyB5b3UgZ28gZnJvbSBhIHBoeXNpY2FsIGFkZHJlc3MgdG8gYSBzdHJ1Y3Qg ZGV2aWNlIGdlbmVyYWxseSBhbmQKPiBpbiBhIHBlcmZvcm1hbnQgbWFubmVyPwoKVGhlcmUgaXNu J3QgZ29vZCBBUEkgYXQgdGhlIG1vbWVudCBBRkFJSywgY2xvc2VzdCB0aGluZyB3b3VsZCBlaXRo ZXIgYmUKbG9va3VwX3Jlc291cmNlKCkgb3IgcmVnaW9uX2ludGVyc2VjdHMoKSwgYnV0IGEgbW9y ZSBhcHByb3ByaWF0ZSBvbmUgY2FuCmVhc2lseSBiZSBhZGRlZCwgY29kZSB0byB3YWxrIGRvd24g dGhlIHRyZWUgaXMgcmVhZGlseSBhdmFpbGFibGUuIE1vcmUtCm92ZXIgdGhpcyBjYW4gYmUgb3B0 aW1pemUgbGlrZSB2bWEgbG9va3VwIGFyZSwgZXZlbiBtb3JlIGFzIHJlc291cmNlIGFyZQpzZWxk b21seSBhZGRlZCBzbyByZWFkIHNpZGUgKGZpbmRpbmcgYSByZXNvdXJjZSkgY2FuIGJlIGhlYXZp bHkgZmF2b3IKb3ZlciB3cml0ZSBzaWRlIChhZGRpbmd8cmVnaXN0ZXJpbmcgYSBuZXcgcmVzb3Vy Y2UpLgoKPiAKPiA+IElJUkMgQ0FQSSBtYWtlIFAyUCBtYW5kYXRvcnkgYnV0IG1heWJlIHRoaXMg aXMgd2l0aCBOVkxpbmsuIFdlIGNhbiBhc2sKPiA+IHRoZSBQb3dlclBDIGZvbGtzIHRvIGNvbmZp cm0uIE5vdGUgQ0FQSSBpcyBQb3dlcjggYW5kIG5ld2VyIEFGQUlDVC4KPiAKPiBQb3dlclBDIGZv bGtzIHJlY2VudGx5IHRvbGQgdXMgc3BlY2lmaWNhbGx5IHRoYXQgUG93ZXI5IGRvZXMgbm90IHN1 cHBvcnQKPiBQMlAgYmV0d2VlbiBQQ0kgcm9vdCBwb3J0cy4gSSd2ZSBzYWlkIHRoaXMgbWFueSB0 aW1lcy4gQ0FQSSBoYXMgbm90aGluZwo+IHRvIGRvIHdpdGggaXQuCgpJIG5lZWQgdG8gY2hlY2sg Q0FQSSwgaSBtdXN0IGhhdmUgY29uZnVzZSB0aGF0IHdpdGggTlZMaW5rIHdoaWNoIGlzIGFsc28K b24gc29tZSBwb3dlcnBjIGFyY2guCgo+IAo+ID4gTWFwcGluZyB0byB1c2Vyc3BhY2UgaGF2ZSBu b3RoaW5nIHRvIGRvIGhlcmUuIEkgYW0gdGFsa2luZyBhdCBoYXJkd2FyZQo+ID4gbGV2ZWwuIEhv dyB0aGluZyBhcmUgZXhwb3NlIHRvIHVzZXJzcGFjZSBpcyBhIGNvbXBsZXRlbHkgZGlmZmVyZW50 Cj4gPiBwcm9ibGVtcyB0aGF0IGRvIG5vdCBoYXZlIG9uZSBzb2x1dGlvbiBmaXQgYWxsLiBGb3Ig R1BVIHlvdSB3YW50IHRoaXMKPiA+IHRvIGJlIHVuZGVyIHRvdGFsIGNvbnRyb2wgb2YgR1BVIGRy aXZlcnMuIEZvciBzdG9yYWdlIGxpa2UgcGVyc2lzdGVudAo+ID4gbWVtb3J5LCB5b3UgbWlnaHQg d2FudCB0byBleHBvc2UgaXQgdXNlcnNwYWNlIG1vcmUgZGlyZWN0bHkgLi4uCj4gCj4gTXkgdW5k ZXJzdGFuZGluZyAoYW5kIEkgd29ya2VkIG9uIHRoaXMgYSB3aGlsZSBhZ28pIGlzIHRoYXQgQ0FQ SQo+IGhhcmR3YXJlIG1hbmFnZXMgbWVtb3J5IG1hcHMgdHlwaWNhbGx5IGZvciB1c2Vyc3BhY2Ug bWVtb3J5LiBXaGVuIGEKPiB1c2Vyc3BhY2UgcHJvZ3JhbSBjaGFuZ2VzIGl0J3MgbWFwcGluZywg dGhlIENBUEkgaGFyZHdhcmUgaXMgdXBkYXRlZCBzbwo+IHRoYXQgaGFyZHdhcmUgaXMgY29oZXJl bnQgd2l0aCB0aGUgdXNlciBhZGRyZXNzIHNwYWNlIGFuZCBpdCBpcyBzYWZlIHRvCj4gRE1BIHRv IGFueSBhZGRyZXNzIHdpdGhvdXQgaGF2aW5nIHRvIHBpbiBtZW1vcnkuIChUaGlzIGlzIHZlcnkg c2ltaWxhcgo+IHRvIE9EUCBpbiBSTklDcy4pIFRoaXMgaXMgKnJlYWxseSogbmljZSBidXQgZG9l c24ndCBzb2x2ZSAqYW55KiBvZiB0aGUKPiBwcm9ibGVtcyB3ZSd2ZSBiZWVuIGRpc2N1c3Npbmcu IE1vcmVvdmVyLCBtYW55IGRldmVsb3BlcnMgd2FudCB0byBrZWVwCj4gUDJQIGluLWtlcm5lbCwg Zm9yIHRoZSB0aW1lIGJlaW5nLCB3aGVyZSB0aGUgcHJvYmxlbSBvZiBwaW5uaW5nIG1lbW9yeQo+ IGRvZXMgbm90IGV4aXN0LgoKV2hhdCB5b3UgZGVzY3JpYmUgaXMgdGhlIEFUUyhBZGRyZXNzIFRy YW5zbGF0aW9uIFNlcnZpY2UpL1BBU0lEKFByb2Nlc3MKQWRkcmVzcyBTcGFjZSBJRGVudGlmaWVy KSBwYXJ0IG9mIENBUEkuIFdoaWNoIGhhdmUgYWxzbyBiZWVuIGF2YWlsYWJsZQpmb3IgeWVhcnMg b24gQU1EIHg4NiBwbGF0Zm9ybSAoQU1EIElPTU1VLXYyKSwgdGhvdWdodCBpdCBpcyBiYXJlbHkg ZXZlcgp1c2UuIEludGVyZXN0aW5nIGFzcGVjdCBvZiBDQVBJIGlzIGl0cyBjYWNoZSBjb2hlcmVu Y3kgcHJvdG9jb2wgYmV0d2VlbgpkZXZpY2VzIGFuZCBDUFVzLiBUaGlzIGluIGJvdGggZGlyZWN0 aW9uLCB0aGUgdXN1YWwgZGV2aWNlIGFjY2VzcyB0bwpzeXN0ZW0gbWVtb3J5IGNhbiBiZSBjYWNo ZSBjb2hlcmVudCB3aXRoIENQVSBhY2Nlc3MgYW5kIHBhcnRpY2lwYXRlIGluCmNhY2hlIGNvaGVy ZW5jeSBwcm90b2NvbCAoYml0IGZ1cnRoZXIgdGhhbiBQQ0lFIHNub29wKS4gQnV0IGFsc28gdGhl Cm90aGVyIGRpcmVjdGlvbiB0aGUgQ1BVIGFjY2VzcyB0byBkZXZpY2UgbWVtb3J5IGNhbiBhbHNv IGJlIGNhY2hlIGNvaGVyZW50LAp3aGljaCBpcyBub3QgdGhlIGNhc2UgaW4gUENJRS4KClRoaXMg Y2FjaGUgY29oZXJlbmN5IGJldHdlZW4gQ1BVIGFuZCBkZXZpY2UgaXMgd2hhdCBtYWRlIG1lIGFz c3VtZSB0aGF0CkNBUEkgbXVzdCBoYXZlIFBlZXIgVG8gUGVlciBzdXBwb3J0IGFzIHBlZXIgbXVz dCBiZSBhYmxlIHRvIHRhbGsgdG8gZWFjaApvdGhlciBmb3IgY2FjaGUgY29oZXJlbmN5IHB1cnBv c2UuIEJ1dCBtYXliZSBhbGwgY2FjaGUgY29oZXJlbmN5CmFyYnJpdHJhdGlvbiBnb2VzIHRocm91 Z2ggY2VudHJhbCBkaXJlY3RvcnkgYWxsaWV2YXRpbmcgUGVlciB0byBQZWVyCnJlcXVpcmVtZW50 LgoKQW55d2F5LCBsaWtlIHlvdSBzYWlkLCB0aGlzIGRvZXMgbm90IG1hdHRlciBmb3IgdGhlIGRp c2N1c3Npb24uIFRoZQpkbWFfbWFwX3Jlc291cmNlKCkgY2FuIGJlIGp1c3Qgc3R1YiBvdXQgb24g cGxhdGZvcm0gdGhhdCBkbyBub3Qgc3VwcG9ydAp0aGlzIGFuZCB0aGV5IHdvdWxkIG5vdCBhbGxv dyBpdC4gSWYgaXQgZ2V0IHVzZSBvbiBvdGhlciBwbGF0Zm9ybSBhbmQKc2hvd3MgZW5vdWdoIGFk dmFudGFnZXMgdGhhdCB1c2VycyBzdGFydCBhc2tpbmcgZm9yIGl0IHRoZW4gbWF5YmUgdGhvc2UK cGxhdGZvcm0gd2lsbCBhdHRlbnRpb24gdG8gdGhlIGhhcmR3YXJlIHJlcXVpcmVtZW50LgoKTm90 ZSB0aGF0IHdpdGggbW11X25vdGlmaWVyIHRoZXJlIGlzbid0IGFueSBuZWVkIHRvIHBpbiBzdHVm ZiAoZXZlbgp3aXRob3V0IGFueSBzcGVjaWFsIGhhcmR3YXJlIGNhcGFiaWxpdGllcyksIGFzIGxv bmcgYXMgeW91IGNhbiBwcmVlbXB0CndoYXQgaXMgaGFwcGVuaW5nIG9uIHlvdXIgaGFyZHdhcmUg dG8gdXBkYXRlIGl0cyBwYWdlIHRhYmxlLgoKQ2hlZXJzLApKw6lyw7RtZQpfX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0 CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3Rv cC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmktZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47708 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752321AbeDBTQx (ORCPT ); Mon, 2 Apr 2018 15:16:53 -0400 Date: Mon, 2 Apr 2018 15:16:50 -0400 From: Jerome Glisse To: Logan Gunthorpe Cc: Christian =?iso-8859-1?Q?K=F6nig?= , Christoph Hellwig , Will Davis , Joerg Roedel , linaro-mm-sig@lists.linaro.org, amd-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, Bjorn Helgaas Subject: Re: [PATCH 2/8] PCI: Add pci_find_common_upstream_dev() Message-ID: <20180402191649.GB18231@redhat.com> References: <70adc2cc-f7aa-d4b9-7d7a-71f3ae99f16c@gmail.com> <98ce6cfd-bcf3-811e-a0f1-757b60da467a@deltatee.com> <8d050848-8970-b8c4-a657-429fefd31769@amd.com> <20180330015854.GA3572@redhat.com> <0234bc5e-495e-0f68-fb0a-debb17a35761@deltatee.com> <20180330194519.GC3198@redhat.com> <31266710-f6bb-99ee-c73d-6e58afe5c38c@deltatee.com> <20180402172027.GA18231@redhat.com> <6f796779-0ba3-d056-de33-341ee55d6b38@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <6f796779-0ba3-d056-de33-341ee55d6b38@deltatee.com> Sender: linux-media-owner@vger.kernel.org List-ID: On Mon, Apr 02, 2018 at 11:37:07AM -0600, Logan Gunthorpe wrote: > > > On 02/04/18 11:20 AM, Jerome Glisse wrote: > > The point i have been trying to get accross is that you do have this > > information with dma_map_resource() you know the device to which you > > are trying to map (dev argument to dma_map_resource()) and you can > > easily get the device to which the memory belongs because you have the > > CPU physical address of the memory hence you can lookup the resource > > and get the device from that. > > How do you go from a physical address to a struct device generally and > in a performant manner? There isn't good API at the moment AFAIK, closest thing would either be lookup_resource() or region_intersects(), but a more appropriate one can easily be added, code to walk down the tree is readily available. More- over this can be optimize like vma lookup are, even more as resource are seldomly added so read side (finding a resource) can be heavily favor over write side (adding|registering a new resource). > > > IIRC CAPI make P2P mandatory but maybe this is with NVLink. We can ask > > the PowerPC folks to confirm. Note CAPI is Power8 and newer AFAICT. > > PowerPC folks recently told us specifically that Power9 does not support > P2P between PCI root ports. I've said this many times. CAPI has nothing > to do with it. I need to check CAPI, i must have confuse that with NVLink which is also on some powerpc arch. > > > Mapping to userspace have nothing to do here. I am talking at hardware > > level. How thing are expose to userspace is a completely different > > problems that do not have one solution fit all. For GPU you want this > > to be under total control of GPU drivers. For storage like persistent > > memory, you might want to expose it userspace more directly ... > > My understanding (and I worked on this a while ago) is that CAPI > hardware manages memory maps typically for userspace memory. When a > userspace program changes it's mapping, the CAPI hardware is updated so > that hardware is coherent with the user address space and it is safe to > DMA to any address without having to pin memory. (This is very similar > to ODP in RNICs.) This is *really* nice but doesn't solve *any* of the > problems we've been discussing. Moreover, many developers want to keep > P2P in-kernel, for the time being, where the problem of pinning memory > does not exist. What you describe is the ATS(Address Translation Service)/PASID(Process Address Space IDentifier) part of CAPI. Which have also been available for years on AMD x86 platform (AMD IOMMU-v2), thought it is barely ever use. Interesting aspect of CAPI is its cache coherency protocol between devices and CPUs. This in both direction, the usual device access to system memory can be cache coherent with CPU access and participate in cache coherency protocol (bit further than PCIE snoop). But also the other direction the CPU access to device memory can also be cache coherent, which is not the case in PCIE. This cache coherency between CPU and device is what made me assume that CAPI must have Peer To Peer support as peer must be able to talk to each other for cache coherency purpose. But maybe all cache coherency arbritration goes through central directory allievating Peer to Peer requirement. Anyway, like you said, this does not matter for the discussion. The dma_map_resource() can be just stub out on platform that do not support this and they would not allow it. If it get use on other platform and shows enough advantages that users start asking for it then maybe those platform will attention to the hardware requirement. Note that with mmu_notifier there isn't any need to pin stuff (even without any special hardware capabilities), as long as you can preempt what is happening on your hardware to update its page table. Cheers, Jérôme