From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [PATCH 4/8] dma-buf: add peer2peer flag Date: Tue, 3 Apr 2018 13:06:45 -0400 Message-ID: <20180403170645.GB5935@redhat.com> References: <20180325110000.2238-1-christian.koenig@amd.com> <20180325110000.2238-4-christian.koenig@amd.com> <20180329065753.GD3881@phenom.ffwll.local> <8b823458-8bdc-3217-572b-509a28aae742@gmail.com> <20180403090909.GN3881@phenom.ffwll.local> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <20180403090909.GN3881-dv86pmgwkMBes7Z6vYuT8azUEOm+Xw19@public.gmane.org> List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org Sender: "amd-gfx" To: christian.koenig-5C7GfCeVMHo@public.gmane.org, linaro-mm-sig-cunTk1MwBs8s++Sfvej+rw@public.gmane.org, linux-media-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org, amd-gfx-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org T24gVHVlLCBBcHIgMDMsIDIwMTggYXQgMTE6MDk6MDlBTSArMDIwMCwgRGFuaWVsIFZldHRlciB3 cm90ZToKPiBPbiBUaHUsIE1hciAyOSwgMjAxOCBhdCAwMTozNDoyNFBNICswMjAwLCBDaHJpc3Rp YW4gS8O2bmlnIHdyb3RlOgo+ID4gQW0gMjkuMDMuMjAxOCB1bSAwODo1NyBzY2hyaWViIERhbmll bCBWZXR0ZXI6Cj4gPiA+IE9uIFN1biwgTWFyIDI1LCAyMDE4IGF0IDEyOjU5OjU2UE0gKzAyMDAs IENocmlzdGlhbiBLw7ZuaWcgd3JvdGU6Cj4gPiA+ID4gQWRkIGEgcGVlcjJwZWVyIGZsYWcgbm90 aW5nIHRoYXQgdGhlIGltcG9ydGVyIGNhbiBkZWFsIHdpdGggZGV2aWNlCj4gPiA+ID4gcmVzb3Vy Y2VzIHdoaWNoIGFyZSBub3QgYmFja2VkIGJ5IHBhZ2VzLgo+ID4gPiA+IAo+ID4gPiA+IFNpZ25l ZC1vZmYtYnk6IENocmlzdGlhbiBLw7ZuaWcgPGNocmlzdGlhbi5rb2VuaWdAYW1kLmNvbT4KPiA+ ID4gVW0gc3RyaWN0bHkgc3BlYWtpbmcgdGhleSBhbGwgc2hvdWxkLCBidXQgdHRtIG5ldmVyIGJv dGhlcmVkIHRvIHVzZSB0aGUKPiA+ID4gcmVhbCBpbnRlcmZhY2VzIGJ1dCBqdXN0IGhhY2tlZCBh cm91bmQgdGhlIHByb3ZpZGVkIHNnIGxpc3QsIGdyYWJiaW5nIHRoZQo+ID4gPiB1bmRlcmx5aW5n IHN0cnVjdCBwYWdlcywgdGhlbiByZWJ1aWxkaW5nJnJlbWFwcGluZyB0aGUgc2cgbGlzdCBhZ2Fp bi4KPiA+IAo+ID4gQWN0dWFsbHkgdGhhdCBpc24ndCBjb3JyZWN0LiBUVE0gY29udmVydHMgdGhl bSB0byBhIGRtYSBhZGRyZXNzIGFycmF5Cj4gPiBiZWNhdXNlIGRyaXZlcnMgbmVlZCBpdCBsaWtl IHRoaXMgKGF0IGxlYXN0IG5vdXZlYXUsIHJhZGVvbiBhbmQgYW1kZ3B1KS4KPiA+IAo+ID4gSSd2 ZSBmaXhlZCByYWRlb24gYW5kIGFtZGdwdSB0byBiZSBhYmxlIHRvIGRlYWwgd2l0aG91dCBpdCBh bmQgbWFpbGVkIHdpdGgKPiA+IEJlbiBhYm91dCBub3V2ZWF1LCBidXQgdGhlIG91dGNvbWUgaXMg dGhleSBkb24ndCByZWFsbHkga25vdy4KPiA+IAo+ID4gVFRNIGl0c2VsZiBkb2Vzbid0IGhhdmUg YW55IG5lZWQgZm9yIHRoZSBwYWdlcyBvbiBpbXBvcnRlZCBCT3MgKHlvdSBjYW4ndAo+ID4gbW1h cCB0aGVtIGFueXdheSksIHRoZSByZWFsIHVuZGVybHlpbmcgcHJvYmxlbSBpcyB0aGF0IHNnIHRh YmxlcyBkb2Vzbid0Cj4gPiBwcm92aWRlIHdoYXQgZHJpdmVycyBuZWVkLgo+ID4gCj4gPiBJIHRo aW5rIHdlIGNvdWxkIHJhdGhlciBlYXNpbHkgZml4IHNnIHRhYmxlcywgYnV0IHRoYXQgaXMgYSB0 b3RhbGx5IHNlcGFyYXRlCj4gPiB0YXNrLgo+IAo+IExvb2tpbmcgYXQgcGF0Y2ggOCwgdGhlIHNn IHRhYmxlIHNlZW1zIHBlcmZlY3RseSBzdWZmaWNpZW50IHRvIGNvbnZleSB0aGUKPiByaWdodCBk bWEgYWRkcmVzc2VzIHRvIHRoZSBpbXBvcnRlci4gT2Zjb3Vyc2UgdGhlIGV4cG9ydGVyIGhhcyB0 byBzZXQgdXAKPiB0aGUgcmlnaHQga2luZCBvZiBpb21tdSBtYXBwaW5ncyB0byBtYWtlIHRoaXMg d29yay4KPiAKPiA+ID4gVGhlIGVudGlyZSBwb2ludCBvZiB1c2luZyBzZyBsaXN0cyB3YXMgZXhh Y3RseSB0byBhbGxvdyB0aGlzIHVzZSBjYXNlIG9mCj4gPiA+IHBlZXIycGVlciBkbWEgKG9yIHdl bGwgaW4gZ2VuZXJhbCBoYXZlIHNwZWNpYWwgZXhwb3J0ZXJzIHdoaWNoIG1hbmFnZWQKPiA+ID4g bWVtb3J5L0lPIHJhbmdlcyBub3QgYmFja2VkIGJ5IHN0cnVjdCBwYWdlKS4gU28gZXNzZW50aWFs bHkgeW91J3JlIGhhdmluZwo+ID4gPiBhICJJJ20gdG90YWxseSBub3QgYnJva2VuIGZsYWciIGhl cmUuCj4gPiAKPiA+IE5vLCBpbmRlcGVuZGVudCBvZiBuZWVkZWQgc3RydWN0IHBhZ2UgcG9pbnRl cnMgd2UgbmVlZCB0byBub3RlIGlmIHRoZQo+ID4gZXhwb3J0ZXIgY2FuIGhhbmRsZSBwZWVyMnBl ZXIgc3R1ZmYgZnJvbSB0aGUgaGFyZHdhcmUgc2lkZSBpbiBnZW5lcmFsLgo+ID4gCj4gPiBTbyB3 aGF0IEkndmUgZGlkIGlzIGp1c3QgdG8gc2V0IHBlZXIycGVlciBhbGxvd2VkIG9uIHRoZSBpbXBv cnRlciBiZWNhdXNlIG9mCj4gPiB0aGUgZHJpdmVyIG5lZWRzIGFuZCBjbGVhciBpdCBpbiB0aGUg ZXhwb3J0ZXIgaWYgdGhlIGhhcmR3YXJlIGNhbid0IGhhbmRsZQo+ID4gdGhhdC4KPiAKPiBUaGUg b25seSB0aGluZyB0aGUgaW1wb3J0ZXIgc2VlbXMgdG8gZG8gaXMgY2FsbCB0aGUKPiBwY2lfcGVl cl90cmFmZmljX3N1cHBvcnRlZCwgd2hpY2ggdGhlIGV4cG9ydGVyIGNvdWxkIGNhbGwgdG9vLiBX aGF0IGFtIEkKPiBtaXNzaW5nIChzaW5jZSB0aGUgc3R1cmN0X3BhZ2Ugc3R1ZmYgc291bmRzIGxp a2UgaXQncyBmaXhlZCBhbHJlYWR5IGJ5Cj4geW91KT8KPiAtRGFuaWVsCgpBRkFJSyBMb2dhbiBw YXRjaHNldCByZXF1aXJlIHRvIHJlZ2lzdGVyIGFuZCBpbml0aWFsaXplIHN0cnVjdCBwYWdlCmZv ciB0aGUgZGV2aWNlIG1lbW9yeSB5b3Ugd2FudCB0byBtYXAgKGV4cG9ydCBmcm9tIGV4cG9ydGVy IHBvaW50IG9mCnZpZXcpLgoKV2l0aCBHUFUgdGhpcyBpc24ndCBzb21ldGhpbmcgd2Ugd2FudCwg c3RydWN0IHBhZ2UgaXMgPn49IDJeNiBzbyBmb3IKNEdCIEdQVSA9IDJeNioyXjMyLzJeMTIgPSAy XjI2ID0gNjRNQiBvZiBSQU0KOEdCIEdQVSA9IDJeNioyXjMzLzJeMTIgPSAyXjI3ID0gMTI4TUIg b2YgUkFNCjE2R0IgR1BVID0gMl42KjJeMzQvMl4xMiA9IDJeMjggPSAyNTZNQiBvZiBSQU0KMzJH QiBHUFUgPSAyXjYqMl4zNC8yXjEyID0gMl4yOSA9IDUxMk1CIG9mIFJBTQoKQWxsIHRoaXMgaXMg bW9zdGx5IHdhc3RlZCBhcyBvbmx5IGEgc21hbGwgc3ViLXNldCAodGhhdCBjYW4gbm90IGJlCmNv bnN0cmFpbnQgdG8gc3BlY2lmaWMgcmFuZ2UpIHdpbGwgZXZlciBiZSBleHBvcnRlZCBhdCBhbnkg cG9pbnQgaW4KdGltZS4gRm9yIEdQVSB3b3JrIGxvYWQgdGhpcyBpcyBoYXJkbHkganVzdGlmaWFi bGUsIGV2ZW4gZm9yIEhNTSBpCmRvIG5vdCBwbGFuIHRvIHJlZ2lzdGVyIGFsbCB0aG9zZSBwYWdl cy4KCkhlbmNlIHdoeSBpIGFyZ3VlIHRoYXQgZG1hX21hcF9yZXNvdXJjZSgpIGxpa2UgdXNlIGJ5 IENocmlzdGlhbiBpcwpnb29kIGVub3VnaCBmb3IgdXMuIFBlb3BsZSB0aGF0IGNhcmUgYWJvdXQg U0cgY2FuIGZpeCB0aGF0IGJ1dCBpCnJhdGhlciBub3QgaGF2ZSB0byBkZXBlbmQgb24gdGhhdCBh bmQgd2FzdGUgc3lzdGVtIG1lbW9yeS4KCkNoZWVycywKSsOpcsO0bWUKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KYW1kLWdmeCBtYWlsaW5nIGxpc3QKYW1k LWdmeEBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcv bWFpbG1hbi9saXN0aW5mby9hbWQtZ2Z4Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mx3-rdu2.redhat.com ([66.187.233.73]:36334 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752397AbeDCRGs (ORCPT ); Tue, 3 Apr 2018 13:06:48 -0400 Date: Tue, 3 Apr 2018 13:06:45 -0400 From: Jerome Glisse To: christian.koenig@amd.com, linaro-mm-sig@lists.linaro.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 4/8] dma-buf: add peer2peer flag Message-ID: <20180403170645.GB5935@redhat.com> References: <20180325110000.2238-1-christian.koenig@amd.com> <20180325110000.2238-4-christian.koenig@amd.com> <20180329065753.GD3881@phenom.ffwll.local> <8b823458-8bdc-3217-572b-509a28aae742@gmail.com> <20180403090909.GN3881@phenom.ffwll.local> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20180403090909.GN3881@phenom.ffwll.local> Sender: linux-media-owner@vger.kernel.org List-ID: On Tue, Apr 03, 2018 at 11:09:09AM +0200, Daniel Vetter wrote: > On Thu, Mar 29, 2018 at 01:34:24PM +0200, Christian König wrote: > > Am 29.03.2018 um 08:57 schrieb Daniel Vetter: > > > On Sun, Mar 25, 2018 at 12:59:56PM +0200, Christian König wrote: > > > > Add a peer2peer flag noting that the importer can deal with device > > > > resources which are not backed by pages. > > > > > > > > Signed-off-by: Christian König > > > Um strictly speaking they all should, but ttm never bothered to use the > > > real interfaces but just hacked around the provided sg list, grabbing the > > > underlying struct pages, then rebuilding&remapping the sg list again. > > > > Actually that isn't correct. TTM converts them to a dma address array > > because drivers need it like this (at least nouveau, radeon and amdgpu). > > > > I've fixed radeon and amdgpu to be able to deal without it and mailed with > > Ben about nouveau, but the outcome is they don't really know. > > > > TTM itself doesn't have any need for the pages on imported BOs (you can't > > mmap them anyway), the real underlying problem is that sg tables doesn't > > provide what drivers need. > > > > I think we could rather easily fix sg tables, but that is a totally separate > > task. > > Looking at patch 8, the sg table seems perfectly sufficient to convey the > right dma addresses to the importer. Ofcourse the exporter has to set up > the right kind of iommu mappings to make this work. > > > > The entire point of using sg lists was exactly to allow this use case of > > > peer2peer dma (or well in general have special exporters which managed > > > memory/IO ranges not backed by struct page). So essentially you're having > > > a "I'm totally not broken flag" here. > > > > No, independent of needed struct page pointers we need to note if the > > exporter can handle peer2peer stuff from the hardware side in general. > > > > So what I've did is just to set peer2peer allowed on the importer because of > > the driver needs and clear it in the exporter if the hardware can't handle > > that. > > The only thing the importer seems to do is call the > pci_peer_traffic_supported, which the exporter could call too. What am I > missing (since the sturct_page stuff sounds like it's fixed already by > you)? > -Daniel AFAIK Logan patchset require to register and initialize struct page for the device memory you want to map (export from exporter point of view). With GPU this isn't something we want, struct page is >~= 2^6 so for 4GB GPU = 2^6*2^32/2^12 = 2^26 = 64MB of RAM 8GB GPU = 2^6*2^33/2^12 = 2^27 = 128MB of RAM 16GB GPU = 2^6*2^34/2^12 = 2^28 = 256MB of RAM 32GB GPU = 2^6*2^34/2^12 = 2^29 = 512MB of RAM All this is mostly wasted as only a small sub-set (that can not be constraint to specific range) will ever be exported at any point in time. For GPU work load this is hardly justifiable, even for HMM i do not plan to register all those pages. Hence why i argue that dma_map_resource() like use by Christian is good enough for us. People that care about SG can fix that but i rather not have to depend on that and waste system memory. Cheers, Jérôme