From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [RFC PATCH 00/13] SVM (share virtual memory) with HMM in nouveau Date: Sat, 10 Mar 2018 12:55:28 -0500 Message-ID: <20180310175528.GA3394@redhat.com> References: <20180310032141.6096-1-jglisse@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" To: christian.koenig@amd.com Cc: Ralph Campbell , nouveau@lists.freedesktop.org, Felix Kuehling , dri-devel@lists.freedesktop.org, Evgeny Baskakov , linux-mm@kvack.org, John Hubbard List-Id: nouveau.vger.kernel.org T24gU2F0LCBNYXIgMTAsIDIwMTggYXQgMDQ6MDE6NThQTSArMDEwMCwgQ2hyaXN0aWFuIEvDtm5p ZyB3cm90ZToKPiBHb29kIHRvIGhhdmUgYW4gZXhhbXBsZSBob3cgdG8gdXNlIEhNTSB3aXRoIGFu IHVwc3RyZWFtIGRyaXZlci4KCkkgaGF2ZSB0cmllZCB0byBrZWVwIGhhcmR3YXJlIHNwZWNpZmlj IGJpdHMgYW5kIG92ZXJhbCBITU0gbG9naWMgc2VwYXJhdGVkCnNvIHBlb3BsZSBjYW4gdXNlIGl0 IGFzIGFuIGV4YW1wbGUgd2l0aG91dCBuZWVkaW5nIHRvIHVuZGVyc3RhbmQgTlZpZGlhIEdQVS4K SSB0aGluayBpIGNhbiBzdGlsbCBzcGxpdCBwYXRjaGVzIGEgYml0IHNvbWUgbW9yZSBhbG9uZyB0 aGF0IGxpbmUuCgo+IEFtIDEwLjAzLjIwMTggdW0gMDQ6MjEgc2NocmllYiBqZ2xpc3NlQHJlZGhh dC5jb206Cj4gPiBUaGlzIHBhdGNoc2V0IGFkZHMgU1ZNIChTaGFyZSBWaXJ0dWFsIE1lbW9yeSkg dXNpbmcgSE1NIChIZXRlcm9nZW5lb3VzCj4gPiBNZW1vcnkgTWFuYWdlbWVudCkgdG8gdGhlIG5v dXZlYXUgZHJpdmVyLiBTVk0gbWVhbnMgdGhhdCBHUFUgdGhyZWFkcwo+ID4gc3Bhd24gYnkgR1BV IGRyaXZlciBmb3IgYSBzcGVjaWZpYyB1c2VyIHByb2Nlc3MgY2FuIGFjY2VzcyBhbnkgdmFsaWQK PiA+IENQVSBhZGRyZXNzIGluIHRoYXQgcHJvY2Vzcy4gQSB2YWxpZCBwb2ludGVyIGlzIGEgcG9p bnRlciBpbnNpZGUgYW4KPiA+IGFyZWEgY29taW5nIGZyb20gbW1hcCBvZiBwcml2YXRlLCBzaGFy ZSBvciByZWd1bGFyIGZpbGUuIFBvaW50ZXIgdG8KPiA+IGEgbW1hcCBvZiBhIGRldmljZSBmaWxl IG9yIHNwZWNpYWwgZmlsZSBhcmUgbm90IHN1cHBvcnRlZC4KPiAKPiBCVFc6IFRoZSByZWNlbnQg SU9NTVUgcGF0Y2hlcyB3aGljaCBnZW5lcmFsaXplZCB0aGUgUEFTSUQgaGFuZGxpbmcgY2FsbHMK PiB0aGlzIFNWQSBmb3Igc2hhcmVkIHZpcnR1YWwgYWRkcmVzcyBzcGFjZS4KPiAKPiBXZSBzaG91 bGQgcHJvYmFibHkgc3luYyB1cCB3aXRoIHRob3NlIGd1eXMgYXQgc29tZSBwb2ludCB3aGF0IG5h bWluZyB0byB1c2UuCgpMZXQncyBjcmVhdGUgYSBjb21taXR0ZWUgdG8gZGVjaWRlIG9uIHRoZSBu YW1lIDspCgo+IAo+ID4gVGhpcyBpcyBhbiBSRkMgZm9yIGZldyByZWFzb25zIHRlY2huaWNhbCBy ZWFzb25zIGxpc3RlZCBiZWxvdyBhbmQgYWxzbwo+ID4gYmVjYXVzZSB3ZSBhcmUgc3RpbGwgd29y a2luZyBvbiBhIHByb3BlciBvcGVuIHNvdXJjZSB1c2Vyc3BhY2UgKG5hbWVseQo+ID4gYSBPcGVu Q0wgMi4wIGZvciBub3V2ZWF1IGluc2lkZSBtZXNhKS4gT3BlbiBzb3VyY2UgdXNlcnNwYWNlIGJl aW5nIGEKPiA+IHJlcXVpcmVtZW50IGZvciB0aGUgRFJNIHN1YnN5c3RlbS4gSSBwdXNoZWQgaW4g WzFdIGEgc2ltcGxlIHN0YW5kYWxvbmUKPiA+IHByb2dyYW0gdGhhdCBjYW4gYmUgdXNlIHRvIHRl c3QgU1ZNIHRocm91Z2ggSE1NIHdpdGggbm91dmVhdS4gSSBleHBlY3QKPiA+IHdlIHdpbGwgaGF2 ZSBhIHNvbWV3aGF0IHdvcmtpbmcgdXNlcnNwYWNlIGluIHRoZSBjb21pbmcgd2Vla3MsIHdvcmsK PiA+IGJlaW5nIHdlbGwgdW5kZXJ3YXkgYW5kIHNvbWUgcGF0Y2hlcyBoYXZlIGFscmVhZHkgYmVl biBwb3N0ZWQgb24gbWVzYQo+ID4gbWFpbGluZyBsaXN0Lgo+IAo+IFlvdSBjb3VsZCB1c2UgdGhl IE9wZW5HTCBleHRlbnNpb25zIHRvIGltcG9ydCBhcmJpdHJhcnkgdXNlciBwb2ludGVycyBhcwo+ IGJyaW5ndXAgdXNlIGNhc2UgZm9yIHRoaXMuCj4gCj4gSSB3YXMgaG9waW5nIHRvIGRvIHRoZSBz YW1lIGZvciBteSBBVEMvSE1NIHdvcmsgb24gcmFkZW9uc2kgYW5kIGFzIGZhcgo+IGFzIEkga25v dyB0aGVyZSBhcmUgZXZlbiBwaWdsaXQgdGVzdHMgZm9yIHRoYXQuCgpPcGVuR0wgZXh0ZW5zaW9u cyBhcmUgYml0IHRvbyBsaW1pdGVkIHdoZW4gaSBjaGVja2VkIHRoZW0gbG9uZyB0aW1lIGFnby4K SSB0aGluayB3ZSByYXRoZXIgaGF2ZSBzb21ldGhpbmcgbGlrZSBPcGVuQ0wgcmVhZHkgc28gdGhh dCBpdCBpcyBlYXNpZXIKdG8ganVzdGlmeSBzb21lIG9mIHRoZSBtb3JlIGNvbXB1dGUgb25seSBm ZWF0dXJlcy4gTXkgdGltZWxpbmUgaXMgNC4xOApmb3IgSE1NIGluc2lkZSBub3V2ZWF1IHVwc3Ry ZWFtIChyb3VnaGx5KSBhcyBmaXJzdCBzb21lIG90aGVyIGNoYW5nZXMgdG8Kbm91dmVhdSBuZWVk IHRvIGxhbmQuIFNvIGkgYW0gdGhpbmtpbmcgKGhvcGluZyA6KSkgdGhhdCBhbGwgdGhlIHN0YXJz CndpbGwgYmUgcHJvcGVybHkgYWxpZ24gYnkgdGhlbi4KCgo+ID4gVGhleSBhcmUgd29yayB1bmRl cndheSB0byByZXZhbXAgbm91dmVhdSBjaGFubmVsIGNyZWF0aW9uIHdpdGggYSBuZXcKPiA+IHVz ZXJzcGFjZSBBUEkuIFNvIHdlIG1pZ2h0IHdhbnQgdG8gZGVsYXkgdXBzdHJlYW1pbmcgdW50aWwg dGhpcyBsYW5kcy4KPiA+IFdlIGNhbiBzdGlsIGRpc2N1c3Mgb25lIGFzcGVjdCBzcGVjaWZpYyB0 byBITU0gaGVyZSBuYW1lbHkgdGhlIGlzc3VlCj4gPiBhcm91bmQgR0VNIG9iamVjdHMgdXNlZCBm b3Igc29tZSBzcGVjaWZpYyBwYXJ0IG9mIHRoZSBHUFUuIFNvbWUgZW5naW5lCj4gPiBpbnNpZGUg dGhlIEdQVSAoZW5naW5lIGFyZSBhIEdQVSBibG9jayBsaWtlIHRoZSBkaXNwbGF5IGJsb2NrIHdo aWNoCj4gPiBpcyByZXNwb25zaWJsZSBvZiBzY2FuaW5nIG1lbW9yeSB0byBzZW5kIG91dCBhIHBp Y3R1cmUgdGhyb3VnaCBzb21lCj4gPiBjb25uZWN0b3IgZm9yIGluc3RhbmNlIEhETUkgb3IgRGlz cGxheVBvcnQpIGNhbiBvbmx5IGFjY2VzcyBtZW1vcnkKPiA+IHdpdGggdmlydHVhbCBhZGRyZXNz IGJlbG93ICgxIDw8IDQwKS4gVG8gYWNjb21vZGF0ZSB0aG9zZSB3ZSBuZWVkIHRvCj4gPiBjcmVh dGUgYSAiaG9sZSIgaW5zaWRlIHRoZSBwcm9jZXNzIGFkZHJlc3Mgc3BhY2UuIFRoaXMgcGF0Y2hz ZXQgaGF2ZQo+ID4gYSBoYWNrIGZvciB0aGF0IChwYXRjaCAxMyBIQUNLIEZPUiBITU0gQVJFQSks IGl0IHJlc2VydmVzIGEgcmFuZ2Ugb2YKPiA+IGRldmljZSBmaWxlIG9mZnNldCBzbyB0aGF0IHBy b2Nlc3MgY2FuIG1tYXAgdGhpcyByYW5nZSB3aXRoIFBST1RfTk9ORQo+ID4gdG8gY3JlYXRlIGEg aG9sZSAocHJvY2VzcyBtdXN0IG1ha2Ugc3VyZSB0aGUgaG9sZSBpcyBiZWxvdyAxIDw8IDQwKS4K PiA+IEkgZmVlbCB1bi1lYXN5IG9mIGRvaW5nIGl0IHRoaXMgd2F5IGJ1dCBtYXliZSBpdCBpcyBv ayB3aXRoIG90aGVyCj4gPiBmb2xrcy4KPiAKPiBXZWxsIHdlIGhhdmUgZXNzZW50aWFsbHkgdGhl IHNhbWUgcHJvYmxlbSB3aXRoIHByZSBnZng5IEFNRCBoYXJkd2FyZS4KPiBGZWxpeCBtaWdodCBo YXZlIHNvbWUgYWR2aXNlIGhvdyBpdCB3YXMgc29sdmVkIGZvciBIU0EuCgpIZXJlIG15IGNvbmNl cm4gaXMgYXJvdW5kIEFQSSBleHBvc2UgdG8gdXNlcnNwYWNlIGZvciB0aGlzICJob2xlIi9yZXNl cnZlZAphcmVhLiBJIGNvbnNpZGVyZWQgc2V2ZXJhbCBvcHRpb25zOgogIC0gSGF2ZSB1c2Vyc3Bh Y2UgYWxsb2NhdGUgYWxsIG9iamVjdCBuZWVkZWQgYnkgR1BVIGFuZCBtbWFwIHRoZW0KICAgIGF0 IHByb3BlciBWQSAoVmlydHVhbCBBZGRyZXNzKSB0aGlzIG5lZWQga2VybmVsIHRvIGRvIHNwZWNp YWwKICAgIGhhbmRsaW5nIGZvciB0aG9zZSBsaWtlIGJsb2NraW5nIHVzZXJzcGFjZSBhY2Nlc3Mg Zm9yIHNlbnNpdGl2ZQogICAgb2JqZWN0IChwYWdlIHRhYmxlLCBjb21tYW5kIGJ1ZmZlciwgLi4u KSBzbyBhIGxvdCBvZiBrZXJuZWwKICAgIGNoYW5nZXMuIFRoaXMgc29tZXdoYXQgbWFrZSBzZW5z ZSB3aXRoIHNvbWUgb2YgdGhlIG5vdXZlYXUgQVBJCiAgICByZXdvcmsgdGhhdCBoYXZlIG5vdCBs YW5kZWQgeWV0LgogIC0gSGF2ZSBrZXJuZWwgZGlyZWN0bHkgY3JlYXRlIGEgUFJPVF9OT05FIHZt YSBhZ2FpbnN0IGRldmljZSBmaWxlLiBOaWNlCiAgICB0aGluZyBpcyB0aGF0IGl0IGlzIGVhc2ll ciBpbiBrZXJuZWwgdG8gZmluZCBhIGhvbGUgb2YgcHJvcGVyIHNpemUKICAgIGluIHByb3BlciBy YW5nZS4gQnV0IHRoaXMgaXMgdWdseSBhbmQgaSB0aGluayBpIHdvdWxkIGJlIHN0b25lIHRvCiAg ICBkZWF0aCBpZiBpIHdlcmUgdG8gcHJvcG9zZSB0aGF0LgogIC0ganVzdCBwaWNrIGEgcmFuZ2Ug YW5kIGNyb3NzIGZpbmdlciB0aGF0IHVzZXJzcGFjZSBuZXZlciBnb3QgYW55IG9mCiAgICBpdHMg YWxsb2NhdGlvbiBpbiBpdCAoYXQgbGVhc3QgYW55IGFsbG9jYXRpb24gaXQgd2FudCB0byB1c2Ug b24gdGhlCiAgICBHUFUpLgogIC0gSGF2ZSB1c2Vyc3BhY2UgbW1hcCB3aXRoIFBST1RfTk9ORSBh IHNwZWNpZmljIHJlZ2lvbiBvZiB0aGUgZGV2aWNlCiAgICBmaWxlIHRvIGNyZWF0ZSB0aGlzIGhv bGUgKHRoaXMgaXMgd2hhdCB0aGlzIHBhdGNoc2V0IGRvZXMpLiBOb3RlCiAgICB0aGF0IFBST1Rf Tk9ORSBpcyBub3Qgc3RyaWN0bHkgbmVlZGVkIGJ1dCB0aGlzIGlzIHdoYXQgaXQgd291bGQgZ2V0 CiAgICBhcyBkZXZpY2UgZHJpdmVyIGJsb2NrIGFueSBhY2Nlc3MgdG8gaXQuCgpBbnkgb3RoZXIg c29sdXRpb24gaSBtaXNzZWQgPwoKSSBkb24ndCBsaWtlIGFueSBvZiB0aGUgYWJvdmUgLi4uIGJ1 dCB0aGlzIGlzIG1vcmUgb2YgYSB0YXN0ZSB0aGluZy4gVGhlCmxhc3Qgb3B0aW9uIGlzIHRoZSBs ZWFzdCB1Z2x5IGluIG15IHZpZXcuIEFsc28gaW4gdGhpcyBwYXRjaHNldCBpZiB1c2VyLQpzcGFj ZSBtdW5tYXAoKSB0aGUgaG9sZSBvciBhbnkgcGFydCBvZiBpdCwgaXQgZG9lcyBraWxsIEhNTSBm b3IgdGhlCnByb2Nlc3MuIFRoaXMgaXMgYW4gaW1wb3J0YW50IGRldGFpbHMgZm9yIHNlY3VyaXR5 IGFuZCBjb25zaXN0YW50IHJlc3VsdAppbiBmcm9udCBvZiBidWdneS9yb2d1ZSBhcHBsaWNhdGlv bnMuCgpPdGhlciBhc3BlY3QgYm90aGVyIG1lIHRvbywgbGlrZSBzaG91bGQgaSBjcmVhdGUgYSBy ZWdpb24gaW4gZGV2aWNlIGZpbGUKc28gdGhhdCBtbWFwIG5lZWQgdG8gaGFwcGVuIGluIHRoZSBy ZWdpb24sIG9yIHNob3VsZCBpIGp1c3QgcGljayBhIHNpbmdsZQpvZmZzZXQgdGhhdCB3b3VsZCB0 cmlnZ2VyIHRoZSBzcGVjaWFsIG1tYXAgcGF0aC4gTm93YWRheXMgaSB0aGluayBub25lCm9mIHRo ZSBkcm0gZHJpdmVyIHByb3Blcmx5IGhhbmRsZSBtbWFwIHRoYXQgY3Jvc3MgdGhlIERSTV9GSUxF X09GRlNFVApib3VuZGFyeSwgYnV0IHdlIGRvbid0IGV4cGVjdCB0aG9zZSB0byBoYXBwZW4gZWl0 aGVyLiBTbyBsYXR0ZXIgb3B0aW9uCihzaW5nbGUgb2Zmc2V0KSB3b3VsZCBtYWtlIHNlbnNlLgoK Q2hlZXJzLApKw6lyw7RtZQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fXwpkcmktZGV2ZWwgbWFpbGluZyBsaXN0CmRyaS1kZXZlbEBsaXN0cy5mcmVlZGVza3Rv cC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9kcmkt ZGV2ZWwK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-f197.google.com (mail-qk0-f197.google.com [209.85.220.197]) by kanga.kvack.org (Postfix) with ESMTP id 8DB666B0005 for ; Sat, 10 Mar 2018 12:55:32 -0500 (EST) Received: by mail-qk0-f197.google.com with SMTP id d85so4051665qke.11 for ; Sat, 10 Mar 2018 09:55:32 -0800 (PST) Received: from mx1.redhat.com (mx3-rdu2.redhat.com. [66.187.233.73]) by mx.google.com with ESMTPS id v10si3418561qth.127.2018.03.10.09.55.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 10 Mar 2018 09:55:31 -0800 (PST) Date: Sat, 10 Mar 2018 12:55:28 -0500 From: Jerome Glisse Subject: Re: [RFC PATCH 00/13] SVM (share virtual memory) with HMM in nouveau Message-ID: <20180310175528.GA3394@redhat.com> References: <20180310032141.6096-1-jglisse@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: christian.koenig@amd.com Cc: dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, Evgeny Baskakov , linux-mm@kvack.org, Ralph Campbell , John Hubbard , Felix Kuehling , "Bridgman, John" On Sat, Mar 10, 2018 at 04:01:58PM +0100, Christian Konig wrote: > Good to have an example how to use HMM with an upstream driver. I have tried to keep hardware specific bits and overal HMM logic separated so people can use it as an example without needing to understand NVidia GPU. I think i can still split patches a bit some more along that line. > Am 10.03.2018 um 04:21 schrieb jglisse@redhat.com: > > This patchset adds SVM (Share Virtual Memory) using HMM (Heterogeneous > > Memory Management) to the nouveau driver. SVM means that GPU threads > > spawn by GPU driver for a specific user process can access any valid > > CPU address in that process. A valid pointer is a pointer inside an > > area coming from mmap of private, share or regular file. Pointer to > > a mmap of a device file or special file are not supported. > > BTW: The recent IOMMU patches which generalized the PASID handling calls > this SVA for shared virtual address space. > > We should probably sync up with those guys at some point what naming to use. Let's create a committee to decide on the name ;) > > > This is an RFC for few reasons technical reasons listed below and also > > because we are still working on a proper open source userspace (namely > > a OpenCL 2.0 for nouveau inside mesa). Open source userspace being a > > requirement for the DRM subsystem. I pushed in [1] a simple standalone > > program that can be use to test SVM through HMM with nouveau. I expect > > we will have a somewhat working userspace in the coming weeks, work > > being well underway and some patches have already been posted on mesa > > mailing list. > > You could use the OpenGL extensions to import arbitrary user pointers as > bringup use case for this. > > I was hoping to do the same for my ATC/HMM work on radeonsi and as far > as I know there are even piglit tests for that. OpenGL extensions are bit too limited when i checked them long time ago. I think we rather have something like OpenCL ready so that it is easier to justify some of the more compute only features. My timeline is 4.18 for HMM inside nouveau upstream (roughly) as first some other changes to nouveau need to land. So i am thinking (hoping :)) that all the stars will be properly align by then. > > They are work underway to revamp nouveau channel creation with a new > > userspace API. So we might want to delay upstreaming until this lands. > > We can stil discuss one aspect specific to HMM here namely the issue > > around GEM objects used for some specific part of the GPU. Some engine > > inside the GPU (engine are a GPU block like the display block which > > is responsible of scaning memory to send out a picture through some > > connector for instance HDMI or DisplayPort) can only access memory > > with virtual address below (1 << 40). To accomodate those we need to > > create a "hole" inside the process address space. This patchset have > > a hack for that (patch 13 HACK FOR HMM AREA), it reserves a range of > > device file offset so that process can mmap this range with PROT_NONE > > to create a hole (process must make sure the hole is below 1 << 40). > > I feel un-easy of doing it this way but maybe it is ok with other > > folks. > > Well we have essentially the same problem with pre gfx9 AMD hardware. > Felix might have some advise how it was solved for HSA. Here my concern is around API expose to userspace for this "hole"/reserved area. I considered several options: - Have userspace allocate all object needed by GPU and mmap them at proper VA (Virtual Address) this need kernel to do special handling for those like blocking userspace access for sensitive object (page table, command buffer, ...) so a lot of kernel changes. This somewhat make sense with some of the nouveau API rework that have not landed yet. - Have kernel directly create a PROT_NONE vma against device file. Nice thing is that it is easier in kernel to find a hole of proper size in proper range. But this is ugly and i think i would be stone to death if i were to propose that. - just pick a range and cross finger that userspace never got any of its allocation in it (at least any allocation it want to use on the GPU). - Have userspace mmap with PROT_NONE a specific region of the device file to create this hole (this is what this patchset does). Note that PROT_NONE is not strictly needed but this is what it would get as device driver block any access to it. Any other solution i missed ? I don't like any of the above ... but this is more of a taste thing. The last option is the least ugly in my view. Also in this patchset if user- space munmap() the hole or any part of it, it does kill HMM for the process. This is an important details for security and consistant result in front of buggy/rogue applications. Other aspect bother me too, like should i create a region in device file so that mmap need to happen in the region, or should i just pick a single offset that would trigger the special mmap path. Nowadays i think none of the drm driver properly handle mmap that cross the DRM_FILE_OFFSET boundary, but we don't expect those to happen either. So latter option (single offset) would make sense. Cheers, Jerome