From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: [RFC PATCH 00/13] SVM (share virtual memory) with HMM in nouveau Date: Mon, 12 Mar 2018 13:50:58 -0400 Message-ID: <20180312175057.GC4214@redhat.com> References: <20180310032141.6096-1-jglisse@redhat.com> <20180312173009.GN8589@phenom.ffwll.local> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <20180312173009.GN8589-dv86pmgwkMBes7Z6vYuT8azUEOm+Xw19@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: nouveau-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org Sender: "Nouveau" To: christian.koenig-5C7GfCeVMHo@public.gmane.org, dri-devel-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org, nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org, Evgeny Baskakov , linux-mm-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org, Ralph Campbell , John Hubbard , Felix Kuehling , "Bridgman, John" List-Id: nouveau.vger.kernel.org T24gTW9uLCBNYXIgMTIsIDIwMTggYXQgMDY6MzA6MDlQTSArMDEwMCwgRGFuaWVsIFZldHRlciB3 cm90ZToKPiBPbiBTYXQsIE1hciAxMCwgMjAxOCBhdCAwNDowMTo1OFBNICswMTAwLCBDaHJpc3Rp YW4gSz8/bmlnIHdyb3RlOgoKWy4uLl0KCj4gPiA+IFRoZXkgYXJlIHdvcmsgdW5kZXJ3YXkgdG8g cmV2YW1wIG5vdXZlYXUgY2hhbm5lbCBjcmVhdGlvbiB3aXRoIGEgbmV3Cj4gPiA+IHVzZXJzcGFj ZSBBUEkuIFNvIHdlIG1pZ2h0IHdhbnQgdG8gZGVsYXkgdXBzdHJlYW1pbmcgdW50aWwgdGhpcyBs YW5kcy4KPiA+ID4gV2UgY2FuIHN0aWwgZGlzY3VzcyBvbmUgYXNwZWN0IHNwZWNpZmljIHRvIEhN TSBoZXJlIG5hbWVseSB0aGUgaXNzdWUKPiA+ID4gYXJvdW5kIEdFTSBvYmplY3RzIHVzZWQgZm9y IHNvbWUgc3BlY2lmaWMgcGFydCBvZiB0aGUgR1BVLiBTb21lIGVuZ2luZQo+ID4gPiBpbnNpZGUg dGhlIEdQVSAoZW5naW5lIGFyZSBhIEdQVSBibG9jayBsaWtlIHRoZSBkaXNwbGF5IGJsb2NrIHdo aWNoCj4gPiA+IGlzIHJlc3BvbnNpYmxlIG9mIHNjYW5pbmcgbWVtb3J5IHRvIHNlbmQgb3V0IGEg cGljdHVyZSB0aHJvdWdoIHNvbWUKPiA+ID4gY29ubmVjdG9yIGZvciBpbnN0YW5jZSBIRE1JIG9y IERpc3BsYXlQb3J0KSBjYW4gb25seSBhY2Nlc3MgbWVtb3J5Cj4gPiA+IHdpdGggdmlydHVhbCBh ZGRyZXNzIGJlbG93ICgxIDw8IDQwKS4gVG8gYWNjb21vZGF0ZSB0aG9zZSB3ZSBuZWVkIHRvCj4g PiA+IGNyZWF0ZSBhICJob2xlIiBpbnNpZGUgdGhlIHByb2Nlc3MgYWRkcmVzcyBzcGFjZS4gVGhp cyBwYXRjaHNldCBoYXZlCj4gPiA+IGEgaGFjayBmb3IgdGhhdCAocGF0Y2ggMTMgSEFDSyBGT1Ig SE1NIEFSRUEpLCBpdCByZXNlcnZlcyBhIHJhbmdlIG9mCj4gPiA+IGRldmljZSBmaWxlIG9mZnNl dCBzbyB0aGF0IHByb2Nlc3MgY2FuIG1tYXAgdGhpcyByYW5nZSB3aXRoIFBST1RfTk9ORQo+ID4g PiB0byBjcmVhdGUgYSBob2xlIChwcm9jZXNzIG11c3QgbWFrZSBzdXJlIHRoZSBob2xlIGlzIGJl bG93IDEgPDwgNDApLgo+ID4gPiBJIGZlZWwgdW4tZWFzeSBvZiBkb2luZyBpdCB0aGlzIHdheSBi dXQgbWF5YmUgaXQgaXMgb2sgd2l0aCBvdGhlcgo+ID4gPiBmb2xrcy4KPiA+IAo+ID4gV2VsbCB3 ZSBoYXZlIGVzc2VudGlhbGx5IHRoZSBzYW1lIHByb2JsZW0gd2l0aCBwcmUgZ2Z4OSBBTUQgaGFy ZHdhcmUuIEZlbGl4Cj4gPiBtaWdodCBoYXZlIHNvbWUgYWR2aXNlIGhvdyBpdCB3YXMgc29sdmVk IGZvciBIU0EuCj4gCj4gQ291bGRuJ3Qgd2UgZG8gYW4gaW4ta2VybmVsIGFkZHJlc3Mgc3BhY2Ug Zm9yIHRob3NlIHNwZWNpYWwgZ3B1IGJsb2Nrcz8gQXMKPiBsb25nIGFzIGl0J3MgZGlzcGxheSB0 aGUga2VybmVsIG5lZWRzIHRvIG1hbmFnZSBpdCBhbnl3YXksIGFuZCBhZGRpbmcgYQo+IDJuZCBt YXBwaW5nIHdoZW4geW91IHBpbi91bnBpbiBmb3Igc2Nhbm91dCB1c2FnZSBzaG91bGRuJ3QgcmVh bGx5IG1hdHRlcgo+IChhcyBsb25nIGFzIHlvdSBjYWNoZSB0aGUgbWFwcGluZyB1bnRpbCB0aGUg YnVmZmVyIGdldHMgdGhyb3duIG91dCBvZgo+IHZyYW0pLiBNb3JlLW9yLWxlc3Mgd2hhdCB3ZSBk byBmb3IgaTkxNSAod2hlcmUgd2UgaGF2ZSBhbiBlbnRpcmVseQo+IHNlcGFyYXRlIGFkZHJlc3Mg c3BhY2UgZm9yIHRoZXNlIHRoaW5ncyB3aGljaCBpcyA0RyBvbiB0aGUgbGF0ZXN0IGNoaXBzKS4K PiAtRGFuaWVsCgpXZSBjYW4gbm90IGRvIGFuIGluLWtlcm5lbCBhZGRyZXNzIHNwYWNlIGZvciB0 aG9zZS4gV2UgYWxyZWFkeSBoYXZlIGFuCmluIGtlcm5lbCBhZGRyZXNzIHNwYWNlIGJ1dCBpdCBk b2VzIG5vdCBhcHBseSBmb3IgdGhlIG9iamVjdCBjb25zaWRlcmVkCmhlcmUuCgpGb3IgTlZpZGlh IChpIGJlbGlldmUgdGhpcyBpcyB0aGUgc2FtZSBmb3IgQU1EIEFGQUlLKSB0aGUgb2JqZWN0cyB3 ZQphcmUgdGFsa2luZyBhYm91dCBhcmUgb2JqZWN0cyB0aGF0IG11c3QgYmUgaW4gdGhlIHNhbWUg YWRkcmVzcyBzcGFjZQphcyB0aGUgb25lIGFnYWluc3Qgd2hpY2ggcHJvY2VzcydzIHNoYWRlci9k bWEvLi4uIGdldCBleGVjdXRlZC4KCkZvciBpbnN0YW5jZSBjb21tYW5kIGJ1ZmZlciBzdWJtaXRl ZCBieSB1c2Vyc3BhY2UgbXVzdCBiZSBpbnNpZGUgYQpHRU0gb2JqZWN0IG1hcHBlZCBpbnNpZGUg dGhlIEdQVSdzIHByb2Nlc3MgYWRkcmVzcyBhZ2FpbnN0IHdoaWNoIHRoZQpjb21tYW5kIGFyZSBl eGVjdXRlZC4gTXkgdW5kZXJzdGFuZGluZyBpcyB0aGF0IHRoZSBQRklGTyAodGhlIGVuZ2luZQpv biBudiBHUFUgdGhhdCBmZXRjaCBjb21tYW5kcykgZmlyc3QgY29udGV4dCBzd2l0Y2ggdG8gYWRk cmVzcyBzcGFjZQphc3NvY2lhdGVkIHdpdGggdGhlIGNoYW5uZWwgYW5kIHRoZW4gc3RhcnRzIGZl dGNoaW5nIGNvbW1hbmRzIHdpdGgKYWxsIGFkZHJlc3MgYmVpbmcgaW50ZXJwcmV0ZWQgYWdhaW5z dCB0aGUgY2hhbm5lbCBhZGRyZXNzIHNwYWNlLgoKSGVuY2Ugd2h5IHdlIG5lZWQgdG8gcmVzZXJ2 ZSBzb21lIHJhbmdlIGluIHRoZSBwcm9jZXNzIHZpcnR1YWwgYWRkcmVzcwpzcGFjZSBpZiB3ZSB3 YW50IHRvIGRvIFNWTSBpbiBhIHNhbmUgd2F5LiBJIG1lYW4gd2UgY291bGQganVzdCBtYXAKYnVm ZmVyIGludG8gR1BVIHBhZ2UgdGFibGUgYW5kIHRoZW4gY3Jvc3MgZmluZ2VycyBhbmQgdG9lcyBo b3BwaW5nIHRoYXQKdGhlIHByb2Nlc3Mgd2lsbCBuZXZlciBnZXQgYW55IG9mIGl0cyBtbWFwIG92 ZXJsYXBwaW5nIHRob3NlIG1hcHBpbmcgOikKCkNoZWVycywKSsOpcsO0bWUKX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KTm91dmVhdSBtYWlsaW5nIGxpc3QK Tm91dmVhdUBsaXN0cy5mcmVlZGVza3RvcC5vcmcKaHR0cHM6Ly9saXN0cy5mcmVlZGVza3RvcC5v cmcvbWFpbG1hbi9saXN0aW5mby9ub3V2ZWF1Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yw0-f198.google.com (mail-yw0-f198.google.com [209.85.161.198]) by kanga.kvack.org (Postfix) with ESMTP id 1CBB46B000A for ; Mon, 12 Mar 2018 13:51:02 -0400 (EDT) Received: by mail-yw0-f198.google.com with SMTP id b7so11926590ywe.17 for ; Mon, 12 Mar 2018 10:51:02 -0700 (PDT) Received: from mx1.redhat.com (mx3-rdu2.redhat.com. [66.187.233.73]) by mx.google.com with ESMTPS id g60si3558138qtd.331.2018.03.12.10.51.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 12 Mar 2018 10:51:00 -0700 (PDT) Date: Mon, 12 Mar 2018 13:50:58 -0400 From: Jerome Glisse Subject: Re: [RFC PATCH 00/13] SVM (share virtual memory) with HMM in nouveau Message-ID: <20180312175057.GC4214@redhat.com> References: <20180310032141.6096-1-jglisse@redhat.com> <20180312173009.GN8589@phenom.ffwll.local> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20180312173009.GN8589@phenom.ffwll.local> Sender: owner-linux-mm@kvack.org List-ID: To: christian.koenig@amd.com, dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org, Evgeny Baskakov , linux-mm@kvack.org, Ralph Campbell , John Hubbard , Felix Kuehling , "Bridgman, John" On Mon, Mar 12, 2018 at 06:30:09PM +0100, Daniel Vetter wrote: > On Sat, Mar 10, 2018 at 04:01:58PM +0100, Christian K??nig wrote: [...] > > > They are work underway to revamp nouveau channel creation with a new > > > userspace API. So we might want to delay upstreaming until this lands. > > > We can stil discuss one aspect specific to HMM here namely the issue > > > around GEM objects used for some specific part of the GPU. Some engine > > > inside the GPU (engine are a GPU block like the display block which > > > is responsible of scaning memory to send out a picture through some > > > connector for instance HDMI or DisplayPort) can only access memory > > > with virtual address below (1 << 40). To accomodate those we need to > > > create a "hole" inside the process address space. This patchset have > > > a hack for that (patch 13 HACK FOR HMM AREA), it reserves a range of > > > device file offset so that process can mmap this range with PROT_NONE > > > to create a hole (process must make sure the hole is below 1 << 40). > > > I feel un-easy of doing it this way but maybe it is ok with other > > > folks. > > > > Well we have essentially the same problem with pre gfx9 AMD hardware. Felix > > might have some advise how it was solved for HSA. > > Couldn't we do an in-kernel address space for those special gpu blocks? As > long as it's display the kernel needs to manage it anyway, and adding a > 2nd mapping when you pin/unpin for scanout usage shouldn't really matter > (as long as you cache the mapping until the buffer gets thrown out of > vram). More-or-less what we do for i915 (where we have an entirely > separate address space for these things which is 4G on the latest chips). > -Daniel We can not do an in-kernel address space for those. We already have an in kernel address space but it does not apply for the object considered here. For NVidia (i believe this is the same for AMD AFAIK) the objects we are talking about are objects that must be in the same address space as the one against which process's shader/dma/... get executed. For instance command buffer submited by userspace must be inside a GEM object mapped inside the GPU's process address against which the command are executed. My understanding is that the PFIFO (the engine on nv GPU that fetch commands) first context switch to address space associated with the channel and then starts fetching commands with all address being interpreted against the channel address space. Hence why we need to reserve some range in the process virtual address space if we want to do SVM in a sane way. I mean we could just map buffer into GPU page table and then cross fingers and toes hopping that the process will never get any of its mmap overlapping those mapping :) Cheers, Jerome