Re: [RFC 06/13] drm/i915/svm: Page table mirroring support

From: Jerome Glisse <jglisse@redhat.com>
To: Niranjan Vishwanathapura <niranjana.vishwanathapura@intel.com>
Cc: "intel-gfx@lists.freedesktop.org"
	<intel-gfx@lists.freedesktop.org>,
	"sanjay.k.kumar@intel.com" <sanjay.k.kumar@intel.com>,
	"sudeep.dutt@intel.com" <sudeep.dutt@intel.com>,
	"dri-devel@lists.freedesktop.org"
	<dri-devel@lists.freedesktop.org>,
	"dave.hansen@intel.com" <dave.hansen@intel.com>,
	Jason Gunthorpe <jgg@mellanox.com>,
	"jon.bloomfield@intel.com" <jon.bloomfield@intel.com>,
	"daniel.vetter@intel.com" <daniel.vetter@intel.com>,
	"dan.j.williams@intel.com" <dan.j.williams@intel.com>,
	"ira.weiny@intel.com" <ira.weiny@intel.com>
Subject: Re: [RFC 06/13] drm/i915/svm: Page table mirroring support
Date: Wed, 4 Dec 2019 16:51:36 -0500	[thread overview]
Message-ID: <20191204215136.GA4049@redhat.com> (raw)
In-Reply-To: <20191203191943.GQ14488@nvishwa1-DESK.sc.intel.com>

On Tue, Dec 03, 2019 at 11:19:43AM -0800, Niranjan Vishwanathapura wrote:
> On Tue, Nov 26, 2019 at 06:32:52PM +0000, Jason Gunthorpe wrote:
> > On Mon, Nov 25, 2019 at 11:33:27AM -0500, Jerome Glisse wrote:
> > > On Fri, Nov 22, 2019 at 11:33:12PM +0000, Jason Gunthorpe wrote:
> > > > On Fri, Nov 22, 2019 at 12:57:27PM -0800, Niranjana Vishwanathapura wrote:
> > > 
> > > [...]
> > > 
> > > > > +static int
> > > > > +i915_range_fault(struct i915_svm *svm, struct hmm_range *range)
> > > > > +{
> > > > > +	long ret;
> > > > > +
> > > > > +	range->default_flags = 0;
> > > > > +	range->pfn_flags_mask = -1UL;
> > > > > +
> > > > > +	ret = hmm_range_register(range, &svm->mirror);
> > > > > +	if (ret) {
> > > > > +		up_read(&svm->mm->mmap_sem);
> > > > > +		return (int)ret;
> > > > > +	}
> > > >
> > > >
> > > > Using a temporary range is the pattern from nouveau, is it really
> > > > necessary in this driver?
> > > 
> > > Just to comment on this, for GPU the usage model is not application
> > > register range of virtual address it wants to use. It is GPU can
> > > access _any_ CPU valid address just like the CPU would (modulo mmap
> > > of device file).
> > > 
> > > This is because the API you want in userspace is application passing
> > > random pointer to the GPU and GPU being able to chase down any kind
> > > of random pointer chain (assuming all valid ie pointing to valid
> > > virtual address for the process).
> > > 
> > > This is unlike the RDMA case.
> > 
> > No, RDMA has the full address space option as well, it is called
> > 'implicit ODP'
> > 
> > This is implemented by registering ranges at a level in our page
> > table (currently 512G) whenever that level has populated pages below
> > it.
> > 
> > I think is a better strategy than temporary ranges.

HMM original design did not have range and was well suited to nouveau.
Recent changes make it more tie to the range and less suited to nouveau.
I would not consider 512GB implicit range as good thing. Plan i have is
to create implicit range and align them on vma. I do not know when i will
have time to get to that.

> > 
> > But other GPU drivers like AMD are using BO and TTM objects with fixed
> > VA ranges and the range is tied to the BO/TTM.
> > 
> > I'm not sure if this i915 usage is closer to AMD or closer to nouveau.
> > 
> 
> I don't know the full details of the HMM usecases in amd and nouveau.
> AMD seems to be using it for usrptr objects which is tied to a BO.
> I am not sure if nouveau has any BO tied to these address ranges.

It is closer to nouveau, AMD usage is old userptr usecase where you have
a BO tie to range. While SVM means any valid CPU address and thus imply
that there is no BO tie to it (there is still BO usecase that must still
work here).

Cheers,
Jérôme

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel