Re: [RFC/POC PATCH 00/12] POC SVM implementation in AMDGPU based on drm_gpusvm

[Matthew Brost <matthew.brost@intel.com>]

On Tue, Apr 28, 2026 at 10:23:18AM +0800, Huang, Honglei1 wrote:
> 
> 
> On 4/28/2026 4:00 AM, Felix Kuehling wrote:
> > 
> > On 2026-04-24 06:43, Huang, Honglei1 wrote:
> > > 
> > > 
> > > On 4/24/2026 4:21 AM, Matthew Brost wrote:
> > > > On Thu, Apr 23, 2026 at 07:03:52PM +0800, Huang, Honglei1 wrote:
> > > > > 
> > > > > 
> > > > > On 4/23/2026 3:18 PM, Matthew Brost wrote:
> > > > > ...
> > > > > > > > > > > > This clarifies a lot. This is
> > > > > > > > > > > > what we’d call in Xe “preemption
> > > > > > > > > > > > fence”
> > > > > > > > > > > > mode for a VM. Anytime memory is
> > > > > > > > > > > > moved, we trigger a GPU
> > > > > > > > > > > > preemption and
> > > > > > > > > > > > resume. We don’t actually
> > > > > > > > > > > > support SVM in this case;
> > > > > > > > > > > > instead, we use
> > > > > > > > > > > > “userptr binds,” which are built on gpusvm for page
> > > > > > > > > > > > collection. However,
> > > > > > > > > > > > we don’t support migrating memory to the device—though we could.
> > > > > > > > > > > > 
> > > > > > > > > > > > I’d look at how we converted
> > > > > > > > > > > > 'userptr' to be based on GPU SVM
> > > > > > > > > > > > [2]. In
> > > > > > > > > > > > this case, don’t maintain a range tree, as those—as you
> > > > > > > > > > > > suggest—are more
> > > > > > > > > > > > of an on-demand fault driver
> > > > > > > > > > > > concern. Instead, just embed
> > > > > > > > > > > > 'struct
> > > > > > > > > > > > drm_gpusvm_pages' in the VMA struct defined by the IOCTLs..
> > > > > > > > > > > > 
> > > > > > > > > > > > We could extend this to support migrating 'userptr', but we
> > > > > > > > > > > > just haven’t
> > > > > > > > > > > > done that yet—this may be what you want to do in “XNACK off..
> > > > > > > > > > > > 
> > > > > > > > > > > > [2] https://patchwork.freedesktop.org/series/146553/
> > > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > Actually we need to swith the xnack mode between on and off, so
> > > > > > > > > > > in xnack off
> > > > > > > > > > > mode, the driver operats in "implicit  prefetch mode". This may
> > > > > > > > > > > be due to
> > > > > > > > > > > compatibility with older hardware
> > > > > > > > > > > and the need for UMD runtime. We
> > > > > > > > > > > will
> > > > > > > > > > > further discuss the handling method under xnack off internally.
> > > > > > > > > > > 
> > > > > > > > 
> > > > > > > > Hi Matt,
> > > > > > > > 
> > > > > > > > I studied the xe_userptr code and the conversion series [2] you
> > > > > > > > pointed to.
> > > > > > > > 
> > > > > > > > I have a question that:
> > > > > > > > Would it be possible to reuse drm_gpusvm_range
> > > > > > > > to handle the hardware
> > > > > > > > without gpu fault feature(xnack off mode).
> > > > > > > 
> > > > > > > That’s not how we’ve done it. We embedded
> > > > > > > drm_gpusvm_pages into our VMA
> > > > > > > structure and then attached a notifier. The notifier attachment is
> > > > > > > open-coded on the Xe side, and this could be normalized and opened up
> > > > > > > for common driver use cases.
> > > > > 
> > > > > The way in xe_userptr likes the implementation in kfd_svm:
> > > > > embeded physical
> > > > > pages into structure and attach same size notifier.
> > > > > But kfd_svm is an implementation of SVM semantics, which
> > > > > supports partial
> > > > > unmap, doesn't need explicitly delete userptr ioctl calling
> > > > > when remove ,
> > > > > and doesn't need a explicitly userptr flag when creating.
> > > > > And actually there is also a existing implementation for
> > > > > userptr semantics
> > > > > in amdgpu kfd: KFD_IOC_ALLOC_MEM_FLAGS_USERPTR.
> > > > > If the no gpu fault mode can not use the drm gpu svm fram
> > > > > work, use the same
> > > > > way for xe_userptr, it seems like doing the duplicate work.
> > > > > 
> > > > > I think the core gap is we are trying to use the drmgpu_svm
> > > > > to implement a
> > > > > SVM semantics driver for no gpu fault hardware instead of
> > > > > userptr semantics.
> > > > > 
> > > > > > > 
> > > > > > > The problem with reusing drm_gpusvm_range directly is that a VMA may
> > > > > > > span multiple gpusvm notifiers—i.e., it can be
> > > > > > > larger than the notifier
> > > > > > > size. Of course, we could rework this as well.
> > > > > 
> > > > > So the "VMA spans multiple gpusvm notifiers" concern: I'd
> > > > > like to clarify
> > > > > that this is not actually a blocker for amdgpu's XNACK-off
> > > > > path, because
> > > > > amdgpu does not try to represent one user ioctl virtual
> > > > > address interval as
> > > > > a single drm_gpusvm_range.
> > > > > 
> > > > > we walk the attr interval and call drm_gpusvm_range_find_or_insert()
> > > > > repeatedly, letting gpusvm pick chunk aligned ranges bounded by
> > > > > notifier_size. One ioctl interval will create N chunk sized ranges.
> > > > > 
> > > > > > > 
> > > > > > 
> > > > > > Sorry for the double reply—I just glanced at the latest
> > > > > > series. I don’t
> > > > > > think creating a range per page of the userptr is desirable. While it
> > > > > > would work, from a time-complexity point of view I don’t think this is
> > > > > > ideal.
> > > > > > 
> > > > > > The issue with spans across multiple notifiers is real, though.
> > > > > > 
> > > > > > My rough idea would be:
> > > > > > 
> > > > > > - Give drivers an interface to create larger ranges.
> > > > > 
> > > > > So maybe we do not need to create larger ranges if we call
> > > > > drm_gpusvm_range_find_or_insert() repeatedly.
> > > > > 
> > > > 
> > > > That will be functional, but consider it from a time-complexity point of
> > > > view.
> > > > 
> > > > Multiple ranges increase the time complexity of range-tree searches.
> > > > This isn’t a huge deal, but it will show up to some extent.
> > > > 
> > > > Multiple ranges will also slow down DMA mapping and migration. We
> > > > switched over to the dma_iova_alloc/link/unlink/sync uAPI here [1].
> > > > While dma_iova_link is a relatively fast radix-tree walk, the allocation
> > > > and sync steps are where things get expensive. Therefore, it is
> > > > advantageous to perform these steps as few times as possible. For
> > > > example, if your SVM buffer is 512MB, instead of doing these steps 256
> > > > times, you do them once. The same logic applies to the migrate_vma_*
> > > > functions—they are quite expensive, so doing them in a single shot is
> > > > significantly faster.
> > > > 
> > > > The same applies to invalidations. If you can invalidate a large range
> > > > in a single shot, it will be faster. Although the logic in the notifier
> > > > should be able to zap multiple ranges in one shot (Xe does this), having
> > > > to DMA-unmap a single large range will still be faster than multiple
> > > > smaller DMA unmaps.
> > > > 
> > > > The TL;DR is if your driver knows size of SVM allocation upfront (e.g.,
> > > > an IOCTL tells you the size) it makes more sense to use a single large
> > > > struct (either embedded drm_gpusvm_pages into a VMA or we figure out an
> > > > interface to insert large ranges / notifiers).
> > > > 
> > > > [1] https://patchwork.freedesktop.org/series/160587/
> > > > 
> > > > > > 
> > > > > > - If the range fits inside a single notifier’s size → done.
> > > > > > 
> > > > > > - If the range spans multiple notifier sizes → round up to a power of
> > > > > >     two and create a larger notifier. This may overlap with existing
> > > > > >     notifiers, which is likely fine given that interval trees support
> > > > > >     overlaps (?). We’d need to double-check and test
> > > > > > this. If overlapping
> > > > > >     notifiers are not acceptable, we’d need some
> > > > > > heavy-handed notifier merge
> > > > > >     logic—it will be complicated, but isolated, so once
> > > > > > we get it right
> > > > > >     everyone can use it.
> > > > > 
> > > > > If we call drm_gpusvm_range_find_or_insert() repeatedly the
> > > > > drmgpu_svm will
> > > > > create the corresponding notifier correctly as far as I can see.
> > > > > 
> > > > 
> > > > I agree this will be functional but not ideal. You can always start the
> > > > approach you have here and optimize it later by adding the required
> > > > support in GPU SVM.
> > > > 
> > > 
> > > Hi Matt,
> > > 
> > > Really thanks for your information, this really helps a lot!
> > > 
> > > 
> > > Hi Christian, Felix,
> > > 
> > > According to the discussion with Matt on the previous thread, I'd
> > > like to align with you on the XNACK off direction before start to
> > > the series.
> > > 
> > > According to the information form Matt:
> > > when the allocation size is known doing one big operation is
> > > significantly faster than doing many small ranges, because
> > > the allocation and sync steps are where things get expensive.
> > > Doing them in a single shot is significantly faster, especially in the
> > > situlation of xnack off mode, which needs pre fault and pre map in
> > > ioctl, and the size is known.
> > > 
> > > It is confirmed that repeatedly calling
> > > drm_gpusvm_range_find_or_insert() is
> > > functional, and suggested we land it first and optimize later by adding
> > > large range support in GPU SVM core. That motivates the two phase
> > > plan below.
> > > 
> > > Phase 1
> > > - Reuse drm_gpusvm_range for XNACK-off, one ioctl interval is split
> > > by drm_gpusvm_range_find_or_insert() into
> > >   N chunk-sized ranges bounded by notifier_size, same mechanism as
> > > the fault path.
> > > - populate all ranges at ioctl / submit time instead of on fault.
> > > - Invalidation -> GPU queue stop -> rebind/restore the pages and gpu
> > > map ->restore queue
> > > 
> > > Phase 2:
> > > Add a large range / large notifier insert interface in GPU SVM core
> > > so one ioctl interval maps to a single range to improve efficiency.
> > > This needs modify the drmgpu_svm frame work.
> > > 
> > > May I know your thoughts on this plan?
> > 
> > I think drm_gpusvm_range_find_or_insert already has all the parameters
> > necessary to allocate larger notifiers and ranges. All it would take is
> > maybe adding a flag in drm_gpusvm_ctx to request larger range allocation
> > instead of arbitrary chunking.

Yes, I agree this a completely reasonable direction. Something like
override 'chunks' with a direct placement + size and then figure out the
notifier install algorithm in gpusvm layer - again this only gets tricky
if a direct placement spans multiple notifiers.

Matt

> > 
> > I agree this could be done as a second phase and is mostly work in the
> > drm_gpusvm code.
> 
> 
> Really thanks for the reply, will implement the large range feature
> according your suggestion.
> 
> Regards,
> Honglei
> 
> > 
> > Regards,
> >    Felix
> > 
> > 
> > > 
> > > Regards,
> > > Honglei
> > > 
> > > 
> > > > Matt
> > > > 
> > > > > Regards,
> > > > > Honglei
> > > > > 
> > > > > > 
> > > > > > - Finally, make sure that individual userptr pages can reside at any
> > > > > >     location.
> > > > > > 
> > > > > > Over conversely:
> > > > > > 
> > > > > > - Normalize embedding of drm_gpusvm_pages in VMA structs + notifier
> > > > > >     creation
> > > > > > 
> > > > > > - Make sure that individual userptr pages can reside at any location.
> > > > > 
> > > > > > 
> > > > > > Both options actually sound really similar after typing this out.
> > > > > > 
> > > > > > Matt
> > > > > > 
> > > > > > > So either way, the Xe userptr + gpusvm
> > > > > > > implementation should be refined
> > > > > > > further for common driver use.
> > > > > > > 
> > > > > > > > 
> > > > > > > > Reusing drm_gpusvm_range for the XNACK-off case would simplify our
> > > > > > > > implementation considerably, it already provides large page chunk
> > > > > > > > optimization, can reuse the existing migration infrastructure.
> > > > > > > > 
> > > > > > > > Building these on top of a standalone drm_gpusvm_pages
> > > > > > > > would mean reimplementing much of what the range
> > > > > > > > layer already offers.
> > > > > > > > It would also let us keep a single code path for both XNACK modes,
> > > > > > > > which reduces maintenance burden and avoids behavioral difference.
> > > > > > > > 
> > > > > > > > Would this direction be acceptable, or do you
> > > > > > > > see concerns with reusing
> > > > > > > > the range infrastructure for the no-fault case?
> > > > > > > > 
> > > > > > > 
> > > > > > > If you prefer something like insert a range exactly
> > > > > > > here + create range
> > > > > > > + notifier I think that completely reasonable direction and Xe would
> > > > > > > likely switch over to using this.
> > > > > > > 
> > > > > > > I guess my only concern is sub-userptr migration. We are trending
> > > > > > > towards allowing userptrs to being migrated either
> > > > > > > via prefetch IOCTLs
> > > > > > > or access counters on the GPU side - access counter
> > > > > > > we'd likely a single
> > > > > > > 2M page at time migration within the userptr.
> > > > > > > get_pages() supports mixed
> > > > > > > mappings between VRAM + system but likely needs some
> > > > > > > more work to really
> > > > > > > make this complete though.
> > > > > > > 
> > > > > > > Matt
> > > > > > > > Regards,
> > > > > > > > Honglei
> > > > > ...
> > > > > 
> > > > > 
> > > 
>