Re: [RFC/POC PATCH 00/12] POC SVM implementation in AMDGPU based on drm_gpusvm

[Felix Kuehling <felix.kuehling@amd.com>]

On 2026-04-24 06:43, Huang, Honglei1 wrote:
>
>
> On 4/24/2026 4:21 AM, Matthew Brost wrote:
>> On Thu, Apr 23, 2026 at 07:03:52PM +0800, Huang, Honglei1 wrote:
>>>
>>>
>>> On 4/23/2026 3:18 PM, Matthew Brost wrote:
>>> ...
>>>>>>>>>> This clarifies a lot. This is what we’d call in Xe 
>>>>>>>>>> “preemption fence”
>>>>>>>>>> mode for a VM. Anytime memory is moved, we trigger a GPU 
>>>>>>>>>> preemption and
>>>>>>>>>> resume. We don’t actually support SVM in this case; instead, 
>>>>>>>>>> we use
>>>>>>>>>> “userptr binds,” which are built on gpusvm for page
>>>>>>>>>> collection. However,
>>>>>>>>>> we don’t support migrating memory to the device—though we could.
>>>>>>>>>>
>>>>>>>>>> I’d look at how we converted 'userptr' to be based on GPU SVM 
>>>>>>>>>> [2]. In
>>>>>>>>>> this case, don’t maintain a range tree, as those—as you
>>>>>>>>>> suggest—are more
>>>>>>>>>> of an on-demand fault driver concern. Instead, just embed 
>>>>>>>>>> 'struct
>>>>>>>>>> drm_gpusvm_pages' in the VMA struct defined by the IOCTLs..
>>>>>>>>>>
>>>>>>>>>> We could extend this to support migrating 'userptr', but we
>>>>>>>>>> just haven’t
>>>>>>>>>> done that yet—this may be what you want to do in “XNACK off..
>>>>>>>>>>
>>>>>>>>>> [2] https://patchwork.freedesktop.org/series/146553/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Actually we need to swith the xnack mode between on and off, so
>>>>>>>>> in xnack off
>>>>>>>>> mode, the driver operats in "implicit  prefetch mode". This may
>>>>>>>>> be due to
>>>>>>>>> compatibility with older hardware and the need for UMD 
>>>>>>>>> runtime. We will
>>>>>>>>> further discuss the handling method under xnack off internally.
>>>>>>>>>
>>>>>>
>>>>>> Hi Matt,
>>>>>>
>>>>>> I studied the xe_userptr code and the conversion series [2] you
>>>>>> pointed to.
>>>>>>
>>>>>> I have a question that:
>>>>>> Would it be possible to reuse drm_gpusvm_range to handle the 
>>>>>> hardware
>>>>>> without gpu fault feature(xnack off mode).
>>>>>
>>>>> That’s not how we’ve done it. We embedded drm_gpusvm_pages into 
>>>>> our VMA
>>>>> structure and then attached a notifier. The notifier attachment is
>>>>> open-coded on the Xe side, and this could be normalized and opened up
>>>>> for common driver use cases.
>>>
>>> The way in xe_userptr likes the implementation in kfd_svm: embeded 
>>> physical
>>> pages into structure and attach same size notifier.
>>> But kfd_svm is an implementation of SVM semantics, which supports 
>>> partial
>>> unmap, doesn't need explicitly delete userptr ioctl calling when 
>>> remove ,
>>> and doesn't need a explicitly userptr flag when creating.
>>> And actually there is also a existing implementation for userptr 
>>> semantics
>>> in amdgpu kfd: KFD_IOC_ALLOC_MEM_FLAGS_USERPTR.
>>> If the no gpu fault mode can not use the drm gpu svm fram work, use 
>>> the same
>>> way for xe_userptr, it seems like doing the duplicate work.
>>>
>>> I think the core gap is we are trying to use the drmgpu_svm to 
>>> implement a
>>> SVM semantics driver for no gpu fault hardware instead of userptr 
>>> semantics.
>>>
>>>>>
>>>>> The problem with reusing drm_gpusvm_range directly is that a VMA may
>>>>> span multiple gpusvm notifiers—i.e., it can be larger than the 
>>>>> notifier
>>>>> size. Of course, we could rework this as well.
>>>
>>> So the "VMA spans multiple gpusvm notifiers" concern: I'd like to 
>>> clarify
>>> that this is not actually a blocker for amdgpu's XNACK-off path, 
>>> because
>>> amdgpu does not try to represent one user ioctl virtual address 
>>> interval as
>>> a single drm_gpusvm_range.
>>>
>>> we walk the attr interval and call drm_gpusvm_range_find_or_insert()
>>> repeatedly, letting gpusvm pick chunk aligned ranges bounded by
>>> notifier_size. One ioctl interval will create N chunk sized ranges.
>>>
>>>>>
>>>>
>>>> Sorry for the double reply—I just glanced at the latest series. I 
>>>> don’t
>>>> think creating a range per page of the userptr is desirable. While it
>>>> would work, from a time-complexity point of view I don’t think this is
>>>> ideal.
>>>>
>>>> The issue with spans across multiple notifiers is real, though.
>>>>
>>>> My rough idea would be:
>>>>
>>>> - Give drivers an interface to create larger ranges.
>>>
>>> So maybe we do not need to create larger ranges if we call
>>> drm_gpusvm_range_find_or_insert() repeatedly.
>>>
>>
>> That will be functional, but consider it from a time-complexity point of
>> view.
>>
>> Multiple ranges increase the time complexity of range-tree searches.
>> This isn’t a huge deal, but it will show up to some extent.
>>
>> Multiple ranges will also slow down DMA mapping and migration. We
>> switched over to the dma_iova_alloc/link/unlink/sync uAPI here [1].
>> While dma_iova_link is a relatively fast radix-tree walk, the allocation
>> and sync steps are where things get expensive. Therefore, it is
>> advantageous to perform these steps as few times as possible. For
>> example, if your SVM buffer is 512MB, instead of doing these steps 256
>> times, you do them once. The same logic applies to the migrate_vma_*
>> functions—they are quite expensive, so doing them in a single shot is
>> significantly faster.
>>
>> The same applies to invalidations. If you can invalidate a large range
>> in a single shot, it will be faster. Although the logic in the notifier
>> should be able to zap multiple ranges in one shot (Xe does this), having
>> to DMA-unmap a single large range will still be faster than multiple
>> smaller DMA unmaps.
>>
>> The TL;DR is if your driver knows size of SVM allocation upfront (e.g.,
>> an IOCTL tells you the size) it makes more sense to use a single large
>> struct (either embedded drm_gpusvm_pages into a VMA or we figure out an
>> interface to insert large ranges / notifiers).
>>
>> [1] https://patchwork.freedesktop.org/series/160587/
>>
>>>>
>>>> - If the range fits inside a single notifier’s size → done.
>>>>
>>>> - If the range spans multiple notifier sizes → round up to a power of
>>>>     two and create a larger notifier. This may overlap with existing
>>>>     notifiers, which is likely fine given that interval trees support
>>>>     overlaps (?). We’d need to double-check and test this. If 
>>>> overlapping
>>>>     notifiers are not acceptable, we’d need some heavy-handed 
>>>> notifier merge
>>>>     logic—it will be complicated, but isolated, so once we get it 
>>>> right
>>>>     everyone can use it.
>>>
>>> If we call drm_gpusvm_range_find_or_insert() repeatedly the 
>>> drmgpu_svm will
>>> create the corresponding notifier correctly as far as I can see.
>>>
>>
>> I agree this will be functional but not ideal. You can always start the
>> approach you have here and optimize it later by adding the required
>> support in GPU SVM.
>>
>
> Hi Matt,
>
> Really thanks for your information, this really helps a lot!
>
>
> Hi Christian, Felix,
>
> According to the discussion with Matt on the previous thread, I'd like 
> to align with you on the XNACK off direction before start to the series.
>
> According to the information form Matt:
> when the allocation size is known doing one big operation is
> significantly faster than doing many small ranges, because
> the allocation and sync steps are where things get expensive.
> Doing them in a single shot is significantly faster, especially in the
> situlation of xnack off mode, which needs pre fault and pre map in 
> ioctl, and the size is known.
>
> It is confirmed that repeatedly calling 
> drm_gpusvm_range_find_or_insert() is
> functional, and suggested we land it first and optimize later by adding
> large range support in GPU SVM core. That motivates the two phase plan 
> below.
>
> Phase 1
> - Reuse drm_gpusvm_range for XNACK-off, one ioctl interval is split by 
> drm_gpusvm_range_find_or_insert() into
>   N chunk-sized ranges bounded by notifier_size, same mechanism as the 
> fault path.
> - populate all ranges at ioctl / submit time instead of on fault.
> - Invalidation -> GPU queue stop -> rebind/restore the pages and gpu 
> map ->restore queue
>
> Phase 2:
> Add a large range / large notifier insert interface in GPU SVM core
> so one ioctl interval maps to a single range to improve efficiency.
> This needs modify the drmgpu_svm frame work.
>
> May I know your thoughts on this plan?

I think drm_gpusvm_range_find_or_insert already has all the parameters 
necessary to allocate larger notifiers and ranges. All it would take is 
maybe adding a flag in drm_gpusvm_ctx to request larger range allocation 
instead of arbitrary chunking.

I agree this could be done as a second phase and is mostly work in the 
drm_gpusvm code.

Regards,
   Felix


>
> Regards,
> Honglei
>
>
>> Matt
>>
>>> Regards,
>>> Honglei
>>>
>>>>
>>>> - Finally, make sure that individual userptr pages can reside at any
>>>>     location.
>>>>
>>>> Over conversely:
>>>>
>>>> - Normalize embedding of drm_gpusvm_pages in VMA structs + notifier
>>>>     creation
>>>>
>>>> - Make sure that individual userptr pages can reside at any location.
>>>
>>>>
>>>> Both options actually sound really similar after typing this out.
>>>>
>>>> Matt
>>>>
>>>>> So either way, the Xe userptr + gpusvm implementation should be 
>>>>> refined
>>>>> further for common driver use.
>>>>>
>>>>>>
>>>>>> Reusing drm_gpusvm_range for the XNACK-off case would simplify our
>>>>>> implementation considerably, it already provides large page chunk
>>>>>> optimization, can reuse the existing migration infrastructure.
>>>>>>
>>>>>> Building these on top of a standalone drm_gpusvm_pages
>>>>>> would mean reimplementing much of what the range layer already 
>>>>>> offers.
>>>>>> It would also let us keep a single code path for both XNACK modes,
>>>>>> which reduces maintenance burden and avoids behavioral difference.
>>>>>>
>>>>>> Would this direction be acceptable, or do you see concerns with 
>>>>>> reusing
>>>>>> the range infrastructure for the no-fault case?
>>>>>>
>>>>>
>>>>> If you prefer something like insert a range exactly here + create 
>>>>> range
>>>>> + notifier I think that completely reasonable direction and Xe would
>>>>> likely switch over to using this.
>>>>>
>>>>> I guess my only concern is sub-userptr migration. We are trending
>>>>> towards allowing userptrs to being migrated either via prefetch 
>>>>> IOCTLs
>>>>> or access counters on the GPU side - access counter we'd likely a 
>>>>> single
>>>>> 2M page at time migration within the userptr. get_pages() supports 
>>>>> mixed
>>>>> mappings between VRAM + system but likely needs some more work to 
>>>>> really
>>>>> make this complete though.
>>>>>
>>>>> Matt
>>>>>> Regards,
>>>>>> Honglei
>>> ...
>>>
>>>
>