Re: [PATCH] drm/xe: Wait on external BO kernel fences in exec IOCTL

[Matthew Brost <matthew.brost@intel.com>]

On Thu, Jul 09, 2026 at 01:01:07PM +0200, Thomas Hellström wrote:
> On Fri, 2026-07-03 at 09:45 +0100, Matthew Auld wrote:
> > On 02/07/2026 22:58, Matthew Brost wrote:
> > > Before arming a user job, xe_exec_ioctl() only added the VM's
> > > dma-resv KERNEL slot as a dependency. That slot covers rebinds and
> > > the kernel operations of the VM's private BOs, but not external BOs
> > > (bo->vm == NULL), which carry their kernel operations (evictions,
> > > moves, ...) in their own dma-resv KERNEL slot.
> > > 
> > > The DMA_RESV_USAGE_KERNEL slot is the cross-driver contract for
> > > memory management operations that must complete before the BO or
> > > its
> > > backing store may be used: any accessor is required to wait on the
> > > KERNEL fences before touching the resv. By skipping the external
> > > BOs'
> > > KERNEL slots, the exec path violated that contract and could
> > > schedule
> > > a user job while a kernel operation on an external BO mapped by the
> > > VM
> > > was still in flight, racing against it and potentially reading or
> > > writing memory that was being moved.
> > > 
> > > Replace the VM-only dependency with an iteration over every object
> > > locked by the exec, adding each object's KERNEL slot as a job
> > > dependency. This covers the VM resv (rebinds and private BOs) as
> > > well
> > > as every external BO, mirroring the drm_gpuvm_resv_add_fence() call
> > > that later publishes the job fence to the same set of objects.
> > > Long-running mode continues to skip this, as before.
> > > 
> > > Fixes: dd08ebf6c352 ("drm/xe: Introduce a new DRM driver for Intel
> > > GPUs")
> > > Cc: stable@vger.kernel.org
> > > Assisted-by: GitHub_Copilot:claude-opus-4.8
> > > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > 
> > Wow, kind of surprised we missed this.
> 
> Hm. Does this actually add any additional kernel fences to the exec
> dep?
> 
> Isn't the safety mechanism we have that no valid GPU PTEs are allowed
> to be set up with active kernel fences, and in the cases (rebinds,
> munmap split) we generate a VM kernel fence. 

I think that is an arbitrary Xe-enforced rule that just happens to be
true today. A different driver could install a KERNEL fence on a shared
BO that effectively says, "don't touch this until I'm done," without
triggering any rebind flows, and we'd break.

What actually exposed this issue is some local WIP where I use the
`dma_iova_*` functions to manage TT mappings. In that model, when memory
moves, a rebind does not need to be triggered because the IOVA remains
the same. What does change is the IOVA linkage, which is protected by a
KERNEL fence. The exec IOCTL did not detect that fence and subsequently
hit a CAT[33] error.

So, in my opinion, this is fixing a clear violation of the semantics of
a KERNEL fence.

Per the doc:

 79          * Drivers *always* must wait for those fences before accessing the
 80          * resource protected by the dma_resv object. The only exception for
 81          * that is when the resource is known to be locked down in place by
 82          * pinning it previously.

Matt

> 
> So if an exec runs trying to access such a bo with an active clear, for
> example, it would typically generate a pagefault?
> 
> Thomas
> 
> 
> 
> 
> 
> > 
> > Reviewed-by: Matthew Auld <matthew.auld@intel.com>
> > 
> > > ---
> > >   drivers/gpu/drm/xe/xe_exec.c | 22 ++++++++++++++++------
> > >   1 file changed, 16 insertions(+), 6 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/xe/xe_exec.c
> > > b/drivers/gpu/drm/xe/xe_exec.c
> > > index e05dabfcd43c..d5293bc33a67 100644
> > > --- a/drivers/gpu/drm/xe/xe_exec.c
> > > +++ b/drivers/gpu/drm/xe/xe_exec.c
> > > @@ -292,13 +292,23 @@ int xe_exec_ioctl(struct drm_device *dev,
> > > void *data, struct drm_file *file)
> > >   		goto err_exec;
> > >   	}
> > >   
> > > -	/* Wait behind rebinds */
> > > +	/*
> > > +	 * Wait behind rebinds and any kernel operations
> > > (evictions, defrag
> > > +	 * moves, ...) on the VM and all external BOs. The VM's
> > > private BOs
> > > +	 * carry their kernel ops in the VM dma-resv KERNEL slot,
> > > while each
> > > +	 * external BO carries them in its own dma-resv KERNEL
> > > slot; both are
> > > +	 * covered by iterating every object locked by the exec,
> > > mirroring the
> > > +	 * drm_gpuvm_resv_add_fence() below.
> > > +	 */
> > >   	if (!xe_vm_in_lr_mode(vm)) {
> > > -		err = xe_sched_job_add_deps(job,
> > > -					    xe_vm_resv(vm),
> > > -					   
> > > DMA_RESV_USAGE_KERNEL);
> > > -		if (err)
> > > -			goto err_put_job;
> > > +		struct drm_gem_object *obj;
> > > +
> > > +		drm_exec_for_each_locked_object(exec, obj) {
> > > +			err = xe_sched_job_add_deps(job, obj-
> > > >resv,
> > > +						   
> > > DMA_RESV_USAGE_KERNEL);
> > > +			if (err)
> > > +				goto err_put_job;
> > > +		}
> > >   	}
> > >   
> > >   	for (i = 0; i < num_syncs && !err; i++)