Re: [PATCH] drm/xe/guc: Handle timing out of signaled jobs gracefully

[Matthew Brost <matthew.brost@intel.com>]

On Mon, Feb 26, 2024 at 10:26:02AM +0100, Thomas Hellström wrote:
> Hi,
> 
> On Fri, 2024-02-23 at 12:46 -0800, Matthew Brost wrote:
> > Timing out of signaled jobs can happen during regular operations
> > (e.g.
> > an exec queue closed immediately after last fence signaled). The TDR
> > can
> > pass the worker which free jobs. Rather than running through the TDR
> > if
> > signaled job is found, simply free it without any debug messages.
> > 
> > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > Reported-by: José Roberto de Souza <jose.souza@intel.com>
> > Closes: https://gitlab.freedesktop.org/drm/xe/kernel/-/issues/1271
> > Signed-off-by: Matthew Brost <matthew.brost@intel.com>
> > ---
> >  drivers/gpu/drm/xe/xe_guc_submit.c | 32 ++++++++++++++++++----------
> > --
> >  1 file changed, 19 insertions(+), 13 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c
> > b/drivers/gpu/drm/xe/xe_guc_submit.c
> > index ff77bc8da1b2..29748e40555f 100644
> > --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> > +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> > @@ -929,20 +929,26 @@ guc_exec_queue_timedout_job(struct
> > drm_sched_job *drm_job)
> >  	int err = -ETIME;
> >  	int i = 0;
> >  
> > -	if (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence-
> > >flags)) {
> > -		drm_notice(&xe->drm, "Timedout job: seqno=%u,
> > guc_id=%d, flags=0x%lx",
> > -			   xe_sched_job_seqno(job), q->guc->id, q-
> > >flags);
> > -		xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_KERNEL,
> > -			   "Kernel-submitted job timed out\n");
> > -		xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_VM &&
> > !exec_queue_killed(q),
> > -			   "VM job timed out on non-killed
> > execqueue\n");
> > -
> > -		simple_error_capture(q);
> > -		xe_devcoredump(job);
> > -	} else {
> > -		drm_dbg(&xe->drm, "Timedout signaled job: seqno=%u,
> > guc_id=%d, flags=0x%lx",
> > -			 xe_sched_job_seqno(job), q->guc->id, q-
> > >flags);
> > +	/*
> > +	 * TDR has fired before free job worker. Common if exec
> > queue
> > +	 * immediately closed after last fence signaled.
> > +	 */
> > +	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &job->fence-
> > >flags)) {
> Perhaps use dma_fence_is_signaled() to double-check?
> 

The reason dma_fence_is_signaled is not used here is because
dma_fence_is_signaled() can signal the hw fence. By design the only
place in the KMD hw fences should be signaled is in
xe_hw_fence.c:hw_fence_irq_run_cb. If I recall correctly, this
simplifies a bunch of things / makes the code less racey. It might be ok
to signal a hw fence here but I'd rather to stick to the current design
of hw fences only signaling in exactly 1 place.

Hope that makes sense.

Matt

> Either way 
> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> 
> > +		guc_exec_queue_free_job(drm_job);
> > +
> > +		return DRM_GPU_SCHED_STAT_NOMINAL;
> >  	}
> > +
> > +	drm_notice(&xe->drm, "Timedout job: seqno=%u, guc_id=%d,
> > flags=0x%lx",
> > +		   xe_sched_job_seqno(job), q->guc->id, q->flags);
> > +	xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_KERNEL,
> > +		   "Kernel-submitted job timed out\n");
> > +	xe_gt_WARN(q->gt, q->flags & EXEC_QUEUE_FLAG_VM &&
> > !exec_queue_killed(q),
> > +		   "VM job timed out on non-killed execqueue\n");
> > +
> > +	simple_error_capture(q);
> > +	xe_devcoredump(job);
> > +
> >  	trace_xe_sched_job_timedout(job);
> >  
> >  	/* Kill the run_job entry point */
>