Re: [RFC 4/8] drm/xe: Convert to per gt scheduler workers

[Matthew Brost <matthew.brost@intel.com>]

On Mon, Jul 06, 2026 at 01:27:46PM +0100, Tvrtko Ursulin wrote:
> 
> 
> On 03/07/2026 10:06, Matthew Brost wrote:
> > On Thu, Jul 02, 2026 at 03:37:41PM +0100, Tvrtko Ursulin wrote:
> > > As the submission side of xe is serialized by the global GuC CT lock, we

Profiling shows that, without backpressure on the firmware interface
channels, guc_send_ct throughput is about 4 H2G messages per
microsecond. As such, the serialization here is very likely just a
spin-wait on a short critical section.

The work items also perform work outside the CT lock, but in practice,
for the non-waiting CT-lock cases, it may not matter whether we use a
single workqueue or multiple workqueues give the slowest point is likely
the GuC dequeuing H2G.

We're also looking at doing ULLS submission in the KMD, in which case
there would be no serialization at all. Likewise, if we created
doorbells, we would not need to serialize either.

This series seems like backing Xe design into a corner for a problem
that compositors / RT submissions have a pretty reasoable different
option (bypass). More below.

> > > can easily afford to create our own per gt work queues. Whereas before
> > > kernel could create up to the number of CPUs threads, we now create one
> > > per GT.
> > > 
> > 
> > This breaks down for any execution queue that waits in a scheduler work
> > item for any reason. Long-running preemption queues do this for
> > time-slicing, multi-queue registration, and possibly a few other cases.
> > 
> > I've looked at this before and rough divide is this works for 3D but not
> > long running.
> 
> Hmmm curious. Could you expand a bit more on this?
> 
> Where do those wait, in run job or prepare job? And on what ie. where is the
> cross-dependency?

I think run_job can wait on multi-q syncs if a sync is outstanding for
the group.

> 
> Is it related to the alien work items xe "injects" onto the scheduler
> workers?

Preempt fences inject suspend/resume messages into the scheduler queue
to toggle scheduling state and wait for a suspend timeslice.
Additionally, if the state machine that tracks asynchronous firmware
commands is in an unexpected state, we wait for the firmware before
issuing another command.

As a general principle, we've accepted that it is okay to sleep because
this runs on a dedicated workqueue. We could rework that, but I'd prefer
not to, as it simplifies the code considerably.

Matt

> 
> Regards,
> 
> Tvrtko
> 
> > > Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@igalia.com>
> > > Cc: Matthew Brost <matthew.brost@intel.com>
> > > Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > > Cc: "Thomas Hellström" <thomas.hellstrom@linux.intel.com>
> > > ---
> > >   drivers/gpu/drm/xe/xe_gt.c         |  7 +++++++
> > >   drivers/gpu/drm/xe/xe_gt_types.h   |  3 +++
> > >   drivers/gpu/drm/xe/xe_guc_submit.c | 11 +++++++----
> > >   3 files changed, 17 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/drivers/gpu/drm/xe/xe_gt.c b/drivers/gpu/drm/xe/xe_gt.c
> > > index 783eb6d631b5..fb2db9eff341 100644
> > > --- a/drivers/gpu/drm/xe/xe_gt.c
> > > +++ b/drivers/gpu/drm/xe/xe_gt.c
> > > @@ -706,6 +706,8 @@ static void xe_gt_fini(void *arg)
> > >   	for (i = 0; i < XE_ENGINE_CLASS_MAX; ++i)
> > >   		xe_hw_fence_irq_finish(&gt->fence_irq[i]);
> > > +	destroy_workqueue(gt->submit_wq);
> > > +
> > >   	xe_gt_disable_host_l2_vram(gt);
> > >   }
> > > @@ -716,6 +718,11 @@ int xe_gt_init(struct xe_gt *gt)
> > >   	INIT_WORK(&gt->reset.worker, gt_reset_worker);
> > > +	gt->submit_wq = alloc_ordered_workqueue("xe-submit-gt%u",
> > > +						WQ_MEM_RECLAIM, gt->info.id);
> > > +	if (IS_ERR(gt->submit_wq))
> > > +		return PTR_ERR(gt->submit_wq);
> > > +
> > >   	for (i = 0; i < XE_ENGINE_CLASS_MAX; ++i) {
> > >   		gt->ring_ops[i] = xe_ring_ops_get(gt, i);
> > >   		xe_hw_fence_irq_init(&gt->fence_irq[i]);
> > > diff --git a/drivers/gpu/drm/xe/xe_gt_types.h b/drivers/gpu/drm/xe/xe_gt_types.h
> > > index e5588c88800a..175c42672546 100644
> > > --- a/drivers/gpu/drm/xe/xe_gt_types.h
> > > +++ b/drivers/gpu/drm/xe/xe_gt_types.h
> > > @@ -248,6 +248,9 @@ struct xe_gt {
> > >   	/** @exec_queue_ops: submission backend exec queue operations */
> > >   	const struct xe_exec_queue_ops *exec_queue_ops;
> > > +	/** @submit_wq: ... */
> > > +	struct workqueue_struct *submit_wq;
> > > +
> > >   	/**
> > >   	 * @ring_ops: ring operations for this hw engine (1 per engine class)
> > >   	 */
> > > diff --git a/drivers/gpu/drm/xe/xe_guc_submit.c b/drivers/gpu/drm/xe/xe_guc_submit.c
> > > index 9458bf477fa6..858f84dc8fed 100644
> > > --- a/drivers/gpu/drm/xe/xe_guc_submit.c
> > > +++ b/drivers/gpu/drm/xe/xe_guc_submit.c
> > > @@ -1928,11 +1928,12 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
> > >   	struct xe_gpu_scheduler *sched;
> > >   	struct xe_guc *guc = exec_queue_to_guc(q);
> > >   	struct workqueue_struct *submit_wq = NULL;
> > > +	struct xe_gt *gt = guc_to_gt(guc);
> > >   	struct xe_guc_exec_queue *ge;
> > >   	long timeout;
> > >   	int err, i;
> > > -	xe_gt_assert(guc_to_gt(guc), xe_device_uc_enabled(guc_to_xe(guc)));
> > > +	xe_gt_assert(gt, xe_device_uc_enabled(guc_to_xe(guc)));
> > >   	ge = kzalloc_obj(*ge);
> > >   	if (!ge)
> > > @@ -1964,12 +1965,14 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
> > >   		struct xe_exec_queue *primary = xe_exec_queue_multi_queue_primary(q);
> > >   		submit_wq = primary->guc->sched.base.submit_wq;
> > > +	} else {
> > > +		submit_wq = gt->submit_wq;
> > >   	}
> > >   	err = xe_sched_init(&ge->sched, &drm_sched_ops, &xe_sched_ops,
> > > -			    submit_wq, xe_lrc_ring_size() / MAX_JOB_SIZE_BYTES, 64,
> > > -			    timeout, guc_to_gt(guc)->ordered_wq, NULL,
> > > -			    q->name, gt_to_xe(q->gt)->drm.dev);
> > > +			    submit_wq, xe_lrc_ring_size() / MAX_JOB_SIZE_BYTES,
> > > +			    64, timeout, gt->ordered_wq, NULL, q->name,
> > > +			    gt_to_xe(gt)->drm.dev);
> > >   	if (err)
> > >   		goto err_release_id;
> > > -- 
> > > 2.54.0
> > > 
>