Re: [RFC 0/8] DRM scheduler kthread_worker for submission latency improvements

[Tvrtko Ursulin <tvrtko.ursulin@igalia.com>]

On 03/07/2026 09:52, Philipp Stanner wrote:
> +Cc Tejun
> 
> On Thu, 2026-07-02 at 15:37 +0100, Tvrtko Ursulin wrote:
>>
> 
> […]
> 
>> DRM scheduler was originally using kthreads but was converted workqueues due
>> desire by xe to create thousands of schedulers. This series also questions
>> whether that was needed, given how the submission is serialized by a device
>> global lock (per GT, so almost device global). Panthor has a similar situation;
>> hence the series contains two patches to move those two to a setup which matches
>> the design of those drivers.
>>
>> Other drivers, like for example amdgpu, v3d, etnaviv etc, which use the
>> scheduler as a hardware scheduler, where number of instances follow the number
>> of hardware blocks instead the number of userspace contexts, are completely
>> fine.
>>
>> There are use cases however which do currently track the number of userspace
>> contexts and which do allow for more parallelism. For those a straight
>> kthread_work conversion would be a problem due an explosion in number of
>> threads.
>>
>> The most direct example is panthor VM bind queue which creates a scheduler per
>> userspace context and relies on work queue concurrency management to keep the
>> number of threads in check.
>>
>> This creates a challenge for the kthread_work conversion. To solve which I for
>> now opted to create a trivial round-robin thread pool. For the RFC this is
>> limited to four CPU threads and is something which will need to be discussed.
>> Ie. how much parallelsim those really need. The true answer is somewhere between
>> "at most the number of active userspace contexts and the number of CPU cores".
>> Or it could be less than that, since after all, VM BIND parallelism is
>> eventually going to choke on a narrower gate of actual GPU execution. We could
>> also allow drivers to pick their number.
> 
> Anyone remember whether that was discussed back in the day of
> converting to workqueue?

AFAIR the only thing was that existing usage of kthreads did not scale 
for the new use case of one DRM scheduler per userspace context.

> What makes this idea suspicious in my mind is that the workqueue
> implementation exists to solve precisely this issue: decide how many
> kthreads really need to be spawned. AFAIK it can spawn additional
> threads dynamically if necessary.
> 
> So the first question I would like to see considered is whether
> workqueue could be improved in any way to address said latency issues.
> 
> Have you already considered that, Tvrtko?

Yes, priority inheritance was justifiably rejected for generic 
workqueues. There is WQ_HIGHPRI, but the idea here is to go one step 
further and allow tracking realtime scheduling policies.

> IOW, is it the nature of workqueue or our way of using them in
> drm_sched which causes this performance difference in the submit path?

Mostly self inflicted pain by the kthread->workqueue DRM scheduler 
conversion.

Before we had threads which were woken up on first submission and would 
fed the GPU until the software side queue was empty. With that design we 
would still have the scheduling latency on the first submit but it would 
be easy to add priority inheritance.

After the workqueue conversion, it is not only that we lost the ability 
to do priority inheritance, but it was also changed to only feed one job 
at the time to the GPU and rely on efficient worker re-queue.

> It would seem that it's the desire for more fine-grained scheduling
> characteristic control that you get with a kthread.

I think we have roughly three options:

1) Just use WQ_HIGHPRI and accept real time clients at best get re-nice 
level boost. But no RT.

2) Do something along the lines of this RFC.

3) Split the scheduler into frontend and backend parts and make 1:1 
drivers not use the actual scheduler.

The last one can be a combination of efforts. For example we can not go 
for splitting the scheduler design, but adding a new dependency sorter 
scheduler like Matthew was recently proposing. As long as that one can 
handle priority inheritance ie. not suffer this submission latency issue 
for 1:1 drivers it works.

M:N drivers we could then revert the code based back to kthreads and 
probably simplify some things.

Downside is two schedulers in the codebase..

Regards,

Tvrtko