Re: [RESEND RFC] sched_ext: Support RT task within sched_ext

[Andrea Righi <arighi@nvidia.com>]

Hi Tangfei,

On Thu, Oct 15, 2026 at 08:47:12PM +0800, Tengfei Fan wrote:
> Sorry for resending this as plain text. The previous email was rejected
> by the sched-ext mailing list because it contained HTML.
> 
> Hi all,
> 
> We would like to discuss how RT tasks can best coordinate with sched_ext
> for cpu placement, cpu balance, and sched load reporting to schedutil
> governor. This RFC is intended to continue the discussion in email list
> from last LPC[1].
> 
> One of the options we would like to explore is allowing sched_ext to
> schedule some explicitly selected SCHED_FIFO/SCHED_RR tasks.
> 
> We received feedback from Changwoo that the issue is valid, and in
> LAVD sched-ext, they tried affining RT tasks to a limited set of CPUs as a
> workaround for the scheduler placement, but we do know that it is not a
> flexible enough solution.
> 
> We also had a discussion with Andrea Righi, that he acknowledged that
> he understands the scenario and current problem, and he would suggest
> not changing the current core scheduler, but instead using bpf ops to
> bring the RT tasks under the sched-ext control. We are trying to explore
> more on this path, and welcome to suggest or comment here.

Yeah, we discussed this with Maria at LPC, thanks for the follow up.

Essentially, my suggestion was to run selected tasks that currently request RT
scheduling under sched_ext as non-RT tasks. You could retain their requested
priority as metadata and let the BPF scheduler implement more controlled RT-like
priority classes for placement and ordering alongside the rest of the workload.

For example, the userspace part of your sched_ext scheduler could manage this
classification. When the scheduler starts, it could identify the application
threads suitable for conversion, record their current RT policy and priority in
BPF task storage and switch them to SCHED_NORMAL (which will move them to
sched_ext). And it could explicitly exclude critical kernel helpers, IRQ threads
and any application threads that genuinely need native RT behavior.

I would avoid making RT-class tasks generally eligible for sched_ext. The
sched_ext core uses FIFO helper kthreads for enable, disable, and recovery work.
The enable helper runs as RT specifically to avoid starvation during the
transition. A priority cutoff may protect those helpers, but it does not
establish that other RT threads are safe to move. Threads that genuinely require
native RT behavior should remain in the RT class. For threads using RT priority
to express frame criticality, I think a classification within sched_ext would
give a more flexible and effective way to coordinate the whole workload.

Thanks,
-Andrea

> 
> LPC topic:
> [1] https://lpc.events/event/20/contributions/2561/
> 
> Slides:
> [2] https://lpc.events/event/20/contributions/2561/attachments/2025/4612/2026_LPC_Support_RT_tasks_within_sched-ext_.pdf
> 
> 
> Here is a text-based description of the main idea presented at LPC.
> If you already have that context, you can skip the section below.
> 
> 
> Background
> ----------
> 
> In most of the Android gaming, video playback, and livestreaming test
> scenarios, non-RT threads work together with RT threads to construct and
> consume a frame. Finishing the frame on time without frame jank is a
> key performance KPI. A balanced workload across CPUs with fewer
> unnecessary CPU frequency bumps is another key power KPI.
> 
> More specifically, taking a gaming scenario for example, UnityMain and
> UnityGfxDeviceW are the critical normal threads that have significantly
> higher load and usually run on the most performance-efficient cpus in
> gold/prime cluster. Other than those high load tasks, the other normal
> tasks mostly have similar workloads.
> 
> It has RT threads like the display HAL tasks, SurfaceFlinger, and the
> GPU related tasks. Those RT tasks have workloads comparable to those
> of other non-high-load normal tasks.
> 
> In our tests, sched_ext runs in switch-all mode, with
> SCX_OPS_SWITCH_PARTIAL unset. It schedules the non-RT tasks
> covered by this mode. SCHED_FIFO/SCHED_RR tasks still run in
> the native RT class.
> 
> Below, we call these non-RT tasks "SCX tasks".
> SCHED_DEADLINE tasks are not part of this proposal.
> 
> 
> Observed problems
> -----------------
> 
> cpu placement:
> 
> In our tests, RT tasks can run on CPUs used by important SCX tasks.
> This can cause the SCX tasks to be preempted, waiting longer to run,
> or lose cache locality.
> 
> Here is a diagram for easier understanding:
> 
>         RT task + important SCX task
>                     |
>                     v
>                  same CPU
>                     |
>                     v
>               CPU contention
>                     |
>         +-----------+-----------+
>         |           |           |
>    preemption   longer wait   cache impact
>         \           |           /
>          \          |          /
>           +---------+---------+
>                     |
>                     v
>        SCX work may finish later
> 
> RT tasks have higher priority, so the preemption itself is
> expected. However, from the whole scenario perspective, better CPU
> placement could reduce this interference when another suitable CPU
> is available.
> 
> 
> Load balance:
> 
> RT tasks can be scheduled in the same cluster of non-high load scx
> tasks. This can also be a preferred placement decision. While the
> load balance within the cluster is also hard to maintain.
> 
> We sometimes see RT and SCX tasks on the same CPU while other CPUs
> can become idle very shortly. We need to check whether those newly
> idle CPUs can pull and run the scx tasks. While if RT tasks are also
> in the same scx scheduler, it is easier to make decisions from a load
> balance perspective.
> 
> 
> sched util calculation:
> 
> We also see a difference in load tracking. Our implementation
> uses a frame-window model for SCX tasks, while the RT load signal
> in our setup uses PELT.
> 
> The SCX task load is flat and steady, while the RT task load
> sometimes spikes. An unnecessary CPU frequency bump up can be triggered
> by RT task load spikes.
> 
> 
> Why CPU affinity is not enough
> ------------------------------
> 
> We could use CPU affinity to keep RT and important SCX tasks
> on different CPUs. This could reduce the time they compete
> for the same CPU.
> 
> However, CPU affinity is fixed and strictly limits how tasks can move
> between CPUs. If the number of background tasks increases, CPUs perferred
> for critical SCX tasks may still have spare capacity. With fixed CPU
> affinity, non-critical SCX tasks cannot use those CPUs even when doing
> so would help balance the system load.
> 
> Ideally, the scheduler should be able to choose CPUs dynamically as
> the load changes, while still keeping tasks within their allowed CPU
> masks.
> 
> CPU affinity may still help, but it does not provide the
> flexibility we want here.
> 
> 
> Possible options
> ----------------
> 
> One option is to let sched_ext schedule some explicitly selected RT
> threads together with SCX tasks.
> 
> This could let one scheduler choose CPUs for both selected RT
> threads and SCX tasks.
> 
> For example, if other cpu is idle, or running a low priority scx task,
> or running a low load scx task, an RT task could use that CPU instead
> of a CPU running an important high load SCX task. This could reduce CPU
> contention for the frame important KPI and help balance RT and SCX tasks
> across CPUs.
> 
> Such placement should remain a preference rather than a fixed
> rule. It must respect the RT task's priority, timing needs, and
> allowed CPU mask. The other CPU must also have enough capacity.
> 
> It could also allow RT and SCX tasks to use the same utilization
> model. We would like to study whether this can make the load
> signal more stable and reduce late CPU frequency changes.
> 
> Another option is to keep all RT tasks in the native RT
> class, but provide enough information for the two schedulers
> to better coordinate CPU placement. However, this would also require
> placement and load synchronization between the RT scheduling class
> and the sched_ext class.
> 
> After the LPC discussion, we'd like to explore option 1 and use
> BPF-only hooks to achieve current needed RT tasks within scx class.
> 
> One option is to have all RT tasks with a priority lower than the SCX
>  helper thread go into sched_ext.
> 
> Migration thread, sched-ext enable/disable threads and IRQ threads need
> to remain outside sched_ext. With this option, we will need to verify
> more common scenarios. We can explore the first option first.
> 
> 
> Open questions
> --------------
> 
> * Can sched deadline can used to optimize those scenarios?
> 
>   It is possible, of course, so we put it here, welcome to share the
>   possible configurations of which tasks goes to deadline. However, here
>   we discuss only the topic of plugging RT tasks into and out of
>   sched_ext for these scenarios. For different scenarios optimization,
>   there is of course no single solution. It still can have different
>   scheduler optimization to make a specific tasks a different sched-class.
> 
> * Other ideas?
> 
> 
> We would appreciate feedback on the problem, the possible
> directions, and what we should try first.
> 
> Thanks,
> Tengfei Fan
> 
>