[REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[Luke.Wildhardt]

Since the DRM scheduler default policy was switched to FAIR in 7.2, sustained 100% GPU load in-game on my RX 9070 XT causes severe performance degradation. In the example title running through Proton, Project Silverfish, the foreground application degrades to roughly 10 fps or freezes outright while audio continues, and the KDE Plasma Wayland session can lock up entirely, requiring a reboot or killing the compositor to recover. The problem is not limited to games. Running the game in the background + alt tabbing, then playing a YouTube video can also cause the entire desktop to freeze. 

Then, if the game starts to stutter, either alt-tabbing, or opening up a menu in game (which reduced the load on the GPU) immediately stops the stuttering. 

The issue is reliably reproducible under sustained GPU saturation, although the time-to-failure varies. Increasing shadow load (which decreases in-game frame rate) substantially shortens the time-to-failure.

In game, MangoHud does not reveal any frametime discrepancy. 

Bisection / test matrix

Reproducer: Project Silverfish (Proton) at settings that saturate the GPU, with a second GPU client active (browser video). Other titles show the same symptom under sustained load; I have not yet re-tested those specific titles against the fix.

Kernel	Result
torvalds/master stock	FAIR (default)	FAIL
same as above + reverts below	gpu_sched.sched_policy=2 (FAIR)	FAIL
same as above + reverts below	gpu_sched.sched_policy=1 (FIFO)	PASS
7.1.5 release	PASS
drm-next	FAIR (default)	FAIL

Reverted Commits:

d09339388b77 drm/sched: Remove drm_sched_init_args->num_rqs
2833a0512b4c drm/sched: Remove drm_sched_init_args->num_rqs usage
16e7698bc04d drm/sched: Embed run queue singleton into the scheduler
77a6809f1dc3 drm/sched: Remove FIFO and RR and simplify to a single run queue
45c211ddf92a drm/sched: Switch default policy to fair
2462a0ce23b0 drm/amdgpu: Remove drm_sched_init_args->num_rqs usage



Kernel (failing):  7.2.0-rc6 (stock), also current drm-next
Kernel (working):  7.2.0-rc6-default-revert-00006-g7e5de9b07e3d with sched_policy=1
                   7.1 release
Distro:            Arch Linux
Session:           KDE Plasma 6.7.4 / Wayland (KWin), Qt 6.11.1, KF 6.28.0
CPU:               AMD Ryzen 9 9950X3D
RAM:               64 GiB
GPU:               Sapphire Pulse Radeon RX 9070 XT
                   03:00.0 [1002:7550] rev c0, subsys Sapphire 1478
Motherboard:       ASUS (AM5 / 800-series chipset)
Mesa:              26.1.6-arch1.1 (RADV, driverVersion 26.1.6)
Vulkan:            instance 1.4.357 / device 1.4.354




On the same reverted kernel build, changing only gpu_sched.sched_policy from 2 (FAIR) to 1 (FIFO) changes the result from reliably failing to passing extended stress testing.



Possible mitigation options

Given that 7.2 is late in the release cycle and 77a6809f1dc3 removes FIFO/RR as selectable policies, there is currently no runtime workaround for affected systems.


Revert 45c211ddf92a so FIFO remains the default while FAIR remains available for further testing.

If necessary, also revert 77a6809f1dc3 and dependent changes to restore runtime policy selection.

Alternatively, fix the underlying FAIR regression if a suitable fix can be identified in time for 7.2.


I mention the revert options because the justification for removing FIFO/RR included the absence of known regressions relative to those policies. This report appears to provide at least one counterexample.


I am willing to test patches responsively and collect data if necessary.  

Re: [REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[Philipp Stanner]

+Cc Tvrtko

On Sat, 2026-08-08 at 23:33 +0000, Luke.Wildhardt@proton.me wrote:
> Since the DRM scheduler default policy was switched to FAIR in 7.2, sustained 100% GPU load in-game on my RX 9070 XT causes severe performance degradation. In the example title running through Proton, Project Silverfish, the foreground application degrades to roughly 10 fps or freezes outright while audio continues, and the KDE Plasma Wayland session can lock up entirely, requiring a reboot or killing the compositor to recover. The problem is not limited to games. Running the game in the background + alt tabbing, then playing a YouTube video can also cause the entire desktop to freeze. 
> 
> Then, if the game starts to stutter, either alt-tabbing, or opening up a menu in game (which reduced the load on the GPU) immediately stops the stuttering. 
> 
> The issue is reliably reproducible under sustained GPU saturation, although the time-to-failure varies. Increasing shadow load (which decreases in-game frame rate) substantially shortens the time-to-failure.
> 
> In game, MangoHud does not reveal any frametime discrepancy. 
> 
> Bisection / test matrix
> 
> Reproducer: Project Silverfish (Proton) at settings that saturate the GPU, with a second GPU client active (browser video). Other titles show the same symptom under sustained load; I have not yet re-tested those specific titles against the fix.
> 
> Kernel	Result
> torvalds/master stock	FAIR (default)	FAIL
> same as above + reverts below	gpu_sched.sched_policy=2 (FAIR)	FAIL
> same as above + reverts below	gpu_sched.sched_policy=1 (FIFO)	PASS
> 7.1.5 release	PASS
> drm-next	FAIR (default)	FAIL
> 
> Reverted Commits:
> 
> d09339388b77 drm/sched: Remove drm_sched_init_args->num_rqs
> 2833a0512b4c drm/sched: Remove drm_sched_init_args->num_rqs usage
> 16e7698bc04d drm/sched: Embed run queue singleton into the scheduler
> 77a6809f1dc3 drm/sched: Remove FIFO and RR and simplify to a single run queue
> 45c211ddf92a drm/sched: Switch default policy to fair
> 2462a0ce23b0 drm/amdgpu: Remove drm_sched_init_args->num_rqs usage
> 
> 
> 
> Kernel (failing):  7.2.0-rc6 (stock), also current drm-next
> Kernel (working):  7.2.0-rc6-default-revert-00006-g7e5de9b07e3d with sched_policy=1
>                    7.1 release
> Distro:            Arch Linux
> Session:           KDE Plasma 6.7.4 / Wayland (KWin), Qt 6.11.1, KF 6.28.0
> CPU:               AMD Ryzen 9 9950X3D
> RAM:               64 GiB
> GPU:               Sapphire Pulse Radeon RX 9070 XT
>                    03:00.0 [1002:7550] rev c0, subsys Sapphire 1478
> Motherboard:       ASUS (AM5 / 800-series chipset)
> Mesa:              26.1.6-arch1.1 (RADV, driverVersion 26.1.6)
> Vulkan:            instance 1.4.357 / device 1.4.354
> 
> 
> 
> 
> On the same reverted kernel build, changing only gpu_sched.sched_policy from 2 (FAIR) to 1 (FIFO) changes the result from reliably failing to passing extended stress testing.
> 
> 
> 
> Possible mitigation options
> 
> Given that 7.2 is late in the release cycle and 77a6809f1dc3 removes FIFO/RR as selectable policies, there is currently no runtime workaround for affected systems.
> 
> 
> Revert 45c211ddf92a so FIFO remains the default while FAIR remains available for further testing.

That might be the wisest thing to do; but let's hear if Tvrtko has an
idea for a hotfix first.


> 
> If necessary, also revert 77a6809f1dc3 and dependent changes to restore runtime policy selection.
> 
> Alternatively, fix the underlying FAIR regression if a suitable fix can be identified in time for 7.2.
> 
> 
> I mention the revert options because the justification for removing FIFO/RR included the absence of known regressions relative to those policies. This report appears to provide at least one counterexample.
> 
> 
> I am willing to test patches responsively and collect data if necessary.  


Thanks for your effort and help. I had dared to hope that we could get
this one through without a regression, but reality always catches up..

P.

Re: [REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[Tvrtko Ursulin]

Hi,

On 09/08/2026 00:33, Luke.Wildhardt@proton.me wrote:
> Since the DRM scheduler default policy was switched to FAIR in 7.2, sustained 100% GPU load in-game on my RX 9070 XT causes severe performance degradation. In the example title running through Proton, Project Silverfish, the foreground application degrades to roughly 10 fps or freezes outright while audio continues, and the KDE Plasma Wayland session can lock up entirely, requiring a reboot or killing the compositor to recover. The problem is not limited to games. Running the game in the background + alt tabbing, then playing a YouTube video can also cause the entire desktop to freeze.
> 
> Then, if the game starts to stutter, either alt-tabbing, or opening up a menu in game (which reduced the load on the GPU) immediately stops the stuttering.
> 
> The issue is reliably reproducible under sustained GPU saturation, although the time-to-failure varies. Increasing shadow load (which decreases in-game frame rate) substantially shortens the time-to-failure.
> 
> In game, MangoHud does not reveal any frametime discrepancy.
> 
> Bisection / test matrix
> 
> Reproducer: Project Silverfish (Proton) at settings that saturate the GPU, with a second GPU client active (browser video). Other titles show the same symptom under sustained load; I have not yet re-tested those specific titles against the fix.
> 
> Kernel	Result
> torvalds/master stock	FAIR (default)	FAIL
> same as above + reverts below	gpu_sched.sched_policy=2 (FAIR)	FAIL
> same as above + reverts below	gpu_sched.sched_policy=1 (FIFO)	PASS
> 7.1.5 release	PASS
> drm-next	FAIR (default)	FAIL
> 
> Reverted Commits:
> 
> d09339388b77 drm/sched: Remove drm_sched_init_args->num_rqs
> 2833a0512b4c drm/sched: Remove drm_sched_init_args->num_rqs usage
> 16e7698bc04d drm/sched: Embed run queue singleton into the scheduler
> 77a6809f1dc3 drm/sched: Remove FIFO and RR and simplify to a single run queue
> 45c211ddf92a drm/sched: Switch default policy to fair
> 2462a0ce23b0 drm/amdgpu: Remove drm_sched_init_args->num_rqs usage
> 
> 
> 
> Kernel (failing):  7.2.0-rc6 (stock), also current drm-next
> Kernel (working):  7.2.0-rc6-default-revert-00006-g7e5de9b07e3d with sched_policy=1
>                     7.1 release
> Distro:            Arch Linux
> Session:           KDE Plasma 6.7.4 / Wayland (KWin), Qt 6.11.1, KF 6.28.0
> CPU:               AMD Ryzen 9 9950X3D
> RAM:               64 GiB
> GPU:               Sapphire Pulse Radeon RX 9070 XT
>                     03:00.0 [1002:7550] rev c0, subsys Sapphire 1478
> Motherboard:       ASUS (AM5 / 800-series chipset)
> Mesa:              26.1.6-arch1.1 (RADV, driverVersion 26.1.6)
> Vulkan:            instance 1.4.357 / device 1.4.354
> 
> 
> 
> 
> On the same reverted kernel build, changing only gpu_sched.sched_policy from 2 (FAIR) to 1 (FIFO) changes the result from reliably failing to passing extended stress testing.
> 
> 
> 
> Possible mitigation options
> 
> Given that 7.2 is late in the release cycle and 77a6809f1dc3 removes FIFO/RR as selectable policies, there is currently no runtime workaround for affected systems.
> 
> 
> Revert 45c211ddf92a so FIFO remains the default while FAIR remains available for further testing.
> 
> If necessary, also revert 77a6809f1dc3 and dependent changes to restore runtime policy selection.
> 
> Alternatively, fix the underlying FAIR regression if a suitable fix can be identified in time for 7.2.
> 
> 
> I mention the revert options because the justification for removing FIFO/RR included the absence of known regressions relative to those policies. This report appears to provide at least one counterexample.
> 
> 
> I am willing to test patches responsively and collect data if necessary.

Initially I thought it could be a missing wakeup but I couldn't spot 
any. Then I thought about the clue that the regressed/stuck state 
happens after some runtime, and is accelerated by the competing clients. 
So I thought could there be an accumulating error in vruntime handling 
somehow. The only problem I found so far is that I think the 
min_vruntime handling has a bug where if an entity never exists the 
run-queue it can get penalised by its' vruntime only growing, while the 
re-joining ones get the pull ahead of it. I think the fix is to make 
sure min_vruntime is strictly monotonic and strictly follows the last 
popped entity. On a re-read I also think that aligns with how the CPU 
scheduler does it.

Could you please test the below diff against the tip (no reverts):

diff --git a/drivers/gpu/drm/scheduler/sched_rq.c 
b/drivers/gpu/drm/scheduler/sched_rq.c
index 044546bcb5f8..6b40f0c44fe5 100644
--- a/drivers/gpu/drm/scheduler/sched_rq.c
+++ b/drivers/gpu/drm/scheduler/sched_rq.c
@@ -95,6 +95,7 @@ void drm_sched_rq_init(struct drm_sched_rq *rq)
  	INIT_LIST_HEAD(&rq->entities);
  	rq->rb_tree_root = RB_ROOT_CACHED;
  	rq->head_prio = DRM_SCHED_PRIORITY_INVALID;
+	rq->min_vruntime = 0;
  }

  /*
@@ -117,28 +118,6 @@ static const unsigned int vruntime_shift[] = {
  	[DRM_SCHED_PRIORITY_LOW]    = 7,
  };

-static ktime_t
-drm_sched_rq_get_min_vruntime(struct drm_sched_rq *rq)
-{
-	ktime_t vruntime = 0;
-	struct rb_node *rb;
-
-	lockdep_assert_held(&rq->lock);
-
-	rb = rb_first_cached(&rq->rb_tree_root);
-	if (rb) {
-		struct drm_sched_entity *entity =
-			rb_entry(rb, typeof(*entity), rb_tree_node);
-		struct drm_sched_entity_stats *stats = entity->stats;
-
-		spin_lock(&stats->lock);
-		vruntime = stats->vruntime;
-		spin_unlock(&stats->lock);
-	}
-
-	return vruntime;
-}
-
  static void
  drm_sched_entity_save_vruntime(struct drm_sched_entity *entity,
  			       ktime_t min_vruntime)
@@ -148,7 +127,7 @@ drm_sched_entity_save_vruntime(struct 
drm_sched_entity *entity,

  	spin_lock(&stats->lock);
  	vruntime = stats->vruntime;
-	if (min_vruntime && vruntime > min_vruntime)
+	if (ktime_after(vruntime, min_vruntime))
  		vruntime = ktime_sub(vruntime, min_vruntime);
  	else
  		vruntime = 0;
@@ -271,8 +250,8 @@ drm_sched_rq_add_entity(struct drm_sched_entity *entity)
  		list_add_tail(&entity->list, &rq->entities);
  	}

-	ts = drm_sched_rq_get_min_vruntime(rq);
-	ts = drm_sched_entity_restore_vruntime(entity, ts, rq->head_prio);
+	ts = drm_sched_entity_restore_vruntime(entity, rq->min_vruntime,
+					       rq->head_prio);
  	drm_sched_rq_update_tree_locked(entity, rq, ts);

  	spin_unlock(&rq->lock);
@@ -318,6 +297,7 @@ void drm_sched_rq_pop_entity(struct drm_sched_entity 
*entity)
  {
  	struct drm_sched_job *next_job;
  	struct drm_sched_rq *rq;
+	ktime_t ts;

  	/*
  	 * Update the entity's location in the min heap according to
@@ -326,18 +306,17 @@ void drm_sched_rq_pop_entity(struct 
drm_sched_entity *entity)
  	spin_lock(&entity->lock);
  	rq = entity->rq;
  	spin_lock(&rq->lock);
+
+	ts = drm_sched_entity_update_vruntime(entity);
+	if (ktime_after(ts, rq->min_vruntime))
+		rq->min_vruntime = ts;
+
  	next_job = drm_sched_entity_queue_peek(entity);
  	if (next_job) {
-		ktime_t ts;
-
-		ts = drm_sched_entity_update_vruntime(entity);
  		drm_sched_rq_update_tree_locked(entity, rq, ts);
  	} else {
-		ktime_t min_vruntime;
-
  		drm_sched_rq_remove_tree_locked(entity, rq);
-		min_vruntime = drm_sched_rq_get_min_vruntime(rq);
-		drm_sched_entity_save_vruntime(entity, min_vruntime);
+		drm_sched_entity_save_vruntime(entity, rq->min_vruntime);
  	}
  	spin_unlock(&rq->lock);
  	spin_unlock(&entity->lock);
diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h
index 363d13fc929f..892ef474c5c1 100644
--- a/include/drm/gpu_scheduler.h
+++ b/include/drm/gpu_scheduler.h
@@ -252,6 +252,7 @@ struct drm_sched_entity {
   * @entities: list of the entities to be scheduled.
   * @rb_tree_root: root of time based priority queue of entities for 
FIFO scheduling
   * @head_prio: priority of the top tree element.
+ * @min_vruntime: Minimum virtual runtime for the run-queue.
   *
   * Run queue is a set of entities scheduling command submissions for
   * one specific ring. It implements the scheduling policy that selects
@@ -263,6 +264,7 @@ struct drm_sched_rq {
  	struct list_head		entities;
  	struct rb_root_cached		rb_tree_root;
  	enum drm_sched_priority		head_prio;
+	ktime_t				min_vruntime;
  };

  /**



Regards,

Tvrtko

Re: [REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[Danilo Krummrich]

On Mon Aug 10, 2026 at 10:28 AM CEST, Philipp Stanner wrote:
>> Reverted Commits:
>> 
>> d09339388b77 drm/sched: Remove drm_sched_init_args->num_rqs
>> 2833a0512b4c drm/sched: Remove drm_sched_init_args->num_rqs usage
>> 16e7698bc04d drm/sched: Embed run queue singleton into the scheduler
>> 77a6809f1dc3 drm/sched: Remove FIFO and RR and simplify to a single run queue
>> 45c211ddf92a drm/sched: Switch default policy to fair
>> 2462a0ce23b0 drm/amdgpu: Remove drm_sched_init_args->num_rqs usage

[...]

> That might be the wisest thing to do; but let's hear if Tvrtko has an idea for
> a hotfix first.

-rc7 was released yesterday; even with a working hotfix today it'd be tricky to
ensure the hotfix does not regress other drivers or workloads with 7.2 being
just a few days ahead.

I suggest to not wait and get the reverts ready.

Thanks,
Danilo

Re: [REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[Tvrtko Ursulin]

On 10/08/2026 13:45, Danilo Krummrich wrote:
> On Mon Aug 10, 2026 at 10:28 AM CEST, Philipp Stanner wrote:
>>> Reverted Commits:
>>>
>>> d09339388b77 drm/sched: Remove drm_sched_init_args->num_rqs
>>> 2833a0512b4c drm/sched: Remove drm_sched_init_args->num_rqs usage
>>> 16e7698bc04d drm/sched: Embed run queue singleton into the scheduler
>>> 77a6809f1dc3 drm/sched: Remove FIFO and RR and simplify to a single run queue
>>> 45c211ddf92a drm/sched: Switch default policy to fair
>>> 2462a0ce23b0 drm/amdgpu: Remove drm_sched_init_args->num_rqs usage
> 
> [...]
> 
>> That might be the wisest thing to do; but let's hear if Tvrtko has an idea for
>> a hotfix first.
> 
> -rc7 was released yesterday; even with a working hotfix today it'd be tricky to
> ensure the hotfix does not regress other drivers or workloads with 7.2 being
> just a few days ahead.
> 
> I suggest to not wait and get the reverts ready.

Yes reverts would be safer. I have them in a branch at 
people.freedesktop.org/~tursulin/drm-intel drm-sched-fair-reverts, 
mostly straightforward apart from one easy conflict in 
amdgpu_xcp_release_sched. Smoke tested on Steam Deck looks fine.

Having said that, the fix for min_vruntime handling I provided earlier 
in the thread also looks fine in my testing and is simple. No 
regressions found with synthetic unit test workloads or messing around 
on the Steam Deck. But we need to wait to hear from Luke since I haven't 
been able to repro his report locally yet.

Regards,

Tvrtko

Re: [REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[Luke.Wildhardt]

Oops, forgot to reply all in my last message.

I did a quick test of the patch before I left this morning, the freezing seems to gone, but when the GPU is under the same condition, the stuttering is more uniform if that makes sense. Granted this was only a 5 minute test but the freeze was rather reliable before, I can do more testing later.

I know these things can be hard to describe, if that helps. If there's any profiling I could do or detailed logging that would help don't hesitate to ask.



-------- Original Message --------
On Monday, 08/10/26 at 06:48 Tvrtko Ursulin <tvrtko.ursulin@igalia.com> wrote:

On 10/08/2026 13:45, Danilo Krummrich wrote:
> On Mon Aug 10, 2026 at 10:28 AM CEST, Philipp Stanner wrote:
>>> Reverted Commits:
>>>
>>> d09339388b77 drm/sched: Remove drm_sched_init_args->num_rqs
>>> 2833a0512b4c drm/sched: Remove drm_sched_init_args->num_rqs usage
>>> 16e7698bc04d drm/sched: Embed run queue singleton into the scheduler
>>> 77a6809f1dc3 drm/sched: Remove FIFO and RR and simplify to a single run queue
>>> 45c211ddf92a drm/sched: Switch default policy to fair
>>> 2462a0ce23b0 drm/amdgpu: Remove drm_sched_init_args->num_rqs usage
>
> [...]
>
>> That might be the wisest thing to do; but let's hear if Tvrtko has an idea for
>> a hotfix first.
>
> -rc7 was released yesterday; even with a working hotfix today it'd be tricky to
> ensure the hotfix does not regress other drivers or workloads with 7.2 being
> just a few days ahead.
>
> I suggest to not wait and get the reverts ready.

Yes reverts would be safer. I have them in a branch at
people.freedesktop.org/~tursulin/drm-intel drm-sched-fair-reverts,
mostly straightforward apart from one easy conflict in
amdgpu_xcp_release_sched. Smoke tested on Steam Deck looks fine.

Having said that, the fix for min_vruntime handling I provided earlier
in the thread also looks fine in my testing and is simple. No
regressions found with synthetic unit test workloads or messing around
on the Steam Deck. But we need to wait to hear from Luke since I haven't
been able to repro his report locally yet.

Regards,

Tvrtko