drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Greg Ociepka]

Hi,

On an ASUS Zenbook A16 (UX3607OA, Snapdragon X2 Elite Extreme "Glymur", Adreno GPU, chip id 0x44070041) we are hitting an unrecoverable GPU recovery loop that, once triggered, can only be cleared by a full reboot.

Environment:
- kernel: linux-next next-20260730, plus a handful of local patches for this laptop (USB wakeup-source/dma-coherent DTS, CPU thermal trips, an eDP link-training retry tweak and a DP debug-print ratelimit); none of them touch the GPU, GMU, GDSC or GPU clock paths
- Mesa 26.1.5 (freedreno, a8xx)
- GMU firmware v5.2.38 (qcom/gen80100_gmu.bin) and gen80100_sqe.fw from linux-firmware
- niri (smithay-based Wayland compositor), ordinary desktop clients

Symptom: a regular client (rotating across sessions: chromium, firefox, Xwayland, thunderbird) eventually triggers a GPU fault. From that point recovery loops at roughly 1 Hz; every iteration logs:

  adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* gpu fault ring 2 fence 28f9c status 00880005 gfx_status 00880004
  adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* BR: status 00880004 rb 0002/0078 ib1 0000000000000000/0000 ib2 0000000000000000/0000 ib3 0000000000000000/0000
  adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* BV: status 00880004 rb 0000/0078 ib1 0000000000000000/0000 ib2 0000000000000000/0000 ib3 0000000000000000/0000
  msm_dpu ae01000.display-controller: [drm:recover_worker [msm]] *ERROR* 44070041: hangcheck recover!
  msm_dpu ae01000.display-controller: [drm:recover_worker [msm]] *ERROR* 44070041: offending task: <client>
  adreno 3d00000.gpu: [drm:a8xx_recover [msm]] *ERROR* cx gdsc didn't collapse

Observations across four separate incidents (2026-08-09 to 2026-08-20; longest 60+ minutes; worst measured rate 161 recover cycles in 5 minutes):

1. The reported offending client differs between incidents and can change within a single incident.
2. Killing the offending client does not stop the loop. We verified this explicitly: after SIGTERM-ing the reported victim the loop continued at an unchanged rate within seconds, and the next client that submitted work became the new "offending task". The corrupted state evidently survives in hardware/firmware rather than in the client.
3. The loop never converges on its own; only a full reboot restores the GPU. While it runs, the victim's surface stays black and the whole desktop degrades, since every recovery stalls the GPU.
4. Onset correlates with many clients creating GPU contexts in parallel, e.g. right after a session restore spawns half a dozen applications at once.

Since every iteration reports "cx gdsc didn't collapse", the recovery path apparently never succeeds in power-cycling the GPU, which would explain why the fault state persists across recovery attempts. Two questions this raises:

- Is the CX GDSC on Glymur shared with, or kept up by, other consumers (the display path, for instance) so that it cannot collapse while the panel is active - which would make this recovery flow structurally ineffective on this SoC?
- Or does gen8/GMU require a different stop/reset sequence before the collapse is attempted?

We are happy to test patches, run instrumented builds, and provide full journals or devcoredumps - the machine reproduces this every couple of days under normal desktop use.

Thanks,
Greg Ociepka

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Rob Clark]

On Thu, Aug 20, 2026 at 6:44 AM Greg Ociepka <greg@ferrisoft.com> wrote:
>
> Hi,
>
> On an ASUS Zenbook A16 (UX3607OA, Snapdragon X2 Elite Extreme "Glymur", Adreno GPU, chip id 0x44070041) we are hitting an unrecoverable GPU recovery loop that, once triggered, can only be cleared by a full reboot.
>
> Environment:
> - kernel: linux-next next-20260730, plus a handful of local patches for this laptop (USB wakeup-source/dma-coherent DTS, CPU thermal trips, an eDP link-training retry tweak and a DP debug-print ratelimit); none of them touch the GPU, GMU, GDSC or GPU clock paths
> - Mesa 26.1.5 (freedreno, a8xx)
> - GMU firmware v5.2.38 (qcom/gen80100_gmu.bin) and gen80100_sqe.fw from linux-firmware
> - niri (smithay-based Wayland compositor), ordinary desktop clients
>
> Symptom: a regular client (rotating across sessions: chromium, firefox, Xwayland, thunderbird) eventually triggers a GPU fault. From that point recovery loops at roughly 1 Hz; every iteration logs:
>
>   adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* gpu fault ring 2 fence 28f9c status 00880005 gfx_status 00880004
>   adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* BR: status 00880004 rb 0002/0078 ib1 0000000000000000/0000 ib2 0000000000000000/0000 ib3 0000000000000000/0000
>   adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* BV: status 00880004 rb 0000/0078 ib1 0000000000000000/0000 ib2 0000000000000000/0000 ib3 0000000000000000/0000
>   msm_dpu ae01000.display-controller: [drm:recover_worker [msm]] *ERROR* 44070041: hangcheck recover!
>   msm_dpu ae01000.display-controller: [drm:recover_worker [msm]] *ERROR* 44070041: offending task: <client>
>   adreno 3d00000.gpu: [drm:a8xx_recover [msm]] *ERROR* cx gdsc didn't collapse
>
> Observations across four separate incidents (2026-08-09 to 2026-08-20; longest 60+ minutes; worst measured rate 161 recover cycles in 5 minutes):
>
> 1. The reported offending client differs between incidents and can change within a single incident.
> 2. Killing the offending client does not stop the loop. We verified this explicitly: after SIGTERM-ing the reported victim the loop continued at an unchanged rate within seconds, and the next client that submitted work became the new "offending task". The corrupted state evidently survives in hardware/firmware rather than in the client.
> 3. The loop never converges on its own; only a full reboot restores the GPU. While it runs, the victim's surface stays black and the whole desktop degrades, since every recovery stalls the GPU.
> 4. Onset correlates with many clients creating GPU contexts in parallel, e.g. right after a session restore spawns half a dozen applications at once.
>
> Since every iteration reports "cx gdsc didn't collapse", the recovery path apparently never succeeds in power-cycling the GPU, which would explain why the fault state persists across recovery attempts. Two questions this raises:
>
> - Is the CX GDSC on Glymur shared with, or kept up by, other consumers (the display path, for instance) so that it cannot collapse while the panel is active - which would make this recovery flow structurally ineffective on this SoC?
> - Or does gen8/GMU require a different stop/reset sequence before the collapse is attempted?

Hmm, I've not seen this on glymur slim7x, and I've crashed the GPU
plenty while burning down the remaining clcts/glcts/vkcts issues.  And
certainly not any GPU hangs/crashes in normal desktop usage.  (Vanilla
gnome-shell / fedora f44 here.)

System mesa is 26.1.6, but for testing/debug it has been newer ToT
mesa.  I kinda suspect that the mesa version does not matter, this
sounds more like a power issue.

Fwiw, the kernel I've been using: [1]..  but that doesn't have the a16
dts. I need to move forward to something newer, hopefully I'll get to
that today.  We'll see if I start seeing this issue.  Or maybe
something x2e vs x2ee is different regarding pmic/supplies?

My device doesn't have the highest OPP (1.85ghz), which I think your
device does.  We might be still missing some power related features
that are (more?) needed at higher freq's, or maybe I have a more lucky
piece of silicon?  Akhil would know more about this.  You might try
removing the highest OPP in gpu_opp_table (in glymur.dtsi) to see if
that improves stability?  (Repeat until it is stable if you have the
same issues w/ 1.7ghz.)

BR,
-R

[1] https://gitlab.freedesktop.org/robclark/msm/-/commits/glymur-next?ref_type=heads

>
> We are happy to test patches, run instrumented builds, and provide full journals or devcoredumps - the machine reproduces this every couple of days under normal desktop use.
>
> Thanks,
> Greg Ociepka
>

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Konrad Dybcio]

On 8/21/26 3:35 PM, Rob Clark wrote:
> On Thu, Aug 20, 2026 at 6:44 AM Greg Ociepka <greg@ferrisoft.com> wrote:
>>
>> Hi,
>>
>> On an ASUS Zenbook A16 (UX3607OA, Snapdragon X2 Elite Extreme "Glymur", Adreno GPU, chip id 0x44070041) we are hitting an unrecoverable GPU recovery loop that, once triggered, can only be cleared by a full reboot.

[...]

> Fwiw, the kernel I've been using: [1]..  but that doesn't have the a16
> dts. I need to move forward to something newer, hopefully I'll get to
> that today.  We'll see if I start seeing this issue.  Or maybe
> something x2e vs x2ee is different regarding pmic/supplies?
> 
> My device doesn't have the highest OPP (1.85ghz), which I think your
> device does.  We might be still missing some power related features
> that are (more?) needed at higher freq's, or maybe I have a more lucky
> piece of silicon?  Akhil would know more about this.  You might try
> removing the highest OPP in gpu_opp_table (in glymur.dtsi) to see if
> that improves stability?  (Repeat until it is stable if you have the
> same issues w/ 1.7ghz.)

Greg, have you tried this?

Konrad

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Akhil P Oommen]

On 8/20/2026 7:14 PM, Greg Ociepka wrote:
> Hi,
> 
> On an ASUS Zenbook A16 (UX3607OA, Snapdragon X2 Elite Extreme "Glymur", Adreno GPU, chip id 0x44070041) we are hitting an unrecoverable GPU recovery loop that, once triggered, can only be cleared by a full reboot.
> 
> Environment:
> - kernel: linux-next next-20260730, plus a handful of local patches for this laptop (USB wakeup-source/dma-coherent DTS, CPU thermal trips, an eDP link-training retry tweak and a DP debug-print ratelimit); none of them touch the GPU, GMU, GDSC or GPU clock paths
> - Mesa 26.1.5 (freedreno, a8xx)
> - GMU firmware v5.2.38 (qcom/gen80100_gmu.bin) and gen80100_sqe.fw from linux-firmware
> - niri (smithay-based Wayland compositor), ordinary desktop clients
> 
> Symptom: a regular client (rotating across sessions: chromium, firefox, Xwayland, thunderbird) eventually triggers a GPU fault. From that point recovery loops at roughly 1 Hz; every iteration logs:
> 
>   adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* gpu fault ring 2 fence 28f9c status 00880005 gfx_status 00880004
>   adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* BR: status 00880004 rb 0002/0078 ib1 0000000000000000/0000 ib2 0000000000000000/0000 ib3 0000000000000000/0000
>   adreno 3d00000.gpu: [drm:a8xx_irq [msm]] *ERROR* BV: status 00880004 rb 0000/0078 ib1 0000000000000000/0000 ib2 0000000000000000/0000 ib3 0000000000000000/0000
>   msm_dpu ae01000.display-controller: [drm:recover_worker [msm]] *ERROR* 44070041: hangcheck recover!
>   msm_dpu ae01000.display-controller: [drm:recover_worker [msm]] *ERROR* 44070041: offending task: <client>
>   adreno 3d00000.gpu: [drm:a8xx_recover [msm]] *ERROR* cx gdsc didn't collapse
> 
> Observations across four separate incidents (2026-08-09 to 2026-08-20; longest 60+ minutes; worst measured rate 161 recover cycles in 5 minutes):
> 
> 1. The reported offending client differs between incidents and can change within a single incident.
> 2. Killing the offending client does not stop the loop. We verified this explicitly: after SIGTERM-ing the reported victim the loop continued at an unchanged rate within seconds, and the next client that submitted work became the new "offending task". The corrupted state evidently survives in hardware/firmware rather than in the client.
> 3. The loop never converges on its own; only a full reboot restores the GPU. While it runs, the victim's surface stays black and the whole desktop degrades, since every recovery stalls the GPU.
> 4. Onset correlates with many clients creating GPU contexts in parallel, e.g. right after a session restore spawns half a dozen applications at once.
> 
> Since every iteration reports "cx gdsc didn't collapse", the recovery path apparently never succeeds in power-cycling the GPU, which would explain why the fault state persists across recovery attempts. Two questions this raises:
> 
> - Is the CX GDSC on Glymur shared with, or kept up by, other consumers (the display path, for instance) so that it cannot collapse while the panel is active - which would make this recovery flow structurally ineffective on this SoC?
> - Or does gen8/GMU require a different stop/reset sequence before the collapse is attempted?
> 
> We are happy to test patches, run instrumented builds, and provide full journals or devcoredumps - the machine reproduces this every couple of days under normal desktop use.

Does your tree have these fixes:
https://lore.kernel.org/lkml/06489359-1b96-43ad-a065-bfb9bee51a10@linaro.org/t/#u

Also, could you try limiting the Fmax to < 1Ghz?

-Akhil.

> 
> Thanks,
> Greg Ociepka

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Greg Ociepka]

On Fri, 4 Sep 2026 04:29:16 +0530, Akhil P Oommen wrote:
> On 8/20/2026 7:14 PM, Greg Ociepka wrote:
[...]
>> We are happy to test patches, run instrumented builds, and provide
>> full journals or devcoredumps - the machine reproduces this every
>> couple of days under normal desktop use.
>
> Does your tree have these fixes:
> https://lore.kernel.org/lkml/06489359-1b96-43ad-a065-bfb9bee51a10@linaro.org/t/#u

Yes, all six of them. I checked each patch of that series against
linux-next (cgit):

  d9108bfdb746 drm/msm/a6xx: Fix stale rpmh votes after suspend
  b303e1d52811 drm/msm: Recover HW before retire hung submit
  fc7ccbc6174b drm/msm/a6xx: Fix A663 GPUCC register list for state capture
  d052d0358fb8 drm/msm/a6xx: Fix A621 GPUCC register list for state capture
  bc024d325e98 drm/msm/a6xx: Fix IRQ storm during msm_recovery test
  40b793714ad8 drm/msm: Fix task_struct reference leak in recover_worker

All of them are already in next-20260730, which is the tree the report
(and the four incidents in it) came from, and they are in
next-20260831 as well, which this machine has been running since the
evening of Sep 3 (7.3.0-rc1-next-20260831, with the same local
platform patches as described in the report, still nothing anywhere
near the GPU, GMU, GDSC or clock paths, plus one unrelated
fs/buffer.c fix).

So the loop reproduced with these fixes in place.

For completeness, the timeline since the report:

- After the report I started running thunderbird, the most frequent
  "offending task" of the earlier incidents, with
  LIBGL_ALWAYS_SOFTWARE=1 (llvmpipe) as a user-space mitigation.
- The last incident on the next-20260730 tree was on Aug 30, 12:10
  CEST: 689 "hangcheck recover" cycles in one boot, with a different
  client as the trigger, so the mitigation did not prevent it.
- Since the switch to next-20260831 on the evening of Sep 3 there has
  been no incident, but against a historical rate of one every few
  days that short a window does not mean anything yet.

> Also, could you try limiting the Fmax to < 1Ghz?

Yes. The OPP table as the kernel sees it here (available_frequencies)
is 310, 410, 572, 760, 820, 915, 1070, 1185, 1350, 1550, 1700 and
1850 MHz, and max_freq is at 1850000000 as I write this, i.e. the
full range. I will cap it at 915 MHz, the highest step below 1 GHz,
through devfreq at runtime:

  echo 915000000 > /sys/class/devfreq/3d00000.gpu/max_freq

and keep the machine at that setting under normal use. It is a sysfs
setting, so there is an uncapped window after every boot until it is
re-applied; I will log that window per boot and check any incident
against it, since onset correlated with session restores in the
earlier incidents. Given the historical rate I would like to give it
at least two weeks before concluding anything, and I will report
either way: at the first incident under the cap (with the journal and
trans_stat, reset when the cap went on, so you can see devfreq never
asked for anything above it), or after the observation window if it
stays clean. One caveat: this stacks on top of the kernel bump, so a
clean run would not tell the two apart on its own; if it stays clean
I can go back to the full range on next-20260831 to separate them.

On Thu, 3 Sep 2026 13:16:47 +0200, Konrad Dybcio wrote:
> On 8/21/26 3:35 PM, Rob Clark wrote:
[...]
>> Fwiw, the kernel I've been using: [1]..  but that doesn't have the a16
>> dts. I need to move forward to something newer, hopefully I'll get to
>> that today.  We'll see if I start seeing this issue.  Or maybe
>> something x2e vs x2ee is different regarding pmic/supplies?
>>
>> My device doesn't have the highest OPP (1.85ghz), which I think your
>> device does.  We might be still missing some power related features
>> that are (more?) needed at higher freq's, or maybe I have a more lucky
>> piece of silicon?  Akhil would know more about this.  You might try
>> removing the highest OPP in gpu_opp_table (in glymur.dtsi) to see if
>> that improves stability?  (Repeat until it is stable if you have the
>> same issues w/ 1.7ghz.)
>
> Greg, have you tried this?

Not yet, and sorry for the late answer on this. We have not run the
machine with the top OPP(s) removed from the DT for any meaningful
period, so there is no data point from that side; what we did instead
is the mitigation and the kernel bump described above. Rob's guess is
right: this is the X2 Elite Extreme part (UX3607OA), it does have the
1.85 GHz OPP, and it is currently running with the full range
(max_freq 1850000000, see above). Mesa here is still 26.1.5, the same
as in the report, so it has not been a variable in any of this.

Rather than removing OPPs from the DT one at a time, I will start with
Akhil's Fmax < 1 GHz test via the runtime cap above, which is
reversible without a rebuild and is also the most direct test of Rob's
"power related features that are (more?) needed at higher freq's"
theory. It is not quite the same thing as dropping the OPP (the GMU
still gets the full table, the GPU just never gets asked for the top
levels), so if the capped run stays clean, walking the cap back up, or
dropping just the top OPP from the DT as Rob suggested, is the natural
next step to find where it starts failing.

On the x2e vs x2ee question: I have no insight into PMIC or supply
differences between the two parts. If there is anything worth dumping
from this machine on that front (regulator state, the level table the
GMU is given, ...), tell me what and I will collect it.

The offer stands: at the next incident I will grab the devcoredump of
the first crash state in the loop, before it expires, together with
the full journal of that boot, and I can run instrumented builds or
debug patches on top of next-20260831.

Thanks,
Greg Ociepka

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Rob Clark]

On Fri, Sep 4, 2026 at 11:54 PM Greg Ociepka <greg@ferrisoft.com> wrote:
>
> On Fri, 4 Sep 2026 04:29:16 +0530, Akhil P Oommen wrote:
> > On 8/20/2026 7:14 PM, Greg Ociepka wrote:
> [...]
> >> We are happy to test patches, run instrumented builds, and provide
> >> full journals or devcoredumps - the machine reproduces this every
> >> couple of days under normal desktop use.
> >
> > Does your tree have these fixes:
> > https://lore.kernel.org/lkml/06489359-1b96-43ad-a065-bfb9bee51a10@linaro.org/t/#u
>
> Yes, all six of them. I checked each patch of that series against
> linux-next (cgit):
>
>   d9108bfdb746 drm/msm/a6xx: Fix stale rpmh votes after suspend
>   b303e1d52811 drm/msm: Recover HW before retire hung submit
>   fc7ccbc6174b drm/msm/a6xx: Fix A663 GPUCC register list for state capture
>   d052d0358fb8 drm/msm/a6xx: Fix A621 GPUCC register list for state capture
>   bc024d325e98 drm/msm/a6xx: Fix IRQ storm during msm_recovery test
>   40b793714ad8 drm/msm: Fix task_struct reference leak in recover_worker
>
> All of them are already in next-20260730, which is the tree the report
> (and the four incidents in it) came from, and they are in
> next-20260831 as well, which this machine has been running since the
> evening of Sep 3 (7.3.0-rc1-next-20260831, with the same local
> platform patches as described in the report, still nothing anywhere
> near the GPU, GMU, GDSC or clock paths, plus one unrelated
> fs/buffer.c fix).
>
> So the loop reproduced with these fixes in place.
>
> For completeness, the timeline since the report:
>
> - After the report I started running thunderbird, the most frequent
>   "offending task" of the earlier incidents, with
>   LIBGL_ALWAYS_SOFTWARE=1 (llvmpipe) as a user-space mitigation.
> - The last incident on the next-20260730 tree was on Aug 30, 12:10
>   CEST: 689 "hangcheck recover" cycles in one boot, with a different
>   client as the trigger, so the mitigation did not prevent it.
> - Since the switch to next-20260831 on the evening of Sep 3 there has
>   been no incident, but against a historical rate of one every few
>   days that short a window does not mean anything yet.
>
> > Also, could you try limiting the Fmax to < 1Ghz?
>
> Yes. The OPP table as the kernel sees it here (available_frequencies)
> is 310, 410, 572, 760, 820, 915, 1070, 1185, 1350, 1550, 1700 and
> 1850 MHz, and max_freq is at 1850000000 as I write this, i.e. the
> full range. I will cap it at 915 MHz, the highest step below 1 GHz,
> through devfreq at runtime:
>
>   echo 915000000 > /sys/class/devfreq/3d00000.gpu/max_freq
>
> and keep the machine at that setting under normal use. It is a sysfs
> setting, so there is an uncapped window after every boot until it is
> re-applied; I will log that window per boot and check any incident
> against it, since onset correlated with session restores in the
> earlier incidents. Given the historical rate I would like to give it
> at least two weeks before concluding anything, and I will report
> either way: at the first incident under the cap (with the journal and
> trans_stat, reset when the cap went on, so you can see devfreq never
> asked for anything above it), or after the observation window if it
> stays clean. One caveat: this stacks on top of the kernel bump, so a
> clean run would not tell the two apart on its own; if it stays clean
> I can go back to the full range on next-20260831 to separate them.
>
> On Thu, 3 Sep 2026 13:16:47 +0200, Konrad Dybcio wrote:
> > On 8/21/26 3:35 PM, Rob Clark wrote:
> [...]
> >> Fwiw, the kernel I've been using: [1]..  but that doesn't have the a16
> >> dts. I need to move forward to something newer, hopefully I'll get to
> >> that today.  We'll see if I start seeing this issue.  Or maybe
> >> something x2e vs x2ee is different regarding pmic/supplies?
> >>
> >> My device doesn't have the highest OPP (1.85ghz), which I think your
> >> device does.  We might be still missing some power related features
> >> that are (more?) needed at higher freq's, or maybe I have a more lucky
> >> piece of silicon?  Akhil would know more about this.  You might try
> >> removing the highest OPP in gpu_opp_table (in glymur.dtsi) to see if
> >> that improves stability?  (Repeat until it is stable if you have the
> >> same issues w/ 1.7ghz.)
> >
> > Greg, have you tried this?
>
> Not yet, and sorry for the late answer on this. We have not run the
> machine with the top OPP(s) removed from the DT for any meaningful
> period, so there is no data point from that side; what we did instead
> is the mitigation and the kernel bump described above. Rob's guess is
> right: this is the X2 Elite Extreme part (UX3607OA), it does have the
> 1.85 GHz OPP, and it is currently running with the full range
> (max_freq 1850000000, see above). Mesa here is still 26.1.5, the same
> as in the report, so it has not been a variable in any of this.
>
> Rather than removing OPPs from the DT one at a time, I will start with
> Akhil's Fmax < 1 GHz test via the runtime cap above, which is
> reversible without a rebuild and is also the most direct test of Rob's
> "power related features that are (more?) needed at higher freq's"
> theory. It is not quite the same thing as dropping the OPP (the GMU
> still gets the full table, the GPU just never gets asked for the top
> levels), so if the capped run stays clean, walking the cap back up, or
> dropping just the top OPP from the DT as Rob suggested, is the natural
> next step to find where it starts failing.
>
> On the x2e vs x2ee question: I have no insight into PMIC or supply
> differences between the two parts. If there is anything worth dumping
> from this machine on that front (regulator state, the level table the
> GMU is given, ...), tell me what and I will collect it.
>
> The offer stands: at the next incident I will grab the devcoredump of
> the first crash state in the loop, before it expires, together with
> the full journal of that boot, and I can run instrumented builds or
> debug patches on top of next-20260831.

Devcoredump is, unfortunately, not fully wired up for gen8, so
unfortunately it won't be too useful.  But this does smell a bit power
related so I expect lower OPPs would be more stable.

BR,
-R

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Rob Clark]

On Fri, Sep 4, 2026 at 11:54 PM Greg Ociepka <greg@ferrisoft.com> wrote:
>
> On Fri, 4 Sep 2026 04:29:16 +0530, Akhil P Oommen wrote:
> > On 8/20/2026 7:14 PM, Greg Ociepka wrote:
> [...]
> >> We are happy to test patches, run instrumented builds, and provide
> >> full journals or devcoredumps - the machine reproduces this every
> >> couple of days under normal desktop use.
> >
> > Does your tree have these fixes:
> > https://lore.kernel.org/lkml/06489359-1b96-43ad-a065-bfb9bee51a10@linaro.org/t/#u
>
> Yes, all six of them. I checked each patch of that series against
> linux-next (cgit):
>
>   d9108bfdb746 drm/msm/a6xx: Fix stale rpmh votes after suspend
>   b303e1d52811 drm/msm: Recover HW before retire hung submit
>   fc7ccbc6174b drm/msm/a6xx: Fix A663 GPUCC register list for state capture
>   d052d0358fb8 drm/msm/a6xx: Fix A621 GPUCC register list for state capture
>   bc024d325e98 drm/msm/a6xx: Fix IRQ storm during msm_recovery test
>   40b793714ad8 drm/msm: Fix task_struct reference leak in recover_worker
>
> All of them are already in next-20260730, which is the tree the report
> (and the four incidents in it) came from, and they are in
> next-20260831 as well, which this machine has been running since the
> evening of Sep 3 (7.3.0-rc1-next-20260831, with the same local
> platform patches as described in the report, still nothing anywhere
> near the GPU, GMU, GDSC or clock paths, plus one unrelated
> fs/buffer.c fix).
>
> So the loop reproduced with these fixes in place.
>
> For completeness, the timeline since the report:
>
> - After the report I started running thunderbird, the most frequent
>   "offending task" of the earlier incidents, with
>   LIBGL_ALWAYS_SOFTWARE=1 (llvmpipe) as a user-space mitigation.
> - The last incident on the next-20260730 tree was on Aug 30, 12:10
>   CEST: 689 "hangcheck recover" cycles in one boot, with a different
>   client as the trigger, so the mitigation did not prevent it.
> - Since the switch to next-20260831 on the evening of Sep 3 there has
>   been no incident, but against a historical rate of one every few
>   days that short a window does not mean anything yet.
>
> > Also, could you try limiting the Fmax to < 1Ghz?
>
> Yes. The OPP table as the kernel sees it here (available_frequencies)
> is 310, 410, 572, 760, 820, 915, 1070, 1185, 1350, 1550, 1700 and
> 1850 MHz, and max_freq is at 1850000000 as I write this, i.e. the
> full range. I will cap it at 915 MHz, the highest step below 1 GHz,
> through devfreq at runtime:
>
>   echo 915000000 > /sys/class/devfreq/3d00000.gpu/max_freq
>
> and keep the machine at that setting under normal use. It is a sysfs
> setting, so there is an uncapped window after every boot until it is
> re-applied; I will log that window per boot and check any incident
> against it, since onset correlated with session restores in the
> earlier incidents. Given the historical rate I would like to give it
> at least two weeks before concluding anything, and I will report
> either way: at the first incident under the cap (with the journal and
> trans_stat, reset when the cap went on, so you can see devfreq never
> asked for anything above it), or after the observation window if it
> stays clean. One caveat: this stacks on top of the kernel bump, so a
> clean run would not tell the two apart on its own; if it stays clean
> I can go back to the full range on next-20260831 to separate them.
>
> On Thu, 3 Sep 2026 13:16:47 +0200, Konrad Dybcio wrote:
> > On 8/21/26 3:35 PM, Rob Clark wrote:
> [...]
> >> Fwiw, the kernel I've been using: [1]..  but that doesn't have the a16
> >> dts. I need to move forward to something newer, hopefully I'll get to
> >> that today.  We'll see if I start seeing this issue.  Or maybe
> >> something x2e vs x2ee is different regarding pmic/supplies?
> >>
> >> My device doesn't have the highest OPP (1.85ghz), which I think your
> >> device does.  We might be still missing some power related features
> >> that are (more?) needed at higher freq's, or maybe I have a more lucky
> >> piece of silicon?  Akhil would know more about this.  You might try
> >> removing the highest OPP in gpu_opp_table (in glymur.dtsi) to see if
> >> that improves stability?  (Repeat until it is stable if you have the
> >> same issues w/ 1.7ghz.)
> >
> > Greg, have you tried this?
>
> Not yet, and sorry for the late answer on this. We have not run the
> machine with the top OPP(s) removed from the DT for any meaningful
> period, so there is no data point from that side; what we did instead
> is the mitigation and the kernel bump described above. Rob's guess is
> right: this is the X2 Elite Extreme part (UX3607OA), it does have the
> 1.85 GHz OPP, and it is currently running with the full range
> (max_freq 1850000000, see above). Mesa here is still 26.1.5, the same
> as in the report, so it has not been a variable in any of this.
>
> Rather than removing OPPs from the DT one at a time, I will start with
> Akhil's Fmax < 1 GHz test via the runtime cap above, which is
> reversible without a rebuild and is also the most direct test of Rob's
> "power related features that are (more?) needed at higher freq's"
> theory. It is not quite the same thing as dropping the OPP (the GMU
> still gets the full table, the GPU just never gets asked for the top
> levels), so if the capped run stays clean, walking the cap back up, or
> dropping just the top OPP from the DT as Rob suggested, is the natural
> next step to find where it starts failing.
>
> On the x2e vs x2ee question: I have no insight into PMIC or supply
> differences between the two parts. If there is anything worth dumping
> from this machine on that front (regulator state, the level table the
> GMU is given, ...), tell me what and I will collect it.
>
> The offer stands: at the next incident I will grab the devcoredump of
> the first crash state in the loop, before it expires, together with
> the full journal of that boot, and I can run instrumented builds or
> debug patches on top of next-20260831.

Fwiw, I've started to see something similar, which seems to start with:

Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
[drm:a6xx_gmu_start [msm]] *ERROR* GMU firmware initialization timed
out
Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
[drm:a6xx_gmu_set_oob [msm]] *ERROR* Timeout waiting for GMU OOB set
GPU_SET: 0x8000
Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
[drm:a6xx_gmu_set_oob [msm]] *ERROR* Timeout waiting for GMU OOB set
GPU_SET: 0x8000
Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
[drm:a6xx_gmu_set_oob [msm]] *ERROR* Timeout waiting for GMU OOB set
GPU_SET: 0x8000
Sep 07 09:19:10 shady audit[236335]: AUDIT1123 pid=236335 uid=1000
auid=1000 ses=3
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
msg='cwd="/home/robclark/Downloads/deleteme/darktable-bench"
cmd="true" exe="/usr/bin/sudo" terminal=pts/3 res=failed'
Sep 07 09:19:10 shady sudo[236335]: robclark : a password is required
; TTY=pts/3 ; PWD=/home/robclark/Downloads/deleteme/darktable-bench ;
USER=root ; COMMAND=/usr/sbin/true
Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
[drm:hangcheck_handler [msm]] *ERROR* 44070041: hangcheck detected gpu
lockup rb 0!
Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
[drm:hangcheck_handler [msm]] *ERROR* 44070041:     completed fence:
581164
Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
[drm:hangcheck_handler [msm]] *ERROR* 44070041:     submitted fence:
581167
Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
[drm:recover_worker [msm]] *ERROR* 44070041: hangcheck recover!
Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
[drm:recover_worker [msm]] *ERROR* 44070041: offending task:
gnome-shell (/usr/bin/gnome-shell --mode=user)

In particular, "GMU firmware initialization timed out"

I _think_ the main change on my end is switching to release kernel
config (from one that had lockdep and other debug stuff enabled),
since I was doing some benchmarking.

BR,
-R

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Konrad Dybcio]

On 9/7/26 6:33 PM, Rob Clark wrote:
> On Fri, Sep 4, 2026 at 11:54 PM Greg Ociepka <greg@ferrisoft.com> wrote:
>>
>> On Fri, 4 Sep 2026 04:29:16 +0530, Akhil P Oommen wrote:
>>> On 8/20/2026 7:14 PM, Greg Ociepka wrote:
>> [...]
>>>> We are happy to test patches, run instrumented builds, and provide
>>>> full journals or devcoredumps - the machine reproduces this every
>>>> couple of days under normal desktop use.

[...]

> Fwiw, I've started to see something similar, which seems to start with:
> 
> Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
> [drm:a6xx_gmu_start [msm]] *ERROR* GMU firmware initialization timed
> out
> Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
> [drm:a6xx_gmu_set_oob [msm]] *ERROR* Timeout waiting for GMU OOB set
> GPU_SET: 0x8000
> Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
> [drm:a6xx_gmu_set_oob [msm]] *ERROR* Timeout waiting for GMU OOB set
> GPU_SET: 0x8000
> Sep 07 09:19:07 shady kernel: platform 3d6c000.gmu:
> [drm:a6xx_gmu_set_oob [msm]] *ERROR* Timeout waiting for GMU OOB set
> GPU_SET: 0x8000
> Sep 07 09:19:10 shady audit[236335]: AUDIT1123 pid=236335 uid=1000
> auid=1000 ses=3
> subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> msg='cwd="/home/robclark/Downloads/deleteme/darktable-bench"
> cmd="true" exe="/usr/bin/sudo" terminal=pts/3 res=failed'
> Sep 07 09:19:10 shady sudo[236335]: robclark : a password is required
> ; TTY=pts/3 ; PWD=/home/robclark/Downloads/deleteme/darktable-bench ;
> USER=root ; COMMAND=/usr/sbin/true
> Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
> [drm:hangcheck_handler [msm]] *ERROR* 44070041: hangcheck detected gpu
> lockup rb 0!
> Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
> [drm:hangcheck_handler [msm]] *ERROR* 44070041:     completed fence:
> 581164
> Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
> [drm:hangcheck_handler [msm]] *ERROR* 44070041:     submitted fence:
> 581167
> Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
> [drm:recover_worker [msm]] *ERROR* 44070041: hangcheck recover!
> Sep 07 09:19:12 shady kernel: msm_dpu ae01000.display-controller:
> [drm:recover_worker [msm]] *ERROR* 44070041: offending task:
> gnome-shell (/usr/bin/gnome-shell --mode=user)
> 
> In particular, "GMU firmware initialization timed out"

I'm surprised the GPU accepts submissions if that happens, it definitely
shouldn't given it's clear it's not going to be powered up

Konrad

Re: drm/msm: unrecoverable a8xx hang-recovery loop on Glymur ("cx gdsc didn't collapse"); killing the offending client does not stop the loop

[Rob Clark]

On Fri, Sep 4, 2026 at 11:54 PM Greg Ociepka <greg@ferrisoft.com> wrote:
>
> On Fri, 4 Sep 2026 04:29:16 +0530, Akhil P Oommen wrote:
> > On 8/20/2026 7:14 PM, Greg Ociepka wrote:
> [...]
> >> We are happy to test patches, run instrumented builds, and provide
> >> full journals or devcoredumps - the machine reproduces this every
> >> couple of days under normal desktop use.
> >
> > Does your tree have these fixes:
> > https://lore.kernel.org/lkml/06489359-1b96-43ad-a065-bfb9bee51a10@linaro.org/t/#u
>
> Yes, all six of them. I checked each patch of that series against
> linux-next (cgit):
>
>   d9108bfdb746 drm/msm/a6xx: Fix stale rpmh votes after suspend
>   b303e1d52811 drm/msm: Recover HW before retire hung submit
>   fc7ccbc6174b drm/msm/a6xx: Fix A663 GPUCC register list for state capture
>   d052d0358fb8 drm/msm/a6xx: Fix A621 GPUCC register list for state capture
>   bc024d325e98 drm/msm/a6xx: Fix IRQ storm during msm_recovery test
>   40b793714ad8 drm/msm: Fix task_struct reference leak in recover_worker
>
> All of them are already in next-20260730, which is the tree the report
> (and the four incidents in it) came from, and they are in
> next-20260831 as well, which this machine has been running since the
> evening of Sep 3 (7.3.0-rc1-next-20260831, with the same local
> platform patches as described in the report, still nothing anywhere
> near the GPU, GMU, GDSC or clock paths, plus one unrelated
> fs/buffer.c fix).
>
> So the loop reproduced with these fixes in place.
>
> For completeness, the timeline since the report:
>
> - After the report I started running thunderbird, the most frequent
>   "offending task" of the earlier incidents, with
>   LIBGL_ALWAYS_SOFTWARE=1 (llvmpipe) as a user-space mitigation.
> - The last incident on the next-20260730 tree was on Aug 30, 12:10
>   CEST: 689 "hangcheck recover" cycles in one boot, with a different
>   client as the trigger, so the mitigation did not prevent it.
> - Since the switch to next-20260831 on the evening of Sep 3 there has
>   been no incident, but against a historical rate of one every few
>   days that short a window does not mean anything yet.
>
> > Also, could you try limiting the Fmax to < 1Ghz?
>
> Yes. The OPP table as the kernel sees it here (available_frequencies)
> is 310, 410, 572, 760, 820, 915, 1070, 1185, 1350, 1550, 1700 and
> 1850 MHz, and max_freq is at 1850000000 as I write this, i.e. the
> full range. I will cap it at 915 MHz, the highest step below 1 GHz,
> through devfreq at runtime:
>
>   echo 915000000 > /sys/class/devfreq/3d00000.gpu/max_freq
>
> and keep the machine at that setting under normal use. It is a sysfs
> setting, so there is an uncapped window after every boot until it is
> re-applied; I will log that window per boot and check any incident
> against it, since onset correlated with session restores in the
> earlier incidents. Given the historical rate I would like to give it
> at least two weeks before concluding anything, and I will report
> either way: at the first incident under the cap (with the journal and
> trans_stat, reset when the cap went on, so you can see devfreq never
> asked for anything above it), or after the observation window if it
> stays clean. One caveat: this stacks on top of the kernel bump, so a
> clean run would not tell the two apart on its own; if it stays clean
> I can go back to the full range on next-20260831 to separate them.
>
> On Thu, 3 Sep 2026 13:16:47 +0200, Konrad Dybcio wrote:
> > On 8/21/26 3:35 PM, Rob Clark wrote:
> [...]
> >> Fwiw, the kernel I've been using: [1]..  but that doesn't have the a16
> >> dts. I need to move forward to something newer, hopefully I'll get to
> >> that today.  We'll see if I start seeing this issue.  Or maybe
> >> something x2e vs x2ee is different regarding pmic/supplies?
> >>
> >> My device doesn't have the highest OPP (1.85ghz), which I think your
> >> device does.  We might be still missing some power related features
> >> that are (more?) needed at higher freq's, or maybe I have a more lucky
> >> piece of silicon?  Akhil would know more about this.  You might try
> >> removing the highest OPP in gpu_opp_table (in glymur.dtsi) to see if
> >> that improves stability?  (Repeat until it is stable if you have the
> >> same issues w/ 1.7ghz.)
> >
> > Greg, have you tried this?
>
> Not yet, and sorry for the late answer on this. We have not run the
> machine with the top OPP(s) removed from the DT for any meaningful
> period, so there is no data point from that side; what we did instead
> is the mitigation and the kernel bump described above. Rob's guess is
> right: this is the X2 Elite Extreme part (UX3607OA), it does have the
> 1.85 GHz OPP, and it is currently running with the full range
> (max_freq 1850000000, see above). Mesa here is still 26.1.5, the same
> as in the report, so it has not been a variable in any of this.
>
> Rather than removing OPPs from the DT one at a time, I will start with
> Akhil's Fmax < 1 GHz test via the runtime cap above, which is
> reversible without a rebuild and is also the most direct test of Rob's
> "power related features that are (more?) needed at higher freq's"
> theory. It is not quite the same thing as dropping the OPP (the GMU
> still gets the full table, the GPU just never gets asked for the top
> levels), so if the capped run stays clean, walking the cap back up, or
> dropping just the top OPP from the DT as Rob suggested, is the natural
> next step to find where it starts failing.
>
> On the x2e vs x2ee question: I have no insight into PMIC or supply
> differences between the two parts. If there is anything worth dumping
> from this machine on that front (regulator state, the level table the
> GMU is given, ...), tell me what and I will collect it.
>
> The offer stands: at the next incident I will grab the devcoredump of
> the first crash state in the loop, before it expires, together with
> the full journal of that boot, and I can run instrumented builds or
> debug patches on top of next-20260831.

If the issue you hit starts w/ "GMU firmware initialization timed
out", like the issue I was hitting, the patch I just sent[1] (and CC'd
you on) increases the timeout to a safe value.  Could you please try
this patch?

BR,
-R

[1] https://lore.kernel.org/all/20260912150915.28700-1-robin.clark@oss.qualcomm.com/