Re: [RFC PATCH 0/3] drm/xe: diagnostics and workaround for GuC TLB invalidation ack stalls on ARL

[Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>]

On 8/6/2026 10:37 AM, Tales A. Mendonça wrote:
>> You've been approved in our CI for future revs.
> Thanks!
>
>> Also I came across this issue on i915 for MTL [1] which appears to
>> indicate the same issue (ARL and MTL are very close and share the same GuC
>> firmware), the suggestion is to disable rc6. Unfortunately Xe
>> doesn't have a knob for this, but per [1] you can turn off rc6
>> in the BIOS. Maybe worth trying.
> Good find - that does look like the same ~2.3s signature, and it being
> visible on i915/MTL too is an important data point (Daniele asked
> exactly that in the other subthread).

I am not sure if 14469 is the same issue. In that one it seems like the 
GuC just stops processing incoming messages (i.e., the H2G CTB has 
unprocessed data in it), while AFAIU in your case the GuC is still 
processing new commands and it is just being slow. It might be different 
manifestations of the same underlying issue, but it might also be 
completely separate bugs.

>
> On the rc6 angle: I ran an A/B experiment earlier that should be
> equivalent to disabling rc6 at the GT level - holding forcewake across
> the whole GT for hours (C6 residency pinned at 0ms for the entire
> window, verified) - and still hit 9 timeouts in a row, with the same
> ~2.3s request-to-ack. So at least keeping the GT out of RC6 does not
> avoid the stall here. If the BIOS suggestion covers more than GT RC6
> (e.g. package C-states), that would be a different experiment - my
> consumer ASUS BIOS does not expose an rc6 knob, but I can look for
> C-state options if you think it is worth isolating.

The fact that keeping the GT awake didn't help also indicates that this 
likely isn't the same 14469.

Daniele

>
> Next on my side: switching to the GuC 70.72.1 build Daniele posted and
> reporting back, plus GuC logs from severe stalls are now attached to
> the gitlab issue.
>
> Thanks,
> Tales
>
> Em qua., 5 de ago. de 2026 às 17:25, Matthew Brost
> <matthew.brost@intel.com> escreveu:
>> On Tue, Aug 04, 2026 at 02:29:27PM -0700, Matthew Brost wrote:
>>> On Tue, Aug 04, 2026 at 01:34:19PM -0300, Tales A. Mendonça wrote:
>>>> Patchwork reports my address is not on the CI allowlist, so CI was not
>>>> triggered for this series:
>>>>
>>>>      Series author address 'talesam@gmail.com' is not on the allowlist,
>>>> which prevents CI from being automatically triggered.
>>>>
>>>>    Could one of the project owners click 'retest' on the series (and/or
>>>> add me to the allowlist)? Series URL:
>>>>
>>>>      https://patchwork.freedesktop.org/series/171539/
>>>>
>>> We'd have to resend this ourselves. I can do this, but I've requested
>>> for you to be on our allow list as well. I'll ping here once that goes
>>> through.
>>>
>> You are approved on our CI for future patches.
>>
>> Also I came across this issue in the i915 for MTL [1] which seems to
>> indicate the same issue (ARL and MTL are very close and share same GuC
>> firmware), the suggestion there is turn off rc6. Unfortunately Xe
>> doesn't have a knob to do this but according to [1] you can turn off rc6
>> in the BIOS. Might be worth a try.
>>
>> Matt
>>
>> [1] https://gitlab.freedesktop.org/drm/i915/kernel/-/work_items/14469
>>
>>> Matt
>>>
>>>>    Thanks!
>>>>    Tales
>>>>
>>>>
>>>> Em seg., 3 de ago. de 2026 às 23:14, Tales A. Mendonça
>>>> <talesam@gmail.com> escreveu:
>>>>> Hi,
>>>>>
>>>>> This series is a follow-up to the TLB invalidation ack stall I have
>>>>> been debugging on ARL, tracked in:
>>>>>
>>>>>    https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8678
>>>>>
>>>>> Summary of the issue: with GuC 70.53.0 on ARL (reproduced on 7d51 and
>>>>> 7dd1 machines here, plus an independent Arc Pro 130T report on the
>>>>> issue above), TLB invalidation acks intermittently stall for ~2.3s.
>>>>> The H2G request is consumed from the CTB immediately and the G2H CTB
>>>>> is empty the whole time - the firmware simply does not send the ack
>>>>> until much later. The fence timeout fires at 2.25s and the ack lands
>>>>> tens of ms after it. Userspace blocked on the invalidation (compositor
>>>>> buffer unmaps etc.) hitches for the full window.
>>>>>
>>>>> Patch 1 adds xe_devcoredump_gt() so this kind of hang - which has no
>>>>> exec queue or job to blame - leaves a devcoredump with the GuC log and
>>>>> CT state behind (Matt suggested capturing devcoredumps when we
>>>>> discussed the issue; devcoredumps from both machines are attached to
>>>>> the issue above).
>>>>>
>>>>> Patch 2 logs when the ack for a timed out invalidation finally
>>>>> arrives. This is what established that the acks are late rather than
>>>>> lost.
>>>>>
>>>>> Patch 3 is the RFC part: a delayed work that pokes the GuC (status
>>>>> register read, CT flush, doorbell ring) every 250ms while an ack is
>>>>> overdue. On my machines this converts the guaranteed 2.3s stall into a
>>>>> sub-500ms hiccup for the majority of occurrences; a minority of severe
>>>>> episodes ignore 8-9 consecutive doorbells, which points at the GuC
>>>>> firmware being internally blocked for the whole window. Full data on
>>>>> the issue. I am happy to rework the approach (different delay,
>>>>> tying it to the G2H handler, dropping the status read, etc.) - mainly
>>>>> I would like the firmware side investigated, since no host-side poke
>>>>> can fix the severe cases.
>>>>>
>>>>> Based on drm-tip. Tested for several days on both ARL machines under
>>>>> desktop and VM-heavy workloads.
>>>>>
>>>>> Thanks,
>>>>> Tales
>>>>>
>>>>> Tales A. Mendonça (3):
>>>>>    drm/xe: Capture devcoredump on TLB invalidation timeout
>>>>>    drm/xe: Log when a timed out TLB invalidation ack finally arrives
>>>>>    drm/xe: Kick GuC while TLB invalidation acks are overdue
>>>>>
>>>>>   drivers/gpu/drm/xe/xe_devcoredump.c     |  68 ++++++++++++
>>>>>   drivers/gpu/drm/xe/xe_devcoredump.h     |   6 ++
>>>>>   drivers/gpu/drm/xe/xe_tlb_inval.c       | 131 +++++++++++++++++++++++-
>>>>>   drivers/gpu/drm/xe/xe_tlb_inval_types.h |  42 ++++++++
>>>>>   4 files changed, 243 insertions(+), 4 deletions(-)
>>>>>
>>>>> --
>>>>> 2.55.0
>>>>>
>>>>
>>>> --
>>>> Com os cumprimentos,
>>>>
>>>> Tales A. Mendonça
>>>> talesam.org
>>>> communitybig.org
>
>