drm: xe: Kernel-submitted job timed out

drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

Actually, this doesn't seem to have actually timed out, it seems to
have never been started, and then subsequent operations were confused.

Because I had to reboot my desktop as non-responsive (the cursor was
moving, but no screen updates) after two lines of

  xe 0000:4b:00.0: [drm] Tile0: GT0: Check job timeout: seqno=4485322,
lrc_seqno=4485322, guc_id=0, not started

followed a few seconds later by some Xe fault and then an endless
stream of "Kernel-submitted job timed out" reports.

Presumably that job was the thing that was never started in the first place.

Cut-down dmesg with the endless repeats deleted (after rebooting to
get a working system) attached.

             Linus

[-- Attachment #2: out --]
[-- Type: application/octet-stream, Size: 5224 bytes --]


May 22 11:09:11 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Check job timeout: seqno=4485322, lrc_seqno=4485322, guc_id=0, not started
May 22 11:09:16 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Check job timeout: seqno=4485322, lrc_seqno=4485322, guc_id=0, not started

May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: 
                                      ASID: 0
                                      Faulted Address: 0x00000002fa9fa000
                                      FaultType: 0
                                      AccessType: 0
                                      FaultLevel: 2
                                      EngineClass: 3 bcs
                                      EngineInstance: 8
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -EINVAL
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Engine memory CAT error [18]: class=bcs, logical_mask: 0x2, guc_id=0
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x2, guc_id=0, state=0x249
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Timedout job: seqno=4485322, lrc_seqno=4485322, guc_id=0, flags=0x73 in no process [-1]
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Xe device coredump has been created
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data
May 22 11:09:19 3970x kernel: ------------[ cut here ]------------
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Kernel-submitted job timed out
May 22 11:09:19 3970x kernel: WARNING: drivers/gpu/drm/xe/xe_guc_submit.c:1627 at guc_exec_queue_timedout_job+0xe29/0x1000 [xe], CPU#17: kworker/u256:0/2306935
May 22 11:09:19 3970x kernel: Modules linked in: uas usb_storage uinput rfcomm nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables sunrpc bnep vfat fat iwlmvm mac80211 libarc4 snd_hda_codec_intelhdmi snd_hda_codec_hdmi snd_hda_intel snd_intel_dspcfg snd_hda_codec snd_usb_audio snd_hda_core btusb snd_hwdep btrtl amd_atl snd_usbmidi_lib iwlwifi snd_seq btintel amd64_edac snd_rawmidi btbcm bluetooth snd_pcm edac_mce_amd snd_seq_device wmi_bmof atlantic cfg80211 pcspkr igb mc macsec dca snd_timer rfkill mxm_wmi snd i2c_piix4 soundcore i2c_smbus k10temp joydev nfnetlink zram dm_crypt xe drm_ttm_helper ttm i2c_algo_bit gpu_sched drm_buddy video drm_client_lib drm_suballoc_helper drm_gpuvm drm_exec drm_gpusvm_helper drm_display_helper drm_kms_helper ccp drm cec nvme sp5100_tco nvme_core wmi i2c_dev fuse
May 22 11:09:19 3970x kernel: CPU: 17 UID: 0 PID: 2306935 Comm: kworker/u256:0 Not tainted 7.1.0-rc3-00073-ga6920214ba75 #46 PREEMPTLAZY 
May 22 11:09:19 3970x kernel: Hardware name: Gigabyte Technology Co., Ltd. TRX40 AORUS MASTER/TRX40 AORUS MASTER, BIOS F7 09/07/2022
May 22 11:09:19 3970x kernel: Workqueue: gt-ordered-wq drm_sched_job_timedout [gpu_sched]
May 22 11:09:19 3970x kernel: RIP: 0010:guc_exec_queue_timedout_job+0xf3b/0x1000 [xe]
May 22 11:09:19 3970x kernel: Code: 8b 11 48 85 d2 74 06 48 8b 7a 08 eb 02 31 ff 48 8b 57 50 48 85 d2 75 03 48 8b 17 44 0f b6 46 26 0f b6 49 08 4c 89 f7 48 89 c6 <67> 48 0f b9 3a 48 8b 43 60 4c 8b 7c 24 20 44 8b 74 24 28 49 89 dc
May 22 11:09:19 3970x kernel: RSP: 0018:ffffd0e0aa6c7d88 EFLAGS: 00010246
May 22 11:09:19 3970x kernel: RAX: ffffffffc0c3b7cd RBX: ffff8a4020c46400 RCX: 0000000000000000
May 22 11:09:19 3970x kernel: RDX: ffff8a4004d369c0 RSI: ffffffffc0c3b7cd RDI: ffffffffc091cd30
May 22 11:09:19 3970x kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8a4f3f0fb240
May 22 11:09:19 3970x kernel: R10: 000000000000bffd R11: 3fffffffffffbfff R12: ffff8a4020c46400
May 22 11:09:19 3970x kernel: R13: ffff8a401c558e00 R14: ffffffffc091cd30 R15: ffff8a40166c0000
May 22 11:09:19 3970x kernel: FS:  0000000000000000(0000) GS:ffff8a4f4e2ed000(0000) knlGS:0000000000000000
May 22 11:09:19 3970x kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 22 11:09:19 3970x kernel: CR2: 0000243c08ff0000 CR3: 00000001994d5000 CR4: 0000000000350ef0
May 22 11:09:19 3970x kernel: Call Trace:
May 22 11:09:19 3970x kernel:  <TASK>
May 22 11:09:19 3970x kernel:  ? wake_bit_function+0x60/0x60
May 22 11:09:19 3970x kernel:  drm_sched_job_timedout+0xb8/0x130 [gpu_sched]
May 22 11:09:19 3970x kernel:  process_scheduled_works+0x1ac/0x380
May 22 11:09:19 3970x kernel:  worker_thread+0x1f4/0x2d0
May 22 11:09:19 3970x kernel:  ? pr_cont_work+0x1b0/0x1b0
May 22 11:09:19 3970x kernel:  kthread+0xee/0x120
May 22 11:09:19 3970x kernel:  ? kthread_blkcg+0x30/0x30
May 22 11:09:19 3970x kernel:  ret_from_fork+0x9d/0x200
May 22 11:09:19 3970x kernel:  ? kthread_blkcg+0x30/0x30
May 22 11:09:19 3970x kernel:  ret_from_fork_asm+0x11/0x20
May 22 11:09:19 3970x kernel:  </TASK>
May 22 11:09:19 3970x kernel: ---[ end trace 0000000000000000 ]---
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Timedout job: seqno=4485325, lrc_seqno=4485325, guc_id=0, flags=0x73 in no process [-1]

Re: drm: xe: Kernel-submitted job timed out

[Maarten Lankhorst]

Hey,

Den 2026-05-22 kl. 20:52, skrev Linus Torvalds:
> Actually, this doesn't seem to have actually timed out, it seems to
> have never been started, and then subsequent operations were confused.
> 
> Because I had to reboot my desktop as non-responsive (the cursor was
> moving, but no screen updates) after two lines of
> 
>   xe 0000:4b:00.0: [drm] Tile0: GT0: Check job timeout: seqno=4485322,
> lrc_seqno=4485322, guc_id=0, not started
> 
> followed a few seconds later by some Xe fault and then an endless
> stream of "Kernel-submitted job timed out" reports.
> 
> Presumably that job was the thing that was never started in the first place.
> 
> Cut-down dmesg with the endless repeats deleted (after rebooting to
> get a working system) attached.
> 
>              Linus

There's a 
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Timedout job: seqno=4485322, lrc_seqno=4485322, guc_id=0, flags=0x73 in no process [-1]
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Xe device coredump has been created
May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data

Do you have this coredump too?

Kind regards,
~Maarten Lankhorst

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Fri, 22 May 2026 at 11:55, Maarten Lankhorst <dev@lankhorst.se> wrote:
>
> There's a
> May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Timedout job: seqno=4485322, lrc_seqno=4485322, guc_id=0, flags=0x73 in no process [-1]
> May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Xe device coredump has been created
> May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data
>
> Do you have this coredump too?

Nope. I was assuming it didn't survive the reboot.

(This machine doesn't allow any remote logins - very much on purpose -
so when the GPU hangs, it's toast).

               Linus

Re: drm: xe: Kernel-submitted job timed out

[Rodrigo Vivi]

On Fri, May 22, 2026 at 12:05:35PM -0700, Linus Torvalds wrote:
> On Fri, 22 May 2026 at 11:55, Maarten Lankhorst <dev@lankhorst.se> wrote:
> >
> > There's a
> > May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Tile0: GT0: Timedout job: seqno=4485322, lrc_seqno=4485322, guc_id=0, flags=0x73 in no process [-1]
> > May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Xe device coredump has been created
> > May 22 11:09:19 3970x kernel: xe 0000:4b:00.0: [drm] Check your /sys/class/drm/card0/device/devcoredump/data
> >
> > Do you have this coredump too?
> 
> Nope. I was assuming it didn't survive the reboot.

It doesn't. In this kind of setup the best way to deal with devcoredump
is to create a udev rule that copies the data file to a persistent place.

> 
> (This machine doesn't allow any remote logins - very much on purpose -
> so when the GPU hangs, it's toast).

Any journal saving the kernel buf log of previous boots? Preferably with
some drm.debug flags enabled 0xf likely

Also:

Any bisect possible in this setup? I imagine it might be painful though...

What was the last drm-fixes pull you got in this 7.1.0-rc3-00073-ga6920214ba75 ?

I believe the quickest path might be to simply drop the xe fixes you might
have recently gotten there while we don't identify the culprit.

Thanks,
Rodrigo.

> 
>                Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Fri, 22 May 2026 at 13:44, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
>
> Any journal saving the kernel buf log of previous boots? Preferably with
> some drm.debug flags enabled 0xf likely

Note that this is very much not repeatable. I have no idea what
triggered it, and I don't think it was necessarily brought on by
anything recent.

I've seen timeouts before, but looking at my logs, the last time it
caused a complete hang was Feb 3. So a few months ago...

> What was the last drm-fixes pull you got in this 7.1.0-rc3-00073-ga6920214ba75 ?

That's just mainline commit v7.1-rc3-71-g31e62c2ebbfd with two random
small patches on top that change some build flags (this is my "built
by clang" tree)

So the last drm merge would have been 51d24842acb9 Merge tag
'drm-fixes-2026-05-08-1' of https://gitlab.freedesktop.org/drm/kernel

                    Linus

Re: drm: xe: Kernel-submitted job timed out

[Maarten Lankhorst]

Hey,

Den 2026-05-22 kl. 22:54, skrev Linus Torvalds:
> On Fri, 22 May 2026 at 13:44, Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
>>
>> Any journal saving the kernel buf log of previous boots? Preferably with
>> some drm.debug flags enabled 0xf likely
> 
> Note that this is very much not repeatable. I have no idea what
> triggered it, and I don't think it was necessarily brought on by
> anything recent.
> 
> I've seen timeouts before, but looking at my logs, the last time it
> caused a complete hang was Feb 3. So a few months ago...
> 
>> What was the last drm-fixes pull you got in this 7.1.0-rc3-00073-ga6920214ba75 ?
> 
> That's just mainline commit v7.1-rc3-71-g31e62c2ebbfd with two random
> small patches on top that change some build flags (this is my "built
> by clang" tree)
> 
> So the last drm merge would have been 51d24842acb9 Merge tag
> 'drm-fixes-2026-05-08-1' of https://gitlab.freedesktop.org/drm/kernel
> 
>                     Linus

Just thinking that since the guc_id=0, the most likely culprit is in the
kernel migration code.

There are 3 places you'll most likely interact with it:
- Zeroing VRAM bo's on allocation
- On integrated, it may clear system memory bo's CCS data.
- Moving memory between system and VRAM.

I'm assuming you only have a discrete card, so it's either happening
on allocation or memory movement.

Since it's sporadic, it *might* be more likely the latter.

Does it happen more frequently when loading VRAM intensive programs?

Kind regards,
~Maarten Lankhorst

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Sat, May 23, 2026 at 1:29 AM Maarten Lankhorst <dev@lankhorst.se> wrote:
>
> Does it happen more frequently when loading VRAM intensive programs?

Well, "more frequently" is hard to say since it's happened twice, but
this time it certainly happened when launching a new program.

This time it was a markdown viewer.

I wouldn't expect that to be particularly VRAM-intensive, but hey,
since I run with two 6k monitors, I suspect *anything* with big
windows will chew up a few hundred megs of VRAM just for the frame
buffer side.

It's a B50 Pro, so it's a discrete card with 16GB on card.

I have no memory of what it might have been back a few months ago. But
I would expect it to be all the usual stuff - ten terminals, a web
browser with a dozen tabs, and whatever gnome and wayland do, and then
the occasional random other thing.

             Linus

Re: drm: xe: Kernel-submitted job timed out

[Matthew Brost]

On Sat, May 23, 2026 at 07:48:49AM -0700, Linus Torvalds wrote:
> On Sat, May 23, 2026 at 1:29 AM Maarten Lankhorst <dev@lankhorst.se> wrote:
> >
> > Does it happen more frequently when loading VRAM intensive programs?
> 
> Well, "more frequently" is hard to say since it's happened twice, but
> this time it certainly happened when launching a new program.
> 
> This time it was a markdown viewer.
> 
> I wouldn't expect that to be particularly VRAM-intensive, but hey,
> since I run with two 6k monitors, I suspect *anything* with big
> windows will chew up a few hundred megs of VRAM just for the frame
> buffer side.
> 
> It's a B50 Pro, so it's a discrete card with 16GB on card.
> 
> I have no memory of what it might have been back a few months ago. But
> I would expect it to be all the usual stuff - ten terminals, a web
> browser with a dozen tabs, and whatever gnome and wayland do, and then
> the occasional random other thing.
> 
>              Linus

I’ve also intermittently seen kernel job timeouts during my development
over the last several months. It truly seems random—on some Linux builds
it happens somewhat frequently when running internal tests, while on
others it disappears, only to show up again in a different build.

I’ve also seen cases where a kernel timed-out job loops indefinitely,
though I haven’t investigated fixing that part. However, Rodrigo just
posted a series that should at least address that issue, allowing us to
focus on root-causing why kernel jobs are timing out in the first place.

Matt

Re: drm: xe: Kernel-submitted job timed out

[Rodrigo Vivi]

On Tue, Jun 09, 2026 at 09:30:45AM -0700, Matthew Brost wrote:
> On Sat, May 23, 2026 at 07:48:49AM -0700, Linus Torvalds wrote:
> > On Sat, May 23, 2026 at 1:29 AM Maarten Lankhorst <dev@lankhorst.se> wrote:
> > >
> > > Does it happen more frequently when loading VRAM intensive programs?
> > 
> > Well, "more frequently" is hard to say since it's happened twice, but
> > this time it certainly happened when launching a new program.
> > 
> > This time it was a markdown viewer.
> > 
> > I wouldn't expect that to be particularly VRAM-intensive, but hey,
> > since I run with two 6k monitors, I suspect *anything* with big
> > windows will chew up a few hundred megs of VRAM just for the frame
> > buffer side.
> > 
> > It's a B50 Pro, so it's a discrete card with 16GB on card.
> > 
> > I have no memory of what it might have been back a few months ago. But
> > I would expect it to be all the usual stuff - ten terminals, a web
> > browser with a dozen tabs, and whatever gnome and wayland do, and then
> > the occasional random other thing.
> > 
> >              Linus
> 
> I’ve also intermittently seen kernel job timeouts during my development
> over the last several months. It truly seems random—on some Linux builds
> it happens somewhat frequently when running internal tests, while on
> others it disappears, only to show up again in a different build.
> 
> I’ve also seen cases where a kernel timed-out job loops indefinitely,
> though I haven’t investigated fixing that part. However, Rodrigo just
> posted a series that should at least address that issue, allowing us to
> focus on root-causing why kernel jobs are timing out in the first place.

Yeap, commit ("drm/xe: fix job timeout recovery for unstarted jobs and kernel queues") [1]
merged on drm-xe-next and on the way for this week's fixes PR won't solve what
caused the initial GPU hang, but it should make the reset more robust and
avoid getting the machine frozen/lock-up. So, the next time you got the hang
you could continue using the machine and getting the devcoredump so we can
debug the hang itself.

[1] https://lore.kernel.org/all/20260610152548.404575-3-rodrigo.vivi@intel.com/

> 
> Matt

Re: drm: xe: Kernel-submitted job timed out

[Matthew Brost]

On Sat, May 23, 2026 at 07:48:49AM -0700, Linus Torvalds wrote:
> On Sat, May 23, 2026 at 1:29 AM Maarten Lankhorst <dev@lankhorst.se> wrote:
> >
> > Does it happen more frequently when loading VRAM intensive programs?
> 
> Well, "more frequently" is hard to say since it's happened twice, but
> this time it certainly happened when launching a new program.
> 
> This time it was a markdown viewer.
> 
> I wouldn't expect that to be particularly VRAM-intensive, but hey,
> since I run with two 6k monitors, I suspect *anything* with big
> windows will chew up a few hundred megs of VRAM just for the frame
> buffer side.
> 
> It's a B50 Pro, so it's a discrete card with 16GB on card.
> 
> I have no memory of what it might have been back a few months ago. But
> I would expect it to be all the usual stuff - ten terminals, a web
> browser with a dozen tabs, and whatever gnome and wayland do, and then
> the occasional random other thing.
> 
>              Linus

I think we may have found one. I haven't been able to reproduce any
kernel job timeouts on drm-tip with this patch applied [1].

If you're still seeing the issue on your end, it's probably worth
pulling it in.

Hopefully this fix will make it into your tree soon enough too.

Matt

[1] https://patchwork.freedesktop.org/series/171805/

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Fri, 7 Aug 2026 at 10:46, Matthew Brost <matthew.brost@intel.com> wrote:
>
> I think we may have found one. I haven't been able to reproduce any
> kernel job timeouts on drm-tip with this patch applied [1].
>
> If you're still seeing the issue on your end, it's probably worth
> pulling it in.

Just checked: I haven't seen any "job timeout" cases since May 26, so
it really seems to be a very rare occurrence for me.

It might have been that something else changed too, of course - some
user program behavior change could have simply made the trigger go
away. Not that I know what such a trigger might have been.

Looking at the logs around the last two cases, one was just after a
dnf update - maybe triggered by some restart swarm issue. The other
was fairly soon after kicad had a core dump. The kicad core dump may
have been a symptom of something going wrong, or the cause.

I dunno. Nothing looks obvious, but clearly this is not biting me very
often. Months in between events.

               Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Fri, 7 Aug 2026 at 10:46, Matthew Brost <matthew.brost@intel.com> wrote:
>
> I think we may have found one. I haven't been able to reproduce any
> kernel job timeouts on drm-tip with this patch applied [1].
>
> If you're still seeing the issue on your end, it's probably worth
> pulling it in.
>
> Hopefully this fix will make it into your tree soon enough too.

Well, that Xe fix made it into my tree as commit 9f83c94469ff
("drm/xe: Order ring writes before ring tail updates"), but it turns
out that now I can generate those drm job timedout issues at will.

[  +1.222075] xe 0000:4b:00.0: [drm] Tile0: GT0: Engine reset:
engine_class=rcs, logical_mask: 0x1, guc_id=12, state=0x3
[  +0.000041] xe 0000:4b:00.0: [drm] Tile0: GT0: Timedout job:
seqno=4294967171, lrc_seqno=4294967171, guc_id=12, flags=0x0 in
systemd-logind [2068]
[  +0.048338] xe 0000:4b:00.0: [drm] Xe device coredump has been created
[  +0.000001] xe 0000:4b:00.0: [drm] Check your
/sys/class/drm/card0/device/devcoredump/data

But at least this time I have ssh into the machine, so now I have that
devcoredump data file.

I will send that separately off-list, because  it's 2.4MB in size.

                Linus

Re: drm: xe: Kernel-submitted job timed out

[Matthew Brost]

On Tue, Aug 18, 2026 at 11:55:46PM -0700, Linus Torvalds wrote:
> On Fri, 7 Aug 2026 at 10:46, Matthew Brost <matthew.brost@intel.com> wrote:
> >
> > I think we may have found one. I haven't been able to reproduce any
> > kernel job timeouts on drm-tip with this patch applied [1].
> >
> > If you're still seeing the issue on your end, it's probably worth
> > pulling it in.
> >
> > Hopefully this fix will make it into your tree soon enough too.
> 
> Well, that Xe fix made it into my tree as commit 9f83c94469ff
> ("drm/xe: Order ring writes before ring tail updates"), but it turns
> out that now I can generate those drm job timedout issues at will.
> 

This is probably an unfortunate coincidence of timing and most likely
unrelated.

What is the kernel verison here? Is it trying to integrate all the PRs
for 7.3 into an RC release? Or is this the released 7.2?

If you have branch we can pull, that might be helpful.

> [  +1.222075] xe 0000:4b:00.0: [drm] Tile0: GT0: Engine reset:
> engine_class=rcs, logical_mask: 0x1, guc_id=12, state=0x3
> [  +0.000041] xe 0000:4b:00.0: [drm] Tile0: GT0: Timedout job:
> seqno=4294967171, lrc_seqno=4294967171, guc_id=12, flags=0x0 in
> systemd-logind [2068]
> [  +0.048338] xe 0000:4b:00.0: [drm] Xe device coredump has been created
> [  +0.000001] xe 0000:4b:00.0: [drm] Check your
> /sys/class/drm/card0/device/devcoredump/data


Did you change anything in userspace, or did you just update the kernel
version? This is a userspace job, which can generally hang if userspace
does something wrong.

Also, are you running a stock distro? It might also be helpful if we can
boot something similar.

> 
> But at least this time I have ssh into the machine, so now I have that
> devcoredump data file.
> 
> I will send that separately off-list, because  it's 2.4MB in size.
> 

I'm looking into it now, and based on the dmesg output, it seems to hang
fairly after boot, so something is going wrong very quickly. I'll
analyze the devcoredump and see if the problem is obvious.

Matt

>                 Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Wed, 19 Aug 2026 at 03:36, Matthew Brost <matthew.brost@intel.com> wrote:
>
> What is the kernel verison here? Is it trying to integrate all the PRs
> for 7.3 into an RC release? Or is this the released 7.2?

That was just tip-of-tree as of yesterday evening - it just has two
small commits on top that are due to how I do my local builds, which
clean up some generated assembly and switch to using clang by default
(because I'm too lazy to use the proper way).

So it was commit

  bd5f485f3f02 ("Merge tag 'soc-arm-7.3' of
git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc")

from this merge window, with no drm merges done yet.

No, the reason it stopped working was just user space doing something
new when I did an update. Plain Fedora 44.

> I'm looking into it now, and based on the dmesg output, it seems to hang
> fairly after boot, so something is going wrong very quickly. I'll
> analyze the devcoredump and see if the problem is obvious.

Yes, it hangs basically when logging in the first time. It shows the
login screen, I can type my password, but then it's done.

With the standard Fedora 44 kernel, it also hung, although at a
slightly different point.

               Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Wed, 19 Aug 2026 at 07:46, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> With the standard Fedora 44 kernel, it also hung, although at a
> slightly different point.

I take that back. It hangs at the same point.

I realize that it says

  GuC firmware: xe/bmg_guc_70.bin
  GuC version: 70.72.1 (wanted 70.54.0)

but that bmg_guc_70.bin is the newest version in F44:

    intel-gpu-firmware-20260810-1.fc44.noarch

which is just a few days old according to that name.

              Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Wed, 19 Aug 2026 at 08:23, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I take that back. It hangs at the same point.

Update: with a single 6k monitor and no rotation, it works.

In fact, it seems to work ok with a single 6k monitor and rotation
(orientation: "Portrait left").

So the problem is triggered by my insane setup.

But I'd really like it back, because I'm a little spoiled princess.

               Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Wed, 19 Aug 2026 at 10:25, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> So the problem is triggered by my insane setup.
>
> But I'd really like it back, because I'm a little spoiled princess.

Oh - it might be triggered by fractional scaling too. The thing that
is now working for me is the 100% unscaled mode, because with just a
single monitor I end up wanting all those pixels to do all the work
they can, instead of giving me nicer looking and slightly bigger text.

I think my old config that caused problems was using 150% scaling.

Did I mention that I'm spoiled?

               Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Wed, 19 Aug 2026 at 10:25, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> Update: with a single 6k monitor and no rotation, it works.
>
> In fact, it seems to work ok with a single 6k monitor and rotation
> (orientation: "Portrait left").

Ok, that was not true.

It's the rotation. I can change the rotation dynamically, but then
gnome - or gdm, or whatever - remembers it, and the next time I log in
it will hang at login.

So even a single-monitor setup breaks in the landscape left setting.

                      Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Tue, 18 Aug 2026 at 23:55, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> It turns out that now I can generate those drm job timedout issues at will.
>
> [  +1.222075] xe 0000:4b:00.0: [drm] Tile0: GT0: Engine reset:

Ok. I've spent all day today on chasing this down in between merge
window work, since it finally *was* repeatable.

And the fix ends up being basically a one-liner, even if the debug
session to get there was not.

The Xe driver takes the memory address used for the memory stolen for
CCS, and rounds it up to the nearest 128kB area.

And that is very VERY wrong, because it means that the xe driver will
then *use* that part of memory that wasn't at a 128kB boundary. And
the HW engine will too and write to it.

And when that memory happens ot be used for GPU page tables, very bad
things happen.

I bet this also explains some occasional random screen corruption I've
seen - when the memory isn't used for something as important as a page
table, it "only" corrupts random bitmap memory and the like.

I committed the fix as commit 818bebeb63dd ("drm/xe: Don't hand out
the flat CCS storage as usable VRAM").

Adding some people from commit 37173392741c ("drm/xe/vram: fix ccs
offset calculation") to the cc, because that's where the problem came
from. Two years ago. I'm not sure why it became so repeatable for me
now, but some user space behavior change clearly triggered it now
pretty much every single boot.

                 Linus

Re: drm: xe: Kernel-submitted job timed out

[Linus Torvalds]

On Thu, 20 Aug 2026 at 20:50, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> I committed the fix as commit 818bebeb63dd ("drm/xe: Don't hand out
> the flat CCS storage as usable VRAM").

Note: that is most definitely a real fix, but it's not necessarily the
*right* fix.

For example, I do not understand the old 128kB alignment at all. It
not only was definitely the wrong way around, but it made no sense in
the first place.

The comment for it said "SW must round up to nearest 128K" and the
original broken commit seems to imply that thatt comes from some
hardware document.

But the result isn't actually *used* for any hardware state, as far as
I can tell. It's just used for "vram_size" (called "usable_size" in
xe_vram_probe, which as far as I could tell is only used by the driver
for the allocator.

So the whole thing was just complete garbage, and I don't understand
*why*. I may be missing something, but yesterday I was so fed up with
the experience of spending 10+ hours debugging what ended up being a
one-liner that I just committed it as-is.

Now I'm back after a night's sleep, bright-eyed and bushy-tailed, and
still not really wanting to look at it, but at least wanting to tie
down some loose ends.

So I just want somebody who actually knows the Xe driver to look at
that thing. I only round down by 4k, not 128k. I think the 128k was
simply wrong and nonsensical. But that number came from *somewhere*.

I claim that the old 128kB number cannot be sane for another reason:
the other side of that

        if (GRAPHICS_VER(xe) >= 20) {

will return an offset that is at a 64k granularity, because that's
apparently the hardware field definition.

But the point is that the return value of get_flat_ccs_offset()
clearly is *not* 128kB-aligned and never was, so not only was the
alignment done in the wrong direction, it was always entirely
nonsensical. It cannot possibly be something that the allocator code
then depends on.

Anyway. It all now works for me, and apparently reliably. But the
keyword is "for me". I committed that thing because the old code was
clearly garbage, and the new code worked.

But the new code may be garbage too - for some reason that I simply
don't know about - so somebody competent needs to look at it.

                 Linus

Re: drm: xe: Kernel-submitted job timed out

[Matthew Brost]

On Fri, Aug 21, 2026 at 09:47:53AM -0700, Linus Torvalds wrote:
> On Thu, 20 Aug 2026 at 20:50, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
> >

First, thanks for looking into this, and I apologize for the
inconvenience. I had gotten to the point where, based on my testing, I
had reasoned that there was memory corruption somewhere in Xe that was
writing garbage into the page tables, but without a reproducer I
couldn't determine exactly where it was occurring.

> > I committed the fix as commit 818bebeb63dd ("drm/xe: Don't hand out
> > the flat CCS storage as usable VRAM").
> 
> Note: that is most definitely a real fix, but it's not necessarily the
> *right* fix.
> 
> For example, I do not understand the old 128kB alignment at all. It
> not only was definitely the wrong way around, but it made no sense in
> the first place.
> 
> The comment for it said "SW must round up to nearest 128K" and the
> original broken commit seems to imply that thatt comes from some
> hardware document.
> 
> But the result isn't actually *used* for any hardware state, as far as
> I can tell. It's just used for "vram_size" (called "usable_size" in
> xe_vram_probe, which as far as I could tell is only used by the driver
> for the allocator.
> 
> So the whole thing was just complete garbage, and I don't understand

Yes, this is garbage and can corrupt memory.

> *why*. I may be missing something, but yesterday I was so fed up with
> the experience of spending 10+ hours debugging what ended up being a
> one-liner that I just committed it as-is.
> 

one-liners tend to be hardest ones to find, again thanks, apologize,
etc...

> Now I'm back after a night's sleep, bright-eyed and bushy-tailed, and
> still not really wanting to look at it, but at least wanting to tie
> down some loose ends.
> 
> So I just want somebody who actually knows the Xe driver to look at
> that thing. I only round down by 4k, not 128k. I think the 128k was
> simply wrong and nonsensical. But that number came from *somewhere*.
> 
> I claim that the old 128kB number cannot be sane for another reason:
> the other side of that
> 
>         if (GRAPHICS_VER(xe) >= 20) {
> 
> will return an offset that is at a 64k granularity, because that's
> apparently the hardware field definition.
> 
> But the point is that the return value of get_flat_ccs_offset()
> clearly is *not* 128kB-aligned and never was, so not only was the
> alignment done in the wrong direction, it was always entirely
> nonsensical. It cannot possibly be something that the allocator code
> then depends on.
> 
> Anyway. It all now works for me, and apparently reliably. But the
> keyword is "for me". I committed that thing because the old code was
> clearly garbage, and the new code worked.
> 
> But the new code may be garbage too - for some reason that I simply

It will pop an assert somewhere else in the code, but that is also
questionable.

> don't know about - so somebody competent needs to look at it.
>

We are looking into this now and get a proper backportable fix post
asap.

Matt

>                  Linus