Nasty console and xe driver interactions

Nasty console and xe driver interactions

[Dave Hansen]

I've got a custom kernel that has CONFIG_DRM_XE=y. I moved a disk from
an old NUC to a new Panther Lake one, and from it being SATA-attached to
USB-attached.

The new system was hanging after GRUB:

	Loading Linux ...
	Loading initial ramdisk ...
	<nothing>

Turns out, the root cause of the boot hang was my USB adapter needing a
rootdelay=15. But it took me a couple of hours, and a USB debug cable to
figure it out. Why?

Because the xe driver was falling over itself and presumably sabotaging
the firmware framebuffer in the process:

> [    1.505387] xe 0000:00:02.0: vgaarb: deactivate vga console
> [    1.507298] xe 0000:00:02.0: [drm] Running in SR-IOV PF mode
> [    1.512831] xe 0000:00:02.0: [drm] Tile0: GT0: GuC firmware xe/ptl_guc_70.bin not found
> [    1.512873] xe 0000:00:02.0: [drm] Tile0: GT0: GuC firmware(s) can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> [    1.512881] xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: Failed to initialize uC (-ENOENT)
> [    1.512894] xe 0000:00:02.0: probe with driver xe failed with error -2

So instead of getting a nice message about my root disk problems, I just
got a hang after GRUB. Basically, missing XE firmware results in a
system which is undebuggable for normal humans*.

There seem to have been similar issues in the past:

> https://lore.kernel.org/lkml/20251229215906.3688205-1-zack.rusin@broadcom.com/

Any chance that could be looked at again? This was a pretty insidious
failure mode.

 * Normal humans don't have USB debug cables lying around

Re: Nasty console and xe driver interactions

[Rodrigo Vivi]

On Wed, Oct 14, 2026 at 04:04:18PM -0700, Dave Hansen wrote:
> I've got a custom kernel that has CONFIG_DRM_XE=y. I moved a disk from
> an old NUC to a new Panther Lake one, and from it being SATA-attached to
> USB-attached.
> 
> The new system was hanging after GRUB:
> 
> 	Loading Linux ...
> 	Loading initial ramdisk ...
> 	<nothing>
> 
> Turns out, the root cause of the boot hang was my USB adapter needing a
> rootdelay=15. But it took me a couple of hours, and a USB debug cable to
> figure it out. Why?
> 
> Because the xe driver was falling over itself and presumably sabotaging
> the firmware framebuffer in the process:
> 
> > [    1.505387] xe 0000:00:02.0: vgaarb: deactivate vga console
> > [    1.507298] xe 0000:00:02.0: [drm] Running in SR-IOV PF mode
> > [    1.512831] xe 0000:00:02.0: [drm] Tile0: GT0: GuC firmware xe/ptl_guc_70.bin not found
> > [    1.512873] xe 0000:00:02.0: [drm] Tile0: GT0: GuC firmware(s) can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> > [    1.512881] xe 0000:00:02.0: [drm] *ERROR* Tile0: GT0: Failed to initialize uC (-ENOENT)
> > [    1.512894] xe 0000:00:02.0: probe with driver xe failed with error -2
> 
> So instead of getting a nice message about my root disk problems, I just
> got a hang after GRUB. Basically, missing XE firmware results in a
> system which is undebuggable for normal humans*.
> 
> There seem to have been similar issues in the past:
> 
> > https://lore.kernel.org/lkml/20251229215906.3688205-1-zack.rusin@broadcom.com/

Cc: dri-devel
Cc: Zack

This definitely looks the right overall solution to the failed probe.
So the VGA can be put back.

> 
> Any chance that could be looked at again? This was a pretty insidious
> failure mode.
> 
>  * Normal humans don't have USB debug cables lying around

Agree. and mandatory FW or initramfs/initrd flows also don't help.

So, perhaps what could be done in this specific case while we don't have
the proper one is something like trying to pre-fetch the mandatory fw.
Before we steal the fb. Something like:

(Untested warning)

https://github.com/rodrigovivi/linux/commit/5589d2883a604d52e89f44620e16d16c7fd278e8

Re: Nasty console and xe driver interactions

[Dave Hansen]

On 10/15/26 06:37, Rodrigo Vivi wrote:
> So, perhaps what could be done in this specific case while we don't have
> the proper one is something like trying to pre-fetch the mandatory fw.
> Before we steal the fb. Something like:
> 
> (Untested warning)
> 
> https://github.com/rodrigovivi/linux/commit/5589d2883a604d52e89f44620e16d16c7fd278e8

I threw caution to the wind and tried this. Id didn't seem to help. I
don't seem the messages about unregistering the old console, but I still
never get any output. The only "xe" gunk in dmesg is:

> [    0.739464] xe 0000:00:02.0: GuC firmware xe/ptl_guc_70.bin not found, can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> [    0.739501] xe 0000:00:02.0: probe with driver xe failed with error -2

BTW, "xe.force_probe=!*" on the command-line also gets the output back
to normal.

Re: Nasty console and xe driver interactions

[Rodrigo Vivi]

On Thu, Oct 15, 2026 at 08:30:42AM -0700, Dave Hansen wrote:
> On 10/15/26 06:37, Rodrigo Vivi wrote:
> > So, perhaps what could be done in this specific case while we don't have
> > the proper one is something like trying to pre-fetch the mandatory fw.
> > Before we steal the fb. Something like:
> > 
> > (Untested warning)
> > 
> > https://github.com/rodrigovivi/linux/commit/5589d2883a604d52e89f44620e16d16c7fd278e8
> 
> I threw caution to the wind and tried this. 

Thanks for being brave! :)

> Id didn't seem to help. I
> don't seem the messages about unregistering the old console, but I still
> never get any output. The only "xe" gunk in dmesg is:
> 
> > [    0.739464] xe 0000:00:02.0: GuC firmware xe/ptl_guc_70.bin not found, can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> > [    0.739501] xe 0000:00:02.0: probe with driver xe failed with error -2
> 
> BTW, "xe.force_probe=!*" on the command-line also gets the output back
> to normal.

the force probe proves that we can still probe safely, but apparently
on my first attempt I missed the fact that pcim_enable_device will
touch the device and register a pci_disable_device that might
clear stuff and mess with the VGA fb...

Perhaps if we move just one step over:

https://github.com/rodrigovivi/linux/commit/616cd8c1b4d4414cd1f7ad56fa5d8abd36d7968d

Re: Nasty console and xe driver interactions

[Zack Rusin]

[-- Attachment #1: Type: text/plain, Size: 2290 bytes --]

On Thu, Oct 15, 2026 at 9:10 AM Rodrigo Vivi <rodrigo.vivi@intel.com> wrote:
>
> On Thu, Oct 15, 2026 at 08:30:42AM -0700, Dave Hansen wrote:
> > On 10/15/26 06:37, Rodrigo Vivi wrote:
> > > So, perhaps what could be done in this specific case while we don't have
> > > the proper one is something like trying to pre-fetch the mandatory fw.
> > > Before we steal the fb. Something like:
> > >
> > > (Untested warning)
> > >
> > > https://github.com/rodrigovivi/linux/commit/5589d2883a604d52e89f44620e16d16c7fd278e8
> >
> > I threw caution to the wind and tried this.
>
> Thanks for being brave! :)
>
> > Id didn't seem to help. I
> > don't seem the messages about unregistering the old console, but I still
> > never get any output. The only "xe" gunk in dmesg is:
> >
> > > [    0.739464] xe 0000:00:02.0: GuC firmware xe/ptl_guc_70.bin not found, can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
> > > [    0.739501] xe 0000:00:02.0: probe with driver xe failed with error -2
> >
> > BTW, "xe.force_probe=!*" on the command-line also gets the output back
> > to normal.
>
> the force probe proves that we can still probe safely, but apparently
> on my first attempt I missed the fact that pcim_enable_device will
> touch the device and register a pci_disable_device that might
> clear stuff and mess with the VGA fb...
>
> Perhaps if we move just one step over:
>
> https://github.com/rodrigovivi/linux/commit/616cd8c1b4d4414cd1f7ad56fa5d8abd36d7968d

The general issue with my series from the above was always that it
required two parts:
1) a generic framework to reload platform fb on module failure (which
i think that series does ok job at)
2) auditing and trying various drivers to make sure they leave the
system in a predictable state in which the reload from #1 can succeed.

Every driver (other than the vmw ones that I maintain) would still
need to go through the second step. The general consesus at the time
was that #2 was impossible for most drivers/gpus and having a generic
infra just to make vmwgfx/vmwvg init failures properly reload fb is
not something that'd be wanted. We don't have spare systems with intel
gpu's for me to play with it so I can't help with #2.

z

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5414 bytes --]