Linux dom0 hangs on boot with Xen 4.22 and Heads firmware

Linux dom0 hangs on boot with Xen 4.22 and Heads firmware

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]

Hi,

After updating Xen to 4.22-rc2 on a system with Heads firmware, dom0
doesn't start anymore. It worked fine with Xen 4.19.
The last messages on console are:

    [    1.495140] installing Xen timer for CPU 2
    [    1.496149] installing Xen timer for CPU 4
    [    1.496587] installing Xen timer for CPU 5
    [    1.496809] installing Xen timer for CPU 7
    [    0.008235] [Firmware Bug]: CPU   2: APIC ID mismatch. CPUID: 0x0002 APIC: 0x0028
    [    0.008235] [Firmware Bug]: CPU   2: APIC ID mismatch. Firmware: 0x0011 APIC: 0x0028
    [    1.497055] cpu 2 spinlock event irq 200
    [    0.008235] [Firmware Bug]: CPU   4: APIC ID mismatch. CPUID: 0x0004 APIC: 0x0000
    [    0.008235] [Firmware Bug]: CPU   4: APIC ID mismatch. Firmware: 0x0019 APIC: 0x0000
    [    1.497074] cpu 4 spinlock event irq 201
    [    0.008235] [Firmware Bug]: CPU   5: APIC ID mismatch. CPUID: 0x0005 APIC: 0x0002
    [    0.008235] [Firmware Bug]: CPU   5: APIC ID mismatch. Firmware: 0x0021 APIC: 0x0002
    [    1.497074] cpu 5 spinlock event irq 202
    [    0.008235] [Firmware Bug]: CPU   7: APIC I

Full console log (containing both successful boot of Xen 4.19, and then
reboot into 4.22):
https://openqa.qubes-os.org/tests/184780/logfile?filename=serial0.txt

It doesn't reach loading graphics driver in dom0, so I don't have
anything interesting on VGA (the last output is about the kexec call
done by Heads). But at least I have a serial console.

This happens both on older Thinkpad t430 (Ivy Bridge), and newer Novacustom V54 (ADL).

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Linux dom0 hangs on boot with Xen 4.22 and Heads firmware

[Jan Beulich]

On 19.06.2026 16:38, Marek Marczykowski-Górecki wrote:
> After updating Xen to 4.22-rc2 on a system with Heads firmware, dom0
> doesn't start anymore. It worked fine with Xen 4.19.
> The last messages on console are:
> 
>     [    1.495140] installing Xen timer for CPU 2
>     [    1.496149] installing Xen timer for CPU 4
>     [    1.496587] installing Xen timer for CPU 5
>     [    1.496809] installing Xen timer for CPU 7
>     [    0.008235] [Firmware Bug]: CPU   2: APIC ID mismatch. CPUID: 0x0002 APIC: 0x0028
>     [    0.008235] [Firmware Bug]: CPU   2: APIC ID mismatch. Firmware: 0x0011 APIC: 0x0028
>     [    1.497055] cpu 2 spinlock event irq 200
>     [    0.008235] [Firmware Bug]: CPU   4: APIC ID mismatch. CPUID: 0x0004 APIC: 0x0000
>     [    0.008235] [Firmware Bug]: CPU   4: APIC ID mismatch. Firmware: 0x0019 APIC: 0x0000
>     [    1.497074] cpu 4 spinlock event irq 201
>     [    0.008235] [Firmware Bug]: CPU   5: APIC ID mismatch. CPUID: 0x0005 APIC: 0x0002
>     [    0.008235] [Firmware Bug]: CPU   5: APIC ID mismatch. Firmware: 0x0021 APIC: 0x0002
>     [    1.497074] cpu 5 spinlock event irq 202
>     [    0.008235] [Firmware Bug]: CPU   7: APIC I
> 
> Full console log (containing both successful boot of Xen 4.19, and then
> reboot into 4.22):
> https://openqa.qubes-os.org/tests/184780/logfile?filename=serial0.txt

The 4.19 log also has an anomaly around this point in time. Can you try
again with sync_console added to both the 4.19 and the 4.22 attempt?

> It doesn't reach loading graphics driver in dom0, so I don't have
> anything interesting on VGA (the last output is about the kexec call
> done by Heads). But at least I have a serial console.

Yet interestingly Xen's "  VGA is ... mode ..." lines differ as well.
That may be a separate issue. Or maybe that's how one of the EFI changes
that were done a while ago actually surfaces?

Another difference is

(XEN) HWP: Using HWP for cpufreq

in the 4.19 log vs

(XEN) HPET: 8 timers usable for broadcast (8 total)

in the 4.22 one. Does the latter perhaps go away if you include [1] in
the build?

For the HWP driver do you perhaps have a patch in 4.19 to default-enable
it, but you don't have a similar patch in 4.22? If so, to compare apples
to apples the driver wants enabling/disabling consistently (via command
line option) for both runs.

And then "mwait-idle: lapic_timer_reliable_states ..." differ as well.

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2026-04/msg00200.html

Re: Linux dom0 hangs on boot with Xen 4.22 and Heads firmware

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 7270 bytes --]

On Mon, Jun 22, 2026 at 10:53:01AM +0200, Jan Beulich wrote:
> On 19.06.2026 16:38, Marek Marczykowski-Górecki wrote:
> > After updating Xen to 4.22-rc2 on a system with Heads firmware, dom0
> > doesn't start anymore. It worked fine with Xen 4.19.
> > The last messages on console are:
> > 
> >     [    1.495140] installing Xen timer for CPU 2
> >     [    1.496149] installing Xen timer for CPU 4
> >     [    1.496587] installing Xen timer for CPU 5
> >     [    1.496809] installing Xen timer for CPU 7
> >     [    0.008235] [Firmware Bug]: CPU   2: APIC ID mismatch. CPUID: 0x0002 APIC: 0x0028
> >     [    0.008235] [Firmware Bug]: CPU   2: APIC ID mismatch. Firmware: 0x0011 APIC: 0x0028
> >     [    1.497055] cpu 2 spinlock event irq 200
> >     [    0.008235] [Firmware Bug]: CPU   4: APIC ID mismatch. CPUID: 0x0004 APIC: 0x0000
> >     [    0.008235] [Firmware Bug]: CPU   4: APIC ID mismatch. Firmware: 0x0019 APIC: 0x0000
> >     [    1.497074] cpu 4 spinlock event irq 201
> >     [    0.008235] [Firmware Bug]: CPU   5: APIC ID mismatch. CPUID: 0x0005 APIC: 0x0002
> >     [    0.008235] [Firmware Bug]: CPU   5: APIC ID mismatch. Firmware: 0x0021 APIC: 0x0002
> >     [    1.497074] cpu 5 spinlock event irq 202
> >     [    0.008235] [Firmware Bug]: CPU   7: APIC I
> > 
> > Full console log (containing both successful boot of Xen 4.19, and then
> > reboot into 4.22):
> > https://openqa.qubes-os.org/tests/184780/logfile?filename=serial0.txt
> 
> The 4.19 log also has an anomaly around this point in time. Can you try
> again with sync_console added to both the 4.19 and the 4.22 attempt?

Yes, sync_console helped quite a bit, now I get full dom0 panic message:

    [   10.334800] vesafb: cannot reserve video memory at 0x0
    [   10.340009] vesafb: mode is 0x0x0, linelength=0, pages=0
    [   10.345515] Oops: divide error: 0000 [#1] SMP NOPTI
    [   10.346503] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.17.9-1.qubes.fc41.x86_64 #1 PREEMPT(full) 
    [   10.346503] Hardware name: Notebook V54x_6x_TU/V54x_6x_TU, BIOS Dasharo (coreboot+heads) v0.9.0 01/01/1970
    [   10.346503] RIP: e030:vesafb_probe.cold+0xd4/0x5fb
    [   10.346503] Code: 08 75 1f 83 3d a8 8c 1d 02 00 75 16 48 c7 c7 90 4a fd 81 e8 8a ef f9 ff c7 05 d4 54 09 02 05 00 00 00 8b 05 c2 54 09 02 31 d2 <f7> 35 d2 54 09 02 8b 15 fc 54 09 02 48 89 c1 48 c1 e1 20 48 09 ca
    [   10.346503] RSP: e02b:ffffc9004001fbb8 EFLAGS: 00010246
    [   10.346503] RAX: 0000000000000000 RBX: ffff888101d86f28 RCX: ffffffff823666e8
    [   10.346503] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
    [   10.346503] RBP: ffff88810197d400 R08: 0000000000000000 R09: 6c656e696c202c30
    [   10.346503] R10: 0000000000000030 R11: 203a626661736576 R12: 0000000000000000
    [   10.346503] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888106351000
    [   10.346503] FS:  0000000000000000(0000) GS:ffff888cef7c1000(0000) knlGS:0000000000000000
    [   10.346503] CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
    [   10.346503] CR2: ffffc900064ff000 CR3: 000000000222c000 CR4: 0000000000050660
    [   10.346503] Call Trace:
    [   10.346503]  <TASK>
    [   10.346503]  ? __pfx___device_attach_driver+0x10/0x10
    [   10.346503]  platform_probe+0x39/0x70
    [   10.346503]  really_probe+0xdb/0x340
    [   10.346503]  ? pm_runtime_barrier+0x54/0x90
    [   10.346503]  __driver_probe_device+0x78/0x110
    [   10.346503]  driver_probe_device+0x1f/0xa0
    [   10.346503]  __device_attach_driver+0x89/0x110
    [   10.346503]  bus_for_each_drv+0x94/0xf0
    [   10.346503]  __device_attach+0xaf/0x1b0
    [   10.346503]  bus_probe_device+0x8d/0xa0
    [   10.346503]  device_add+0x508/0x710
    [   10.346503]  platform_device_add+0xed/0x250
    [   10.346503]  sysfb_init+0x283/0x320
    [   10.346503]  ? __pfx_sysfb_init+0x10/0x10
    [   10.346503]  do_one_initcall+0x57/0x310
    [   10.346503]  do_initcalls+0x1ef/0x240
    [   10.346503]  kernel_init_freeable+0x187/0x210
    [   10.346503]  ? __pfx_kernel_init+0x10/0x10
    [   10.346503]  kernel_init+0x1a/0x140
    [   10.346503]  ret_from_fork+0xf2/0x110
    [   10.346503]  ? __pfx_kernel_init+0x10/0x10
    [   10.346503]  ret_from_fork_asm+0x1a/0x30
    [   10.346503]  </TASK>
    [   10.346503] Modules linked in:
    [   10.559786] ---[ end trace 0000000000000000 ]---
    [   10.564581] RIP: e030:vesafb_probe.cold+0xd4/0x5fb
    [   10.569546] Code: 08 75 1f 83 3d a8 8c 1d 02 00 75 16 48 c7 c7 90 4a fd 81 e8 8a ef f9 ff c7 05 d4 54 09 02 05 00 00 00 8b 05 c2 54 09 02 31 d2 <f7> 35 d2 54 09 02 8b 15 fc 54 09 02 48 89 c1 48 c1 e1 20 48 09 ca
    [   10.588833] RSP: e02b:ffffc9004001fbb8 EFLAGS: 00010246
    [   10.594255] RAX: 0000000000000000 RBX: ffff888101d86f28 RCX: ffffffff823666e8
    [   10.601622] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
    [   10.609000] RBP: ffff88810197d400 R08: 0000000000000000 R09: 6c656e696c202c30
    [   10.616378] R10: 0000000000000030 R11: 203a626661736576 R12: 0000000000000000
    [   10.623755] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888106351000
    [   10.631136] FS:  0000000000000000(0000) GS:ffff888cef7c1000(0000) knlGS:0000000000000000
    [   10.639483] CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
    [   10.645434] CR2: ffffc900064ff000 CR3: 000000000222c000 CR4: 0000000000050660
    [   10.652817] Kernel panic - not syncing: Fatal exception
    [   10.653803] Kernel Offset: disabled
    (XEN) Hardware Dom0 crashed: 'noreboot' set - not rebooting.

> 
> > It doesn't reach loading graphics driver in dom0, so I don't have
> > anything interesting on VGA (the last output is about the kexec call
> > done by Heads). But at least I have a serial console.
> 
> Yet interestingly Xen's "  VGA is ... mode ..." lines differ as well.

This might be relevant given the above.

> That may be a separate issue. Or maybe that's how one of the EFI changes
> that were done a while ago actually surfaces?

But this isn't booting via EFI... Heads is coreboot + Linux payload and
it uses kexec to boot Xen (via multiboot2 entry point).

> Another difference is
> 
> (XEN) HWP: Using HWP for cpufreq
> 
> in the 4.19 log vs
> 
> (XEN) HPET: 8 timers usable for broadcast (8 total)
> 
> in the 4.22 one. Does the latter perhaps go away if you include [1] in
> the build?

Yes

> 
> For the HWP driver do you perhaps have a patch in 4.19 to default-enable
> it, but you don't have a similar patch in 4.22? 

Hm, I do have that patch in 4.22 too, but maybe I failed at rebase...

> If so, to compare apples
> to apples the driver wants enabling/disabling consistently (via command
> line option) for both runs.

Since here booting just dom0 is enough, I don't need any of the qubes
patches. So, now I tested vanilla 4.19 vs staging - both console logs at:
https://gist.github.com/marmarek/e017f0a9ad3cc6ab2bac8849492c41a9

> And then "mwait-idle: lapic_timer_reliable_states ..." differ as well.
> 
> Jan
> 
> [1] https://lists.xen.org/archives/html/xen-devel/2026-04/msg00200.html

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]