* Linux dom0 hangs on boot with Xen 4.22 and Heads firmware
@ 2026-06-19 14:38 Marek Marczykowski-Górecki
2026-06-22 8:53 ` Jan Beulich
0 siblings, 1 reply; 3+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-06-19 14:38 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 1643 bytes --]
Hi,
After updating Xen to 4.22-rc2 on a system with Heads firmware, dom0
doesn't start anymore. It worked fine with Xen 4.19.
The last messages on console are:
[ 1.495140] installing Xen timer for CPU 2
[ 1.496149] installing Xen timer for CPU 4
[ 1.496587] installing Xen timer for CPU 5
[ 1.496809] installing Xen timer for CPU 7
[ 0.008235] [Firmware Bug]: CPU 2: APIC ID mismatch. CPUID: 0x0002 APIC: 0x0028
[ 0.008235] [Firmware Bug]: CPU 2: APIC ID mismatch. Firmware: 0x0011 APIC: 0x0028
[ 1.497055] cpu 2 spinlock event irq 200
[ 0.008235] [Firmware Bug]: CPU 4: APIC ID mismatch. CPUID: 0x0004 APIC: 0x0000
[ 0.008235] [Firmware Bug]: CPU 4: APIC ID mismatch. Firmware: 0x0019 APIC: 0x0000
[ 1.497074] cpu 4 spinlock event irq 201
[ 0.008235] [Firmware Bug]: CPU 5: APIC ID mismatch. CPUID: 0x0005 APIC: 0x0002
[ 0.008235] [Firmware Bug]: CPU 5: APIC ID mismatch. Firmware: 0x0021 APIC: 0x0002
[ 1.497074] cpu 5 spinlock event irq 202
[ 0.008235] [Firmware Bug]: CPU 7: APIC I
Full console log (containing both successful boot of Xen 4.19, and then
reboot into 4.22):
https://openqa.qubes-os.org/tests/184780/logfile?filename=serial0.txt
It doesn't reach loading graphics driver in dom0, so I don't have
anything interesting on VGA (the last output is about the kexec call
done by Heads). But at least I have a serial console.
This happens both on older Thinkpad t430 (Ivy Bridge), and newer Novacustom V54 (ADL).
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: Linux dom0 hangs on boot with Xen 4.22 and Heads firmware
2026-06-19 14:38 Linux dom0 hangs on boot with Xen 4.22 and Heads firmware Marek Marczykowski-Górecki
@ 2026-06-22 8:53 ` Jan Beulich
2026-06-24 0:31 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 3+ messages in thread
From: Jan Beulich @ 2026-06-22 8:53 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel
On 19.06.2026 16:38, Marek Marczykowski-Górecki wrote:
> After updating Xen to 4.22-rc2 on a system with Heads firmware, dom0
> doesn't start anymore. It worked fine with Xen 4.19.
> The last messages on console are:
>
> [ 1.495140] installing Xen timer for CPU 2
> [ 1.496149] installing Xen timer for CPU 4
> [ 1.496587] installing Xen timer for CPU 5
> [ 1.496809] installing Xen timer for CPU 7
> [ 0.008235] [Firmware Bug]: CPU 2: APIC ID mismatch. CPUID: 0x0002 APIC: 0x0028
> [ 0.008235] [Firmware Bug]: CPU 2: APIC ID mismatch. Firmware: 0x0011 APIC: 0x0028
> [ 1.497055] cpu 2 spinlock event irq 200
> [ 0.008235] [Firmware Bug]: CPU 4: APIC ID mismatch. CPUID: 0x0004 APIC: 0x0000
> [ 0.008235] [Firmware Bug]: CPU 4: APIC ID mismatch. Firmware: 0x0019 APIC: 0x0000
> [ 1.497074] cpu 4 spinlock event irq 201
> [ 0.008235] [Firmware Bug]: CPU 5: APIC ID mismatch. CPUID: 0x0005 APIC: 0x0002
> [ 0.008235] [Firmware Bug]: CPU 5: APIC ID mismatch. Firmware: 0x0021 APIC: 0x0002
> [ 1.497074] cpu 5 spinlock event irq 202
> [ 0.008235] [Firmware Bug]: CPU 7: APIC I
>
> Full console log (containing both successful boot of Xen 4.19, and then
> reboot into 4.22):
> https://openqa.qubes-os.org/tests/184780/logfile?filename=serial0.txt
The 4.19 log also has an anomaly around this point in time. Can you try
again with sync_console added to both the 4.19 and the 4.22 attempt?
> It doesn't reach loading graphics driver in dom0, so I don't have
> anything interesting on VGA (the last output is about the kexec call
> done by Heads). But at least I have a serial console.
Yet interestingly Xen's " VGA is ... mode ..." lines differ as well.
That may be a separate issue. Or maybe that's how one of the EFI changes
that were done a while ago actually surfaces?
Another difference is
(XEN) HWP: Using HWP for cpufreq
in the 4.19 log vs
(XEN) HPET: 8 timers usable for broadcast (8 total)
in the 4.22 one. Does the latter perhaps go away if you include [1] in
the build?
For the HWP driver do you perhaps have a patch in 4.19 to default-enable
it, but you don't have a similar patch in 4.22? If so, to compare apples
to apples the driver wants enabling/disabling consistently (via command
line option) for both runs.
And then "mwait-idle: lapic_timer_reliable_states ..." differ as well.
Jan
[1] https://lists.xen.org/archives/html/xen-devel/2026-04/msg00200.html
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Linux dom0 hangs on boot with Xen 4.22 and Heads firmware
2026-06-22 8:53 ` Jan Beulich
@ 2026-06-24 0:31 ` Marek Marczykowski-Górecki
0 siblings, 0 replies; 3+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-06-24 0:31 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 7270 bytes --]
On Mon, Jun 22, 2026 at 10:53:01AM +0200, Jan Beulich wrote:
> On 19.06.2026 16:38, Marek Marczykowski-Górecki wrote:
> > After updating Xen to 4.22-rc2 on a system with Heads firmware, dom0
> > doesn't start anymore. It worked fine with Xen 4.19.
> > The last messages on console are:
> >
> > [ 1.495140] installing Xen timer for CPU 2
> > [ 1.496149] installing Xen timer for CPU 4
> > [ 1.496587] installing Xen timer for CPU 5
> > [ 1.496809] installing Xen timer for CPU 7
> > [ 0.008235] [Firmware Bug]: CPU 2: APIC ID mismatch. CPUID: 0x0002 APIC: 0x0028
> > [ 0.008235] [Firmware Bug]: CPU 2: APIC ID mismatch. Firmware: 0x0011 APIC: 0x0028
> > [ 1.497055] cpu 2 spinlock event irq 200
> > [ 0.008235] [Firmware Bug]: CPU 4: APIC ID mismatch. CPUID: 0x0004 APIC: 0x0000
> > [ 0.008235] [Firmware Bug]: CPU 4: APIC ID mismatch. Firmware: 0x0019 APIC: 0x0000
> > [ 1.497074] cpu 4 spinlock event irq 201
> > [ 0.008235] [Firmware Bug]: CPU 5: APIC ID mismatch. CPUID: 0x0005 APIC: 0x0002
> > [ 0.008235] [Firmware Bug]: CPU 5: APIC ID mismatch. Firmware: 0x0021 APIC: 0x0002
> > [ 1.497074] cpu 5 spinlock event irq 202
> > [ 0.008235] [Firmware Bug]: CPU 7: APIC I
> >
> > Full console log (containing both successful boot of Xen 4.19, and then
> > reboot into 4.22):
> > https://openqa.qubes-os.org/tests/184780/logfile?filename=serial0.txt
>
> The 4.19 log also has an anomaly around this point in time. Can you try
> again with sync_console added to both the 4.19 and the 4.22 attempt?
Yes, sync_console helped quite a bit, now I get full dom0 panic message:
[ 10.334800] vesafb: cannot reserve video memory at 0x0
[ 10.340009] vesafb: mode is 0x0x0, linelength=0, pages=0
[ 10.345515] Oops: divide error: 0000 [#1] SMP NOPTI
[ 10.346503] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 6.17.9-1.qubes.fc41.x86_64 #1 PREEMPT(full)
[ 10.346503] Hardware name: Notebook V54x_6x_TU/V54x_6x_TU, BIOS Dasharo (coreboot+heads) v0.9.0 01/01/1970
[ 10.346503] RIP: e030:vesafb_probe.cold+0xd4/0x5fb
[ 10.346503] Code: 08 75 1f 83 3d a8 8c 1d 02 00 75 16 48 c7 c7 90 4a fd 81 e8 8a ef f9 ff c7 05 d4 54 09 02 05 00 00 00 8b 05 c2 54 09 02 31 d2 <f7> 35 d2 54 09 02 8b 15 fc 54 09 02 48 89 c1 48 c1 e1 20 48 09 ca
[ 10.346503] RSP: e02b:ffffc9004001fbb8 EFLAGS: 00010246
[ 10.346503] RAX: 0000000000000000 RBX: ffff888101d86f28 RCX: ffffffff823666e8
[ 10.346503] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
[ 10.346503] RBP: ffff88810197d400 R08: 0000000000000000 R09: 6c656e696c202c30
[ 10.346503] R10: 0000000000000030 R11: 203a626661736576 R12: 0000000000000000
[ 10.346503] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888106351000
[ 10.346503] FS: 0000000000000000(0000) GS:ffff888cef7c1000(0000) knlGS:0000000000000000
[ 10.346503] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 10.346503] CR2: ffffc900064ff000 CR3: 000000000222c000 CR4: 0000000000050660
[ 10.346503] Call Trace:
[ 10.346503] <TASK>
[ 10.346503] ? __pfx___device_attach_driver+0x10/0x10
[ 10.346503] platform_probe+0x39/0x70
[ 10.346503] really_probe+0xdb/0x340
[ 10.346503] ? pm_runtime_barrier+0x54/0x90
[ 10.346503] __driver_probe_device+0x78/0x110
[ 10.346503] driver_probe_device+0x1f/0xa0
[ 10.346503] __device_attach_driver+0x89/0x110
[ 10.346503] bus_for_each_drv+0x94/0xf0
[ 10.346503] __device_attach+0xaf/0x1b0
[ 10.346503] bus_probe_device+0x8d/0xa0
[ 10.346503] device_add+0x508/0x710
[ 10.346503] platform_device_add+0xed/0x250
[ 10.346503] sysfb_init+0x283/0x320
[ 10.346503] ? __pfx_sysfb_init+0x10/0x10
[ 10.346503] do_one_initcall+0x57/0x310
[ 10.346503] do_initcalls+0x1ef/0x240
[ 10.346503] kernel_init_freeable+0x187/0x210
[ 10.346503] ? __pfx_kernel_init+0x10/0x10
[ 10.346503] kernel_init+0x1a/0x140
[ 10.346503] ret_from_fork+0xf2/0x110
[ 10.346503] ? __pfx_kernel_init+0x10/0x10
[ 10.346503] ret_from_fork_asm+0x1a/0x30
[ 10.346503] </TASK>
[ 10.346503] Modules linked in:
[ 10.559786] ---[ end trace 0000000000000000 ]---
[ 10.564581] RIP: e030:vesafb_probe.cold+0xd4/0x5fb
[ 10.569546] Code: 08 75 1f 83 3d a8 8c 1d 02 00 75 16 48 c7 c7 90 4a fd 81 e8 8a ef f9 ff c7 05 d4 54 09 02 05 00 00 00 8b 05 c2 54 09 02 31 d2 <f7> 35 d2 54 09 02 8b 15 fc 54 09 02 48 89 c1 48 c1 e1 20 48 09 ca
[ 10.588833] RSP: e02b:ffffc9004001fbb8 EFLAGS: 00010246
[ 10.594255] RAX: 0000000000000000 RBX: ffff888101d86f28 RCX: ffffffff823666e8
[ 10.601622] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
[ 10.609000] RBP: ffff88810197d400 R08: 0000000000000000 R09: 6c656e696c202c30
[ 10.616378] R10: 0000000000000030 R11: 203a626661736576 R12: 0000000000000000
[ 10.623755] R13: 0000000000000000 R14: 0000000000000000 R15: ffff888106351000
[ 10.631136] FS: 0000000000000000(0000) GS:ffff888cef7c1000(0000) knlGS:0000000000000000
[ 10.639483] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 10.645434] CR2: ffffc900064ff000 CR3: 000000000222c000 CR4: 0000000000050660
[ 10.652817] Kernel panic - not syncing: Fatal exception
[ 10.653803] Kernel Offset: disabled
(XEN) Hardware Dom0 crashed: 'noreboot' set - not rebooting.
>
> > It doesn't reach loading graphics driver in dom0, so I don't have
> > anything interesting on VGA (the last output is about the kexec call
> > done by Heads). But at least I have a serial console.
>
> Yet interestingly Xen's " VGA is ... mode ..." lines differ as well.
This might be relevant given the above.
> That may be a separate issue. Or maybe that's how one of the EFI changes
> that were done a while ago actually surfaces?
But this isn't booting via EFI... Heads is coreboot + Linux payload and
it uses kexec to boot Xen (via multiboot2 entry point).
> Another difference is
>
> (XEN) HWP: Using HWP for cpufreq
>
> in the 4.19 log vs
>
> (XEN) HPET: 8 timers usable for broadcast (8 total)
>
> in the 4.22 one. Does the latter perhaps go away if you include [1] in
> the build?
Yes
>
> For the HWP driver do you perhaps have a patch in 4.19 to default-enable
> it, but you don't have a similar patch in 4.22?
Hm, I do have that patch in 4.22 too, but maybe I failed at rebase...
> If so, to compare apples
> to apples the driver wants enabling/disabling consistently (via command
> line option) for both runs.
Since here booting just dom0 is enough, I don't need any of the qubes
patches. So, now I tested vanilla 4.19 vs staging - both console logs at:
https://gist.github.com/marmarek/e017f0a9ad3cc6ab2bac8849492c41a9
> And then "mwait-idle: lapic_timer_reliable_states ..." differ as well.
>
> Jan
>
> [1] https://lists.xen.org/archives/html/xen-devel/2026-04/msg00200.html
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-06-24 0:32 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-19 14:38 Linux dom0 hangs on boot with Xen 4.22 and Heads firmware Marek Marczykowski-Górecki
2026-06-22 8:53 ` Jan Beulich
2026-06-24 0:31 ` Marek Marczykowski-Górecki
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.