* domU crash when resuming from suspend - Xen 4.22
@ 2026-06-19 11:24 Marek Marczykowski-Górecki
2026-07-14 17:12 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 10+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-06-19 11:24 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 4467 bytes --]
Hi,
I'm continuing testing 4.22 and found this:
When a domU is suspended (even without host suspend) it crashes on
resume with:
[ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
[ 24.374647] #PF: supervisor read access in kernel mode
[ 24.374657] #PF: error_code(0x0000) - not-present page
[ 24.374668] PGD 0 P4D 0
[ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
[ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
[ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
[ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
[ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
[ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
[ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
[ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
[ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
[ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
[ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
[ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
[ 24.374873] PKRU: 00000000
[ 24.374877] Call Trace:
[ 24.374884] <TASK>
[ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
[ 24.374900] cpu_stopper_thread+0xa3/0x170
[ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
[ 24.374917] smpboot_thread_fn+0xf3/0x220
[ 24.374925] kthread+0xfc/0x240
[ 24.374933] ? __pfx_kthread+0x10/0x10
[ 24.374940] ? __pfx_kthread+0x10/0x10
[ 24.374949] ret_from_fork+0x158/0x170
[ 24.374957] ? __pfx_kthread+0x10/0x10
[ 24.374965] ret_from_fork_asm+0x1a/0x30
[ 24.374973] </TASK>
[ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
[ 24.375064] CR2: 0000000000000020
[ 24.375071] ---[ end trace 0000000000000000 ]---
[ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
[ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
[ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
[ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
[ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
[ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
[ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
[ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
[ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
[ 24.375241] PKRU: 00000000
[ 24.375247] Kernel panic - not syncing: Fatal exception
[ 25.417403] Shutting down cpus with NMI
[ 25.417447] Kernel Offset: disabled
There is no change on the Linux kernel side (neither dom0 nor domU), the
only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
tried PV domU (but dom0 survives host suspend).
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-06-19 11:24 domU crash when resuming from suspend - Xen 4.22 Marek Marczykowski-Górecki
@ 2026-07-14 17:12 ` Marek Marczykowski-Górecki
2026-07-15 6:16 ` Jan Beulich
2026-07-17 20:22 ` Frediano Ziglio
0 siblings, 2 replies; 10+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-07-14 17:12 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 4681 bytes --]
On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> Hi,
>
> I'm continuing testing 4.22 and found this:
>
> When a domU is suspended (even without host suspend) it crashes on
> resume with:
>
> [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> [ 24.374647] #PF: supervisor read access in kernel mode
> [ 24.374657] #PF: error_code(0x0000) - not-present page
> [ 24.374668] PGD 0 P4D 0
> [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> [ 24.374873] PKRU: 00000000
> [ 24.374877] Call Trace:
> [ 24.374884] <TASK>
> [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> [ 24.374900] cpu_stopper_thread+0xa3/0x170
> [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> [ 24.374917] smpboot_thread_fn+0xf3/0x220
> [ 24.374925] kthread+0xfc/0x240
> [ 24.374933] ? __pfx_kthread+0x10/0x10
> [ 24.374940] ? __pfx_kthread+0x10/0x10
> [ 24.374949] ret_from_fork+0x158/0x170
> [ 24.374957] ? __pfx_kthread+0x10/0x10
> [ 24.374965] ret_from_fork_asm+0x1a/0x30
> [ 24.374973] </TASK>
> [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> [ 24.375064] CR2: 0000000000000020
> [ 24.375071] ---[ end trace 0000000000000000 ]---
> [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> [ 24.375241] PKRU: 00000000
> [ 24.375247] Kernel panic - not syncing: Fatal exception
> [ 25.417403] Shutting down cpus with NMI
> [ 25.417447] Kernel Offset: disabled
>
> There is no change on the Linux kernel side (neither dom0 nor domU), the
> only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> tried PV domU (but dom0 survives host suspend).
Any ideas?
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-14 17:12 ` Marek Marczykowski-Górecki
@ 2026-07-15 6:16 ` Jan Beulich
2026-07-17 20:22 ` Frediano Ziglio
1 sibling, 0 replies; 10+ messages in thread
From: Jan Beulich @ 2026-07-15 6:16 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel
On 14.07.2026 19:12, Marek Marczykowski-Górecki wrote:
> On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
>> Hi,
>>
>> I'm continuing testing 4.22 and found this:
>>
>> When a domU is suspended (even without host suspend) it crashes on
>> resume with:
>>
>> [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
>> [ 24.374647] #PF: supervisor read access in kernel mode
>> [ 24.374657] #PF: error_code(0x0000) - not-present page
>> [ 24.374668] PGD 0 P4D 0
>> [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
>> [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
>> [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
>> [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
>> [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
>> [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
>> [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
>> [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
>> [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
>> [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
>> [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
>> [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
>> [ 24.374873] PKRU: 00000000
>> [ 24.374877] Call Trace:
>> [ 24.374884] <TASK>
>> [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
>> [ 24.374900] cpu_stopper_thread+0xa3/0x170
>> [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
>> [ 24.374917] smpboot_thread_fn+0xf3/0x220
>> [ 24.374925] kthread+0xfc/0x240
>> [ 24.374933] ? __pfx_kthread+0x10/0x10
>> [ 24.374940] ? __pfx_kthread+0x10/0x10
>> [ 24.374949] ret_from_fork+0x158/0x170
>> [ 24.374957] ? __pfx_kthread+0x10/0x10
>> [ 24.374965] ret_from_fork_asm+0x1a/0x30
>> [ 24.374973] </TASK>
>> [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
>> [ 24.375064] CR2: 0000000000000020
>> [ 24.375071] ---[ end trace 0000000000000000 ]---
>> [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
>> [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
>> [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
>> [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
>> [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
>> [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
>> [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
>> [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
>> [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
>> [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
>> [ 24.375241] PKRU: 00000000
>> [ 24.375247] Kernel panic - not syncing: Fatal exception
>> [ 25.417403] Shutting down cpus with NMI
>> [ 25.417447] Kernel Offset: disabled
>>
>> There is no change on the Linux kernel side (neither dom0 nor domU), the
>> only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
>> broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
>> tried PV domU (but dom0 survives host suspend).
>
> Any ideas?
Didn't you indicate on Matrix that there's also a kernel version dependency?
Iirc 6.12 doesn't exhibit the issue.
Short of anyone having a good idea from just the data available, perhaps the
next step would be to further limit version ranges on both sides?
Jan
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-14 17:12 ` Marek Marczykowski-Górecki
2026-07-15 6:16 ` Jan Beulich
@ 2026-07-17 20:22 ` Frediano Ziglio
2026-07-18 14:03 ` Marek Marczykowski-Górecki
1 sibling, 1 reply; 10+ messages in thread
From: Frediano Ziglio @ 2026-07-17 20:22 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel
On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:
>
> On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> > Hi,
> >
> > I'm continuing testing 4.22 and found this:
> >
> > When a domU is suspended (even without host suspend) it crashes on
> > resume with:
> >
> > [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > [ 24.374647] #PF: supervisor read access in kernel mode
> > [ 24.374657] #PF: error_code(0x0000) - not-present page
> > [ 24.374668] PGD 0 P4D 0
> > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > [ 24.374873] PKRU: 00000000
> > [ 24.374877] Call Trace:
> > [ 24.374884] <TASK>
> > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> > [ 24.374900] cpu_stopper_thread+0xa3/0x170
> > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> > [ 24.374917] smpboot_thread_fn+0xf3/0x220
> > [ 24.374925] kthread+0xfc/0x240
> > [ 24.374933] ? __pfx_kthread+0x10/0x10
> > [ 24.374940] ? __pfx_kthread+0x10/0x10
> > [ 24.374949] ret_from_fork+0x158/0x170
> > [ 24.374957] ? __pfx_kthread+0x10/0x10
> > [ 24.374965] ret_from_fork_asm+0x1a/0x30
> > [ 24.374973] </TASK>
> > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> > [ 24.375064] CR2: 0000000000000020
> > [ 24.375071] ---[ end trace 0000000000000000 ]---
> > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > [ 24.375241] PKRU: 00000000
> > [ 24.375247] Kernel panic - not syncing: Fatal exception
> > [ 25.417403] Shutting down cpus with NMI
> > [ 25.417447] Kernel Offset: disabled
> >
> > There is no change on the Linux kernel side (neither dom0 nor domU), the
> > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> > tried PV domU (but dom0 survives host suspend).
>
> Any ideas?
>
Hi,
not many ideas but while testing my libxenguest changes I found
similar failures using xl create/save/restore. But I was testing more
PV guests.
Does 4.21 work ?
Frediano
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-17 20:22 ` Frediano Ziglio
@ 2026-07-18 14:03 ` Marek Marczykowski-Górecki
2026-07-19 1:59 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 10+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-07-18 14:03 UTC (permalink / raw)
To: Frediano Ziglio; +Cc: xen-devel
[-- Attachment #1: Type: text/plain, Size: 5538 bytes --]
On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote:
> On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
> <marmarek@invisiblethingslab.com> wrote:
> >
> > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> > > Hi,
> > >
> > > I'm continuing testing 4.22 and found this:
> > >
> > > When a domU is suspended (even without host suspend) it crashes on
> > > resume with:
> > >
> > > [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > > [ 24.374647] #PF: supervisor read access in kernel mode
> > > [ 24.374657] #PF: error_code(0x0000) - not-present page
> > > [ 24.374668] PGD 0 P4D 0
> > > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> > > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > [ 24.374873] PKRU: 00000000
> > > [ 24.374877] Call Trace:
> > > [ 24.374884] <TASK>
> > > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> > > [ 24.374900] cpu_stopper_thread+0xa3/0x170
> > > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> > > [ 24.374917] smpboot_thread_fn+0xf3/0x220
> > > [ 24.374925] kthread+0xfc/0x240
> > > [ 24.374933] ? __pfx_kthread+0x10/0x10
> > > [ 24.374940] ? __pfx_kthread+0x10/0x10
> > > [ 24.374949] ret_from_fork+0x158/0x170
> > > [ 24.374957] ? __pfx_kthread+0x10/0x10
> > > [ 24.374965] ret_from_fork_asm+0x1a/0x30
> > > [ 24.374973] </TASK>
> > > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> > > [ 24.375064] CR2: 0000000000000020
> > > [ 24.375071] ---[ end trace 0000000000000000 ]---
> > > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > [ 24.375241] PKRU: 00000000
> > > [ 24.375247] Kernel panic - not syncing: Fatal exception
> > > [ 25.417403] Shutting down cpus with NMI
> > > [ 25.417447] Kernel Offset: disabled
> > >
> > > There is no change on the Linux kernel side (neither dom0 nor domU), the
> > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > > broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> > > tried PV domU (but dom0 survives host suspend).
> >
> > Any ideas?
> >
>
> Hi,
> not many ideas but while testing my libxenguest changes I found
> similar failures using xl create/save/restore. But I was testing more
> PV guests.
> Does 4.21 work ?
It seems 4.21 also has this issue:
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523
But 4.20 is okay:
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095
I guess I can bisect it further.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-18 14:03 ` Marek Marczykowski-Górecki
@ 2026-07-19 1:59 ` Marek Marczykowski-Górecki
2026-07-19 7:31 ` Frediano Ziglio
0 siblings, 1 reply; 10+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-07-19 1:59 UTC (permalink / raw)
To: Frediano Ziglio; +Cc: xen-devel, Andrew Cooper
[-- Attachment #1: Type: text/plain, Size: 6304 bytes --]
On Sat, Jul 18, 2026 at 04:03:29PM +0200, Marek Marczykowski-Górecki wrote:
> On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote:
> > On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
> > <marmarek@invisiblethingslab.com> wrote:
> > >
> > > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> > > > Hi,
> > > >
> > > > I'm continuing testing 4.22 and found this:
> > > >
> > > > When a domU is suspended (even without host suspend) it crashes on
> > > > resume with:
> > > >
> > > > [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > > > [ 24.374647] #PF: supervisor read access in kernel mode
> > > > [ 24.374657] #PF: error_code(0x0000) - not-present page
> > > > [ 24.374668] PGD 0 P4D 0
> > > > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > > > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > > > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> > > > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > [ 24.374873] PKRU: 00000000
> > > > [ 24.374877] Call Trace:
> > > > [ 24.374884] <TASK>
> > > > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> > > > [ 24.374900] cpu_stopper_thread+0xa3/0x170
> > > > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> > > > [ 24.374917] smpboot_thread_fn+0xf3/0x220
> > > > [ 24.374925] kthread+0xfc/0x240
> > > > [ 24.374933] ? __pfx_kthread+0x10/0x10
> > > > [ 24.374940] ? __pfx_kthread+0x10/0x10
> > > > [ 24.374949] ret_from_fork+0x158/0x170
> > > > [ 24.374957] ? __pfx_kthread+0x10/0x10
> > > > [ 24.374965] ret_from_fork_asm+0x1a/0x30
> > > > [ 24.374973] </TASK>
> > > > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> > > > [ 24.375064] CR2: 0000000000000020
> > > > [ 24.375071] ---[ end trace 0000000000000000 ]---
> > > > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > [ 24.375241] PKRU: 00000000
> > > > [ 24.375247] Kernel panic - not syncing: Fatal exception
> > > > [ 25.417403] Shutting down cpus with NMI
> > > > [ 25.417447] Kernel Offset: disabled
> > > >
> > > > There is no change on the Linux kernel side (neither dom0 nor domU), the
> > > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > > > broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> > > > tried PV domU (but dom0 survives host suspend).
> > >
> > > Any ideas?
> > >
> >
> > Hi,
> > not many ideas but while testing my libxenguest changes I found
> > similar failures using xl create/save/restore. But I was testing more
> > PV guests.
> > Does 4.21 work ?
>
> It seems 4.21 also has this issue:
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523
> But 4.20 is okay:
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095
>
> I guess I can bisect it further.
So, bisect says the first bad commit is:
commit 9f892f84c27970840d255aef7e2851b762967de7
Author: Andrew Cooper <andrew.cooper3@citrix.com>
Date: Mon Dec 30 11:49:14 2024 +0000
x86/domctl: Stop using XLAT_cpu_user_regs()
My test isn't 100% reliable, but I retested parent of this commit a
couple of times and it was good every time.
I wanted to test reverting it on top of 4.21, but there is a lot of
FRED-related work on top that makes such revert quite hard, to say the
least...
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-19 1:59 ` Marek Marczykowski-Górecki
@ 2026-07-19 7:31 ` Frediano Ziglio
2026-07-19 19:40 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 10+ messages in thread
From: Frediano Ziglio @ 2026-07-19 7:31 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel, Andrew Cooper
On Sun, 19 Jul 2026 at 03:00, Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:
>
> On Sat, Jul 18, 2026 at 04:03:29PM +0200, Marek Marczykowski-Górecki wrote:
> > On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote:
> > > On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
> > > <marmarek@invisiblethingslab.com> wrote:
> > > >
> > > > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > Hi,
> > > > >
> > > > > I'm continuing testing 4.22 and found this:
> > > > >
> > > > > When a domU is suspended (even without host suspend) it crashes on
> > > > > resume with:
> > > > >
> > > > > [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > > > > [ 24.374647] #PF: supervisor read access in kernel mode
> > > > > [ 24.374657] #PF: error_code(0x0000) - not-present page
> > > > > [ 24.374668] PGD 0 P4D 0
> > > > > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > > > > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > > > > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> > > > > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > [ 24.374873] PKRU: 00000000
> > > > > [ 24.374877] Call Trace:
> > > > > [ 24.374884] <TASK>
> > > > > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> > > > > [ 24.374900] cpu_stopper_thread+0xa3/0x170
> > > > > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> > > > > [ 24.374917] smpboot_thread_fn+0xf3/0x220
> > > > > [ 24.374925] kthread+0xfc/0x240
> > > > > [ 24.374933] ? __pfx_kthread+0x10/0x10
> > > > > [ 24.374940] ? __pfx_kthread+0x10/0x10
> > > > > [ 24.374949] ret_from_fork+0x158/0x170
> > > > > [ 24.374957] ? __pfx_kthread+0x10/0x10
> > > > > [ 24.374965] ret_from_fork_asm+0x1a/0x30
> > > > > [ 24.374973] </TASK>
> > > > > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> > > > > [ 24.375064] CR2: 0000000000000020
> > > > > [ 24.375071] ---[ end trace 0000000000000000 ]---
> > > > > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > [ 24.375241] PKRU: 00000000
> > > > > [ 24.375247] Kernel panic - not syncing: Fatal exception
> > > > > [ 25.417403] Shutting down cpus with NMI
> > > > > [ 25.417447] Kernel Offset: disabled
> > > > >
> > > > > There is no change on the Linux kernel side (neither dom0 nor domU), the
> > > > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > > > > broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> > > > > tried PV domU (but dom0 survives host suspend).
> > > >
> > > > Any ideas?
> > > >
> > >
> > > Hi,
> > > not many ideas but while testing my libxenguest changes I found
> > > similar failures using xl create/save/restore. But I was testing more
> > > PV guests.
> > > Does 4.21 work ?
> >
> > It seems 4.21 also has this issue:
> > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523
> > But 4.20 is okay:
> > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095
> >
> > I guess I can bisect it further.
>
> So, bisect says the first bad commit is:
>
> commit 9f892f84c27970840d255aef7e2851b762967de7
> Author: Andrew Cooper <andrew.cooper3@citrix.com>
> Date: Mon Dec 30 11:49:14 2024 +0000
>
> x86/domctl: Stop using XLAT_cpu_user_regs()
>
> My test isn't 100% reliable, but I retested parent of this commit a
> couple of times and it was good every time.
>
> I wanted to test reverting it on top of 4.21, but there is a lot of
> FRED-related work on top that makes such revert quite hard, to say the
> least...
>
The generated XLAT_cpu_user_regs was
#define XLAT_cpu_user_regs(_d_, _s_) do { \
(_d_)->__DECL_REG_LO8(b) = (_s_)->__DECL_REG_LO8(b); \
(_d_)->__DECL_REG_LO8(c) = (_s_)->__DECL_REG_LO8(c); \
(_d_)->__DECL_REG_LO8(d) = (_s_)->__DECL_REG_LO8(d); \
(_d_)->__DECL_REG_LO16(si) = (_s_)->__DECL_REG_LO16(si); \
(_d_)->__DECL_REG_LO16(di) = (_s_)->__DECL_REG_LO16(di); \
(_d_)->__DECL_REG_LO16(bp) = (_s_)->__DECL_REG_LO16(bp); \
(_d_)->__DECL_REG_LO8(a) = (_s_)->__DECL_REG_LO8(a); \
(_d_)->error_code = (_s_)->error_code; \
(_d_)->entry_vector = (_s_)->entry_vector; \
(_d_)->__DECL_REG_LO16(ip) = (_s_)->__DECL_REG_LO16(ip); \
(_d_)->cs = (_s_)->cs; \
(_d_)->saved_upcall_mask = (_s_)->saved_upcall_mask; \
(_d_)->__DECL_REG_LO16(flags) = (_s_)->__DECL_REG_LO16(flags); \
(_d_)->__DECL_REG_LO16(sp) = (_s_)->__DECL_REG_LO16(sp); \
(_d_)->ss = (_s_)->ss; \
(_d_)->es = (_s_)->es; \
(_d_)->ds = (_s_)->ds; \
(_d_)->fs = (_s_)->fs; \
(_d_)->gs = (_s_)->gs; \
} while (0)
registers are copied, but additional fields (error_code, entry_vector,
saved_upcall_mask).
Maybe you could try to copy them again.
Also, just to be on the safe side, you could memset the structure
where "Backing memory is pre-zeroed" comments are.
Frediano
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-19 7:31 ` Frediano Ziglio
@ 2026-07-19 19:40 ` Marek Marczykowski-Górecki
2026-07-19 21:04 ` Frediano Ziglio
0 siblings, 1 reply; 10+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-07-19 19:40 UTC (permalink / raw)
To: Frediano Ziglio; +Cc: xen-devel, Andrew Cooper
[-- Attachment #1: Type: text/plain, Size: 9461 bytes --]
On Sun, Jul 19, 2026 at 08:31:28AM +0100, Frediano Ziglio wrote:
> On Sun, 19 Jul 2026 at 03:00, Marek Marczykowski-Górecki
> <marmarek@invisiblethingslab.com> wrote:
> >
> > On Sat, Jul 18, 2026 at 04:03:29PM +0200, Marek Marczykowski-Górecki wrote:
> > > On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote:
> > > > On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
> > > > <marmarek@invisiblethingslab.com> wrote:
> > > > >
> > > > > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm continuing testing 4.22 and found this:
> > > > > >
> > > > > > When a domU is suspended (even without host suspend) it crashes on
> > > > > > resume with:
> > > > > >
> > > > > > [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > > > > > [ 24.374647] #PF: supervisor read access in kernel mode
> > > > > > [ 24.374657] #PF: error_code(0x0000) - not-present page
> > > > > > [ 24.374668] PGD 0 P4D 0
> > > > > > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > > > > > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > > > > > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> > > > > > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > > [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > > [ 24.374873] PKRU: 00000000
> > > > > > [ 24.374877] Call Trace:
> > > > > > [ 24.374884] <TASK>
> > > > > > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> > > > > > [ 24.374900] cpu_stopper_thread+0xa3/0x170
> > > > > > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> > > > > > [ 24.374917] smpboot_thread_fn+0xf3/0x220
> > > > > > [ 24.374925] kthread+0xfc/0x240
> > > > > > [ 24.374933] ? __pfx_kthread+0x10/0x10
> > > > > > [ 24.374940] ? __pfx_kthread+0x10/0x10
> > > > > > [ 24.374949] ret_from_fork+0x158/0x170
> > > > > > [ 24.374957] ? __pfx_kthread+0x10/0x10
> > > > > > [ 24.374965] ret_from_fork_asm+0x1a/0x30
> > > > > > [ 24.374973] </TASK>
> > > > > > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> > > > > > [ 24.375064] CR2: 0000000000000020
> > > > > > [ 24.375071] ---[ end trace 0000000000000000 ]---
> > > > > > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > > [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > > [ 24.375241] PKRU: 00000000
> > > > > > [ 24.375247] Kernel panic - not syncing: Fatal exception
> > > > > > [ 25.417403] Shutting down cpus with NMI
> > > > > > [ 25.417447] Kernel Offset: disabled
> > > > > >
> > > > > > There is no change on the Linux kernel side (neither dom0 nor domU), the
> > > > > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > > > > > broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> > > > > > tried PV domU (but dom0 survives host suspend).
> > > > >
> > > > > Any ideas?
> > > > >
> > > >
> > > > Hi,
> > > > not many ideas but while testing my libxenguest changes I found
> > > > similar failures using xl create/save/restore. But I was testing more
> > > > PV guests.
> > > > Does 4.21 work ?
> > >
> > > It seems 4.21 also has this issue:
> > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523
> > > But 4.20 is okay:
> > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095
> > >
> > > I guess I can bisect it further.
> >
> > So, bisect says the first bad commit is:
> >
> > commit 9f892f84c27970840d255aef7e2851b762967de7
> > Author: Andrew Cooper <andrew.cooper3@citrix.com>
> > Date: Mon Dec 30 11:49:14 2024 +0000
> >
> > x86/domctl: Stop using XLAT_cpu_user_regs()
> >
> > My test isn't 100% reliable, but I retested parent of this commit a
> > couple of times and it was good every time.
> >
> > I wanted to test reverting it on top of 4.21, but there is a lot of
> > FRED-related work on top that makes such revert quite hard, to say the
> > least...
> >
>
> The generated XLAT_cpu_user_regs was
>
> #define XLAT_cpu_user_regs(_d_, _s_) do { \
> (_d_)->__DECL_REG_LO8(b) = (_s_)->__DECL_REG_LO8(b); \
> (_d_)->__DECL_REG_LO8(c) = (_s_)->__DECL_REG_LO8(c); \
> (_d_)->__DECL_REG_LO8(d) = (_s_)->__DECL_REG_LO8(d); \
> (_d_)->__DECL_REG_LO16(si) = (_s_)->__DECL_REG_LO16(si); \
> (_d_)->__DECL_REG_LO16(di) = (_s_)->__DECL_REG_LO16(di); \
> (_d_)->__DECL_REG_LO16(bp) = (_s_)->__DECL_REG_LO16(bp); \
> (_d_)->__DECL_REG_LO8(a) = (_s_)->__DECL_REG_LO8(a); \
> (_d_)->error_code = (_s_)->error_code; \
> (_d_)->entry_vector = (_s_)->entry_vector; \
> (_d_)->__DECL_REG_LO16(ip) = (_s_)->__DECL_REG_LO16(ip); \
> (_d_)->cs = (_s_)->cs; \
> (_d_)->saved_upcall_mask = (_s_)->saved_upcall_mask; \
> (_d_)->__DECL_REG_LO16(flags) = (_s_)->__DECL_REG_LO16(flags); \
> (_d_)->__DECL_REG_LO16(sp) = (_s_)->__DECL_REG_LO16(sp); \
> (_d_)->ss = (_s_)->ss; \
> (_d_)->es = (_s_)->es; \
> (_d_)->ds = (_s_)->ds; \
> (_d_)->fs = (_s_)->fs; \
> (_d_)->gs = (_s_)->gs; \
> } while (0)
>
> registers are copied, but additional fields (error_code, entry_vector,
> saved_upcall_mask).
> Maybe you could try to copy them again.
> Also, just to be on the safe side, you could memset the structure
> where "Backing memory is pre-zeroed" comments are.
This does not help :(
Specifically:
1. Applied on top of staging-4.21:
- build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/188ca612c51a57ef54b917c833eaeb59666994c9
- test (fails): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688813821
2. Applied directly on top of 9f892f84c27970840d255aef7e2851b762967de7:
- build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dd3dee35fc4fcd6ba284f3f5fa78b333c472888e
- test (fails): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688832398
Yet, parent of 9f892f84c27970840d255aef7e2851b762967de7 still passes:
- build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dbcbbed4e9dc25faa211d359c2f04a9c70f087c9
- test (pass): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688835512
Hint about reading the above: I use two branches, one for building
(branch "test") and one for running the test (branch "automation-uki-grub-only"),
which takes build artifacts from the build branch. I do that to not
rebase test scripts. It's a bit opaque, but the console log of the test
contains "Latest ChangeSet" fields telling what actually was tested.
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-19 19:40 ` Marek Marczykowski-Górecki
@ 2026-07-19 21:04 ` Frediano Ziglio
2026-07-19 22:50 ` Marek Marczykowski-Górecki
0 siblings, 1 reply; 10+ messages in thread
From: Frediano Ziglio @ 2026-07-19 21:04 UTC (permalink / raw)
To: Marek Marczykowski-Górecki; +Cc: xen-devel, Andrew Cooper
On Sun, 19 Jul 2026 at 20:40, Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:
>
> On Sun, Jul 19, 2026 at 08:31:28AM +0100, Frediano Ziglio wrote:
> > On Sun, 19 Jul 2026 at 03:00, Marek Marczykowski-Górecki
> > <marmarek@invisiblethingslab.com> wrote:
> > >
> > > On Sat, Jul 18, 2026 at 04:03:29PM +0200, Marek Marczykowski-Górecki wrote:
> > > > On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote:
> > > > > On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
> > > > > <marmarek@invisiblethingslab.com> wrote:
> > > > > >
> > > > > > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm continuing testing 4.22 and found this:
> > > > > > >
> > > > > > > When a domU is suspended (even without host suspend) it crashes on
> > > > > > > resume with:
> > > > > > >
> > > > > > > [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > > > > > > [ 24.374647] #PF: supervisor read access in kernel mode
> > > > > > > [ 24.374657] #PF: error_code(0x0000) - not-present page
> > > > > > > [ 24.374668] PGD 0 P4D 0
> > > > > > > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > > > > > > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > > > > > > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> > > > > > > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > > > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > > > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > > > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > > > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > > > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > > > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > > > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > > > [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > > > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > > > [ 24.374873] PKRU: 00000000
> > > > > > > [ 24.374877] Call Trace:
> > > > > > > [ 24.374884] <TASK>
> > > > > > > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> > > > > > > [ 24.374900] cpu_stopper_thread+0xa3/0x170
> > > > > > > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> > > > > > > [ 24.374917] smpboot_thread_fn+0xf3/0x220
> > > > > > > [ 24.374925] kthread+0xfc/0x240
> > > > > > > [ 24.374933] ? __pfx_kthread+0x10/0x10
> > > > > > > [ 24.374940] ? __pfx_kthread+0x10/0x10
> > > > > > > [ 24.374949] ret_from_fork+0x158/0x170
> > > > > > > [ 24.374957] ? __pfx_kthread+0x10/0x10
> > > > > > > [ 24.374965] ret_from_fork_asm+0x1a/0x30
> > > > > > > [ 24.374973] </TASK>
> > > > > > > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> > > > > > > [ 24.375064] CR2: 0000000000000020
> > > > > > > [ 24.375071] ---[ end trace 0000000000000000 ]---
> > > > > > > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > > > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > > > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > > > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > > > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > > > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > > > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > > > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > > > [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > > > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > > > [ 24.375241] PKRU: 00000000
> > > > > > > [ 24.375247] Kernel panic - not syncing: Fatal exception
> > > > > > > [ 25.417403] Shutting down cpus with NMI
> > > > > > > [ 25.417447] Kernel Offset: disabled
> > > > > > >
> > > > > > > There is no change on the Linux kernel side (neither dom0 nor domU), the
> > > > > > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > > > > > > broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> > > > > > > tried PV domU (but dom0 survives host suspend).
> > > > > >
> > > > > > Any ideas?
> > > > > >
> > > > >
> > > > > Hi,
> > > > > not many ideas but while testing my libxenguest changes I found
> > > > > similar failures using xl create/save/restore. But I was testing more
> > > > > PV guests.
> > > > > Does 4.21 work ?
> > > >
> > > > It seems 4.21 also has this issue:
> > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523
> > > > But 4.20 is okay:
> > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095
> > > >
> > > > I guess I can bisect it further.
> > >
> > > So, bisect says the first bad commit is:
> > >
> > > commit 9f892f84c27970840d255aef7e2851b762967de7
> > > Author: Andrew Cooper <andrew.cooper3@citrix.com>
> > > Date: Mon Dec 30 11:49:14 2024 +0000
> > >
> > > x86/domctl: Stop using XLAT_cpu_user_regs()
> > >
> > > My test isn't 100% reliable, but I retested parent of this commit a
> > > couple of times and it was good every time.
> > >
> > > I wanted to test reverting it on top of 4.21, but there is a lot of
> > > FRED-related work on top that makes such revert quite hard, to say the
> > > least...
> > >
> >
> > The generated XLAT_cpu_user_regs was
> >
> > #define XLAT_cpu_user_regs(_d_, _s_) do { \
> > (_d_)->__DECL_REG_LO8(b) = (_s_)->__DECL_REG_LO8(b); \
> > (_d_)->__DECL_REG_LO8(c) = (_s_)->__DECL_REG_LO8(c); \
> > (_d_)->__DECL_REG_LO8(d) = (_s_)->__DECL_REG_LO8(d); \
> > (_d_)->__DECL_REG_LO16(si) = (_s_)->__DECL_REG_LO16(si); \
> > (_d_)->__DECL_REG_LO16(di) = (_s_)->__DECL_REG_LO16(di); \
> > (_d_)->__DECL_REG_LO16(bp) = (_s_)->__DECL_REG_LO16(bp); \
> > (_d_)->__DECL_REG_LO8(a) = (_s_)->__DECL_REG_LO8(a); \
> > (_d_)->error_code = (_s_)->error_code; \
> > (_d_)->entry_vector = (_s_)->entry_vector; \
> > (_d_)->__DECL_REG_LO16(ip) = (_s_)->__DECL_REG_LO16(ip); \
> > (_d_)->cs = (_s_)->cs; \
> > (_d_)->saved_upcall_mask = (_s_)->saved_upcall_mask; \
> > (_d_)->__DECL_REG_LO16(flags) = (_s_)->__DECL_REG_LO16(flags); \
> > (_d_)->__DECL_REG_LO16(sp) = (_s_)->__DECL_REG_LO16(sp); \
> > (_d_)->ss = (_s_)->ss; \
> > (_d_)->es = (_s_)->es; \
> > (_d_)->ds = (_s_)->ds; \
> > (_d_)->fs = (_s_)->fs; \
> > (_d_)->gs = (_s_)->gs; \
> > } while (0)
> >
> > registers are copied, but additional fields (error_code, entry_vector,
> > saved_upcall_mask).
> > Maybe you could try to copy them again.
> > Also, just to be on the safe side, you could memset the structure
> > where "Backing memory is pre-zeroed" comments are.
>
> This does not help :(
>
> Specifically:
> 1. Applied on top of staging-4.21:
> - build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/188ca612c51a57ef54b917c833eaeb59666994c9
> - test (fails): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688813821
> 2. Applied directly on top of 9f892f84c27970840d255aef7e2851b762967de7:
> - build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dd3dee35fc4fcd6ba284f3f5fa78b333c472888e
> - test (fails): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688832398
>
> Yet, parent of 9f892f84c27970840d255aef7e2851b762967de7 still passes:
> - build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dbcbbed4e9dc25faa211d359c2f04a9c70f087c9
> - test (pass): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688835512
>
> Hint about reading the above: I use two branches, one for building
> (branch "test") and one for running the test (branch "automation-uki-grub-only"),
> which takes build artifacts from the build branch. I do that to not
> rebase test scripts. It's a bit opaque, but the console log of the test
> contains "Latest ChangeSet" fields telling what actually was tested.
>
If it does not take long to test you could revert hunk by hunk
starting from xen/include/xlat.lst.
Frediano
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: domU crash when resuming from suspend - Xen 4.22
2026-07-19 21:04 ` Frediano Ziglio
@ 2026-07-19 22:50 ` Marek Marczykowski-Górecki
0 siblings, 0 replies; 10+ messages in thread
From: Marek Marczykowski-Górecki @ 2026-07-19 22:50 UTC (permalink / raw)
To: Frediano Ziglio; +Cc: xen-devel, Andrew Cooper
[-- Attachment #1: Type: text/plain, Size: 12708 bytes --]
On Sun, Jul 19, 2026 at 10:04:42PM +0100, Frediano Ziglio wrote:
> On Sun, 19 Jul 2026 at 20:40, Marek Marczykowski-Górecki
> <marmarek@invisiblethingslab.com> wrote:
> >
> > On Sun, Jul 19, 2026 at 08:31:28AM +0100, Frediano Ziglio wrote:
> > > On Sun, 19 Jul 2026 at 03:00, Marek Marczykowski-Górecki
> > > <marmarek@invisiblethingslab.com> wrote:
> > > >
> > > > On Sat, Jul 18, 2026 at 04:03:29PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > On Fri, Jul 17, 2026 at 09:22:47PM +0100, Frediano Ziglio wrote:
> > > > > > On Tue, 14 Jul 2026 at 18:12, Marek Marczykowski-Górecki
> > > > > > <marmarek@invisiblethingslab.com> wrote:
> > > > > > >
> > > > > > > On Fri, Jun 19, 2026 at 01:24:09PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I'm continuing testing 4.22 and found this:
> > > > > > > >
> > > > > > > > When a domU is suspended (even without host suspend) it crashes on
> > > > > > > > resume with:
> > > > > > > >
> > > > > > > > [ 24.374629] BUG: kernel NULL pointer dereference, address: 0000000000000020
> > > > > > > > [ 24.374647] #PF: supervisor read access in kernel mode
> > > > > > > > [ 24.374657] #PF: error_code(0x0000) - not-present page
> > > > > > > > [ 24.374668] PGD 0 P4D 0
> > > > > > > > [ 24.374675] Oops: Oops: 0000 [#1] SMP NOPTI
> > > > > > > > [ 24.374685] CPU: 0 UID: 0 PID: 19 Comm: migration/0 Not tainted 6.18.31-1.qubes.13.fc41.x86_64 #1 PREEMPT(full)
> > > > > > > > [ 24.374705] Stopper: multi_cpu_stop+0x0/0x140 <- __stop_cpus.constprop.0+0x6f/0xb0
> > > > > > > > [ 24.374723] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > > > > [ 24.374734] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > > > > [ 24.374764] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > > > > [ 24.374775] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > > > > [ 24.374790] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > > > > [ 24.374801] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > > > > [ 24.374813] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > > > > [ 24.374825] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > > > > [ 24.374837] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > > > > [ 24.374850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > > > [ 24.374860] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > > > > [ 24.374873] PKRU: 00000000
> > > > > > > > [ 24.374877] Call Trace:
> > > > > > > > [ 24.374884] <TASK>
> > > > > > > > [ 24.374891] ? __pfx_multi_cpu_stop+0x10/0x10
> > > > > > > > [ 24.374900] cpu_stopper_thread+0xa3/0x170
> > > > > > > > [ 24.374908] ? __pfx_smpboot_thread_fn+0x10/0x10
> > > > > > > > [ 24.374917] smpboot_thread_fn+0xf3/0x220
> > > > > > > > [ 24.374925] kthread+0xfc/0x240
> > > > > > > > [ 24.374933] ? __pfx_kthread+0x10/0x10
> > > > > > > > [ 24.374940] ? __pfx_kthread+0x10/0x10
> > > > > > > > [ 24.374949] ret_from_fork+0x158/0x170
> > > > > > > > [ 24.374957] ? __pfx_kthread+0x10/0x10
> > > > > > > > [ 24.374965] ret_from_fork_asm+0x1a/0x30
> > > > > > > > [ 24.374973] </TASK>
> > > > > > > > [ 24.374978] Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer snd soundcore cfg80211 rfkill vfat fat xenfs binfmt_misc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_ssram_telemetry intel_vsec polyval_clmulni ghash_clmulni_intel xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn fuse loop nfnetlink ip_tables overlay xen_blkfront
> > > > > > > > [ 24.375064] CR2: 0000000000000020
> > > > > > > > [ 24.375071] ---[ end trace 0000000000000000 ]---
> > > > > > > > [ 24.375080] RIP: 0010:multi_cpu_stop+0x63/0x140
> > > > > > > > [ 24.375089] Code: 49 0f a3 1c 24 41 0f 92 c6 c7 44 24 04 00 00 00 00 31 ed 49 8d 5f 24 eb 2d 41 83 fd 02 74 6e 41 83 fd 03 74 49 f0 ff 0b 75 13 <41> 8b 47 20 41 8b 77 10 83 c0 01 41 89 77 24 41 89 47 20 41 83 fd
> > > > > > > > [ 24.375115] RSP: 0018:ffffc900000abe30 EFLAGS: 00010046
> > > > > > > > [ 24.375124] RAX: 0000000000000000 RBX: ffffc90000a13de4 RCX: 0000000000000404
> > > > > > > > [ 24.375142] RDX: 0000000000000040 RSI: ffffffffffffffff RDI: 0000000000770ef0
> > > > > > > > [ 24.375155] RBP: 0000000000000002 R08: ffffc900000abc80 R09: ffffc900000abc8c
> > > > > > > > [ 24.375170] R10: 0000000000000000 R11: ffff88801e81cdc0 R12: ffffffff81835ac0
> > > > > > > > [ 24.375184] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > > > > > > > [ 24.375200] FS: 0000000000000000(0000) GS:ffff88809b369000(0000) knlGS:0000000000000000
> > > > > > > > [ 24.375214] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > > > > > [ 24.375226] CR2: 0000000000000020 CR3: 000000000d6a2002 CR4: 0000000000770ef0
> > > > > > > > [ 24.375241] PKRU: 00000000
> > > > > > > > [ 24.375247] Kernel panic - not syncing: Fatal exception
> > > > > > > > [ 25.417403] Shutting down cpus with NMI
> > > > > > > > [ 25.417447] Kernel Offset: disabled
> > > > > > > >
> > > > > > > > There is no change on the Linux kernel side (neither dom0 nor domU), the
> > > > > > > > only change is updating Xen 4.19.4 -> 4.22-rc2 (seems to be already
> > > > > > > > broken on -rc1 too). This happens for both HVM and PVH domains. Haven't
> > > > > > > > tried PV domU (but dom0 survives host suspend).
> > > > > > >
> > > > > > > Any ideas?
> > > > > > >
> > > > > >
> > > > > > Hi,
> > > > > > not many ideas but while testing my libxenguest changes I found
> > > > > > similar failures using xl create/save/restore. But I was testing more
> > > > > > PV guests.
> > > > > > Does 4.21 work ?
> > > > >
> > > > > It seems 4.21 also has this issue:
> > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414211523
> > > > > But 4.20 is okay:
> > > > > https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/15414227095
> > > > >
> > > > > I guess I can bisect it further.
> > > >
> > > > So, bisect says the first bad commit is:
> > > >
> > > > commit 9f892f84c27970840d255aef7e2851b762967de7
> > > > Author: Andrew Cooper <andrew.cooper3@citrix.com>
> > > > Date: Mon Dec 30 11:49:14 2024 +0000
> > > >
> > > > x86/domctl: Stop using XLAT_cpu_user_regs()
> > > >
> > > > My test isn't 100% reliable, but I retested parent of this commit a
> > > > couple of times and it was good every time.
> > > >
> > > > I wanted to test reverting it on top of 4.21, but there is a lot of
> > > > FRED-related work on top that makes such revert quite hard, to say the
> > > > least...
> > > >
> > >
> > > The generated XLAT_cpu_user_regs was
> > >
> > > #define XLAT_cpu_user_regs(_d_, _s_) do { \
> > > (_d_)->__DECL_REG_LO8(b) = (_s_)->__DECL_REG_LO8(b); \
> > > (_d_)->__DECL_REG_LO8(c) = (_s_)->__DECL_REG_LO8(c); \
> > > (_d_)->__DECL_REG_LO8(d) = (_s_)->__DECL_REG_LO8(d); \
> > > (_d_)->__DECL_REG_LO16(si) = (_s_)->__DECL_REG_LO16(si); \
> > > (_d_)->__DECL_REG_LO16(di) = (_s_)->__DECL_REG_LO16(di); \
> > > (_d_)->__DECL_REG_LO16(bp) = (_s_)->__DECL_REG_LO16(bp); \
> > > (_d_)->__DECL_REG_LO8(a) = (_s_)->__DECL_REG_LO8(a); \
> > > (_d_)->error_code = (_s_)->error_code; \
> > > (_d_)->entry_vector = (_s_)->entry_vector; \
> > > (_d_)->__DECL_REG_LO16(ip) = (_s_)->__DECL_REG_LO16(ip); \
> > > (_d_)->cs = (_s_)->cs; \
> > > (_d_)->saved_upcall_mask = (_s_)->saved_upcall_mask; \
> > > (_d_)->__DECL_REG_LO16(flags) = (_s_)->__DECL_REG_LO16(flags); \
> > > (_d_)->__DECL_REG_LO16(sp) = (_s_)->__DECL_REG_LO16(sp); \
> > > (_d_)->ss = (_s_)->ss; \
> > > (_d_)->es = (_s_)->es; \
> > > (_d_)->ds = (_s_)->ds; \
> > > (_d_)->fs = (_s_)->fs; \
> > > (_d_)->gs = (_s_)->gs; \
> > > } while (0)
> > >
> > > registers are copied, but additional fields (error_code, entry_vector,
> > > saved_upcall_mask).
> > > Maybe you could try to copy them again.
> > > Also, just to be on the safe side, you could memset the structure
> > > where "Backing memory is pre-zeroed" comments are.
> >
> > This does not help :(
> >
> > Specifically:
> > 1. Applied on top of staging-4.21:
> > - build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/188ca612c51a57ef54b917c833eaeb59666994c9
> > - test (fails): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688813821
> > 2. Applied directly on top of 9f892f84c27970840d255aef7e2851b762967de7:
> > - build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dd3dee35fc4fcd6ba284f3f5fa78b333c472888e
> > - test (fails): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688832398
> >
> > Yet, parent of 9f892f84c27970840d255aef7e2851b762967de7 still passes:
> > - build: https://gitlab.com/xen-project/people/marmarek/xen/-/commits/dbcbbed4e9dc25faa211d359c2f04a9c70f087c9
> > - test (pass): https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/2688835512
> >
> > Hint about reading the above: I use two branches, one for building
> > (branch "test") and one for running the test (branch "automation-uki-grub-only"),
> > which takes build artifacts from the build branch. I do that to not
> > rebase test scripts. It's a bit opaque, but the console log of the test
> > contains "Latest ChangeSet" fields telling what actually was tested.
> >
>
> If it does not take long to test you could revert hunk by hunk
> starting from xen/include/xlat.lst.
Ok, I found what is wrong. It isn't about XLAT branch, but the native
one, x86_64 has a couple more registers...
This fixes it:
diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c
index 9af833b6da59f66d58eac3369c30a89919d0d805..893ebaa13a6f934e1af026f9eb13b1d18f746214 100644
--- a/xen/arch/x86/domain.c
+++ b/xen/arch/x86/domain.c
@@ -1280,6 +1280,14 @@ int arch_set_info_guest(
if ( !compat )
{
+ v->arch.user_regs.r15 = c.nat->user_regs.r15;
+ v->arch.user_regs.r14 = c.nat->user_regs.r14;
+ v->arch.user_regs.r13 = c.nat->user_regs.r13;
+ v->arch.user_regs.r12 = c.nat->user_regs.r12;
+ v->arch.user_regs.r11 = c.nat->user_regs.r11;
+ v->arch.user_regs.r10 = c.nat->user_regs.r10;
+ v->arch.user_regs.r9 = c.nat->user_regs.r9;
+ v->arch.user_regs.r8 = c.nat->user_regs.r8;
v->arch.user_regs.rbx = c.nat->user_regs.rbx;
v->arch.user_regs.rcx = c.nat->user_regs.rcx;
v->arch.user_regs.rdx = c.nat->user_regs.rdx;
diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c
index b8bc0caaf205828f9254e03584709482f3ba38a6..40f4cb049ce3f84ef38a604254392548a41342de 100644
--- a/xen/arch/x86/domctl.c
+++ b/xen/arch/x86/domctl.c
@@ -1440,6 +1440,14 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_context_u c)
if ( !compat )
{
/* Backing memory is pre-zeroed. */
+ c.nat->user_regs.r15 = v->arch.user_regs.r15;
+ c.nat->user_regs.r14 = v->arch.user_regs.r14;
+ c.nat->user_regs.r13 = v->arch.user_regs.r13;
+ c.nat->user_regs.r12 = v->arch.user_regs.r12;
+ c.nat->user_regs.r11 = v->arch.user_regs.r11;
+ c.nat->user_regs.r10 = v->arch.user_regs.r10;
+ c.nat->user_regs.r9 = v->arch.user_regs.r9;
+ c.nat->user_regs.r8 = v->arch.user_regs.r8;
c.nat->user_regs.rbx = v->arch.user_regs.rbx;
c.nat->user_regs.rcx = v->arch.user_regs.rcx;
c.nat->user_regs.rdx = v->arch.user_regs.rdx;
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2026-07-19 22:51 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-19 11:24 domU crash when resuming from suspend - Xen 4.22 Marek Marczykowski-Górecki
2026-07-14 17:12 ` Marek Marczykowski-Górecki
2026-07-15 6:16 ` Jan Beulich
2026-07-17 20:22 ` Frediano Ziglio
2026-07-18 14:03 ` Marek Marczykowski-Górecki
2026-07-19 1:59 ` Marek Marczykowski-Górecki
2026-07-19 7:31 ` Frediano Ziglio
2026-07-19 19:40 ` Marek Marczykowski-Górecki
2026-07-19 21:04 ` Frediano Ziglio
2026-07-19 22:50 ` Marek Marczykowski-Górecki
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.