* issue with dom0_pvh on Xen 4.20
@ 2025-09-02 10:17 Manuel Bouyer
2025-09-02 10:44 ` Andrew Cooper
0 siblings, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 10:17 UTC (permalink / raw)
To: xen-devel
[-- Attachment #1: Type: text/plain, Size: 2768 bytes --]
Hello,
I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
The same NetBSD kernel works fine with Xen 4.18
The boot options are:
menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
and the full log from serial console is attached.
With 4.20 the boot fails with:
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 664kB init memory
(XEN) d0v0 Triple fault - invoking HVM shutdown action 1
(XEN) *** Dumping Dom0 vcpu#0 state: ***
(XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
(XEN) CPU: 7
(XEN) RIP: 0008:[<000000000020e268>]
(XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
(XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
(XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
(XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
(XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
(XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
(XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
(XEN) cr3: 0000000000000000 cr2: 0000000000000000
(XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
(XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
because of the triple fault the RIP above doens't point to the code.
I tracked it down to this code:
cmpl $0,%ecx ; /* zero-sized? */ \
je 2f ; \
pushl %ebp ; \
movl RELOC(nox_flag),%ebp ; \
1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
movl %eax,(%ebx) ; /* store phys addr */ \
addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
addl $PAGE_SIZE,%eax ; /* next phys page */ \
loop 1b ; \
popl %ebp ; \
2: ;
there are others pushl/popl before so I don't think that's the problem
(in fact the exact same fragment is called just before with different
inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
which would be 0x20e260
This is in the range:
(XEN) [0000000000100000, 0000000040068e77] (usable)
so I can't see why this would be a problem.
Any idea, including how to debug this further, welcome
thanks
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
[-- Attachment #2: 4.20.log.gz --]
[-- Type: application/octet-stream, Size: 8806 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 10:17 issue with dom0_pvh on Xen 4.20 Manuel Bouyer
@ 2025-09-02 10:44 ` Andrew Cooper
2025-09-02 10:56 ` Manuel Bouyer
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Cooper @ 2025-09-02 10:44 UTC (permalink / raw)
To: Manuel Bouyer, xen-devel
On 02/09/2025 11:17 am, Manuel Bouyer wrote:
> Hello,
> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
> The same NetBSD kernel works fine with Xen 4.18
>
> The boot options are:
> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>
> and the full log from serial console is attached.
>
> With 4.20 the boot fails with:
>
> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> (XEN) Freed 664kB init memory
> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
> (XEN) *** Dumping Dom0 vcpu#0 state: ***
> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
> (XEN) CPU: 7
> (XEN) RIP: 0008:[<000000000020e268>]
> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>
> because of the triple fault the RIP above doens't point to the code.
>
> I tracked it down to this code:
> cmpl $0,%ecx ; /* zero-sized? */ \
> je 2f ; \
> pushl %ebp ; \
> movl RELOC(nox_flag),%ebp ; \
> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
> movl %eax,(%ebx) ; /* store phys addr */ \
> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
> addl $PAGE_SIZE,%eax ; /* next phys page */ \
> loop 1b ; \
> popl %ebp ; \
> 2: ;
>
> there are others pushl/popl before so I don't think that's the problem
> (in fact the exact same fragment is called just before with different
> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
> which would be 0x20e260
> This is in the range:
> (XEN) [0000000000100000, 0000000040068e77] (usable)
> so I can't see why this would be a problem.
>
> Any idea, including how to debug this further, welcome
Even though triple fault's are aborts, they're generally accurate under
virt, so 0x20e268 is most likely where things die.
Your best bet is to instrument hvm_triple_fault(). e.g.
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 23bd7f078a1d..0f960576b3e6 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -1746,14 +1746,22 @@ void hvm_hlt(unsigned int eflags)
void hvm_triple_fault(void)
{
+ const struct cpu_user_regs *regs = guest_cpu_user_regs();
struct vcpu *v = current;
struct domain *d = v->domain;
u8 reason = d->arch.hvm.params[HVM_PARAM_TRIPLE_FAULT_REASON];
+ char insns[32];
gprintk(XENLOG_ERR,
"Triple fault - invoking HVM shutdown action %d\n",
reason);
vcpu_show_execution_state(v);
+
+ if ( copy_from_user_hvm(insns, _p(regs->rip), ARRAY_SIZE(insns)) )
+ printk("Guest code stream: %32ph\n", insns);
+ else
+ printk("Bad pagetables for %%rip\n");
+
domain_shutdown(d, reason);
}
will try and get you the instruction which finally caused the fault.
(compile tested only)
Given that 4.18 works and 4.20 doesn't, it's probably to do with the
initial memory map of the guest. Do you have the full logs of both boots?
Just for sanity sake (I don't know if it's going to be relevant or not),
boot with dom0=pvh,pf-fixup which just might highlight if we've got a
mixup with the memory map presented to the guest vs the system.
~Andrew
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 10:44 ` Andrew Cooper
@ 2025-09-02 10:56 ` Manuel Bouyer
2025-09-02 11:13 ` Andrew Cooper
` (2 more replies)
0 siblings, 3 replies; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 10:56 UTC (permalink / raw)
To: Andrew Cooper; +Cc: xen-devel
On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
> > Hello,
> > I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
> > The same NetBSD kernel works fine with Xen 4.18
> >
> > The boot options are:
> > menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
> >
> > and the full log from serial console is attached.
> >
> > With 4.20 the boot fails with:
> >
> > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> > (XEN) Freed 664kB init memory
> > (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
> > (XEN) *** Dumping Dom0 vcpu#0 state: ***
> > (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
> > (XEN) CPU: 7
> > (XEN) RIP: 0008:[<000000000020e268>]
> > (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
> > (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
> > (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
> > (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
> > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> > (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> > (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
> > (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> > (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> > (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
> >
> > because of the triple fault the RIP above doens't point to the code.
> >
> > I tracked it down to this code:
> > cmpl $0,%ecx ; /* zero-sized? */ \
> > je 2f ; \
> > pushl %ebp ; \
> > movl RELOC(nox_flag),%ebp ; \
> > 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
> > movl %eax,(%ebx) ; /* store phys addr */ \
> > addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
> > addl $PAGE_SIZE,%eax ; /* next phys page */ \
> > loop 1b ; \
> > popl %ebp ; \
> > 2: ;
> >
> > there are others pushl/popl before so I don't think that's the problem
> > (in fact the exact same fragment is called just before with different
> > inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
> > which would be 0x20e260
> > This is in the range:
> > (XEN) [0000000000100000, 0000000040068e77] (usable)
> > so I can't see why this would be a problem.
> >
> > Any idea, including how to debug this further, welcome
>
> Even though triple fault's are aborts, they're generally accurate under
> virt, so 0x20e268 is most likely where things die.
but that's the RIP of the last fault, not the first one, right ?
0x20e268 isn't in the text segment of the kernel, my guess is that the
first fault triggers an exception, but the exeption handler isn't set up yet
so we end up jumping to some random value.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 10:56 ` Manuel Bouyer
@ 2025-09-02 11:13 ` Andrew Cooper
2025-09-02 11:23 ` Manuel Bouyer
2025-09-02 12:00 ` Jan Beulich
2025-09-02 12:22 ` Juergen Gross
2 siblings, 1 reply; 29+ messages in thread
From: Andrew Cooper @ 2025-09-02 11:13 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: xen-devel
On 02/09/2025 11:56 am, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>> Hello,
>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>> The same NetBSD kernel works fine with Xen 4.18
>>>
>>> The boot options are:
>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>
>>> and the full log from serial console is attached.
>>>
>>> With 4.20 the boot fails with:
>>>
>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>> (XEN) Freed 664kB init memory
>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>> (XEN) CPU: 7
>>> (XEN) RIP: 0008:[<000000000020e268>]
>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>
>>> because of the triple fault the RIP above doens't point to the code.
>>>
>>> I tracked it down to this code:
>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>> je 2f ; \
>>> pushl %ebp ; \
>>> movl RELOC(nox_flag),%ebp ; \
>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>> loop 1b ; \
>>> popl %ebp ; \
>>> 2: ;
>>>
>>> there are others pushl/popl before so I don't think that's the problem
>>> (in fact the exact same fragment is called just before with different
>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>> which would be 0x20e260
>>> This is in the range:
>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>> so I can't see why this would be a problem.
>>>
>>> Any idea, including how to debug this further, welcome
>> Even though triple fault's are aborts, they're generally accurate under
>> virt, so 0x20e268 is most likely where things die.
> but that's the RIP of the last fault, not the first one, right ?
> 0x20e268 isn't in the text segment of the kernel, my guess is that the
> first fault triggers an exception, but the exeption handler isn't set up yet
> so we end up jumping to some random value.
Double and Triple faults occur when trying to deliver an exception
generates an exception. So while multiple faults are involved, only one
instruction typically is.
Is this an Intel or an AMD system? One thing virt can do is break apart
a triple fault, but the logic to do so is vendor specific.
~Andrew
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 11:13 ` Andrew Cooper
@ 2025-09-02 11:23 ` Manuel Bouyer
2025-09-02 12:14 ` Andrew Cooper
0 siblings, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 11:23 UTC (permalink / raw)
To: Andrew Cooper; +Cc: xen-devel
On Tue, Sep 02, 2025 at 12:13:27PM +0100, Andrew Cooper wrote:
> On 02/09/2025 11:56 am, Manuel Bouyer wrote:
> > On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
> >> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
> >>> Hello,
> >>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
> >>> The same NetBSD kernel works fine with Xen 4.18
> >>>
> >>> The boot options are:
> >>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
> >>>
> >>> and the full log from serial console is attached.
> >>>
> >>> With 4.20 the boot fails with:
> >>>
> >>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> >>> (XEN) Freed 664kB init memory
> >>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
> >>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
> >>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
> >>> (XEN) CPU: 7
> >>> (XEN) RIP: 0008:[<000000000020e268>]
> >>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
> >>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
> >>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
> >>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
> >>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> >>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> >>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
> >>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> >>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> >>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
> >>>
> >>> because of the triple fault the RIP above doens't point to the code.
> >>>
> >>> I tracked it down to this code:
> >>> cmpl $0,%ecx ; /* zero-sized? */ \
> >>> je 2f ; \
> >>> pushl %ebp ; \
> >>> movl RELOC(nox_flag),%ebp ; \
> >>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
> >>> movl %eax,(%ebx) ; /* store phys addr */ \
> >>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
> >>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
> >>> loop 1b ; \
> >>> popl %ebp ; \
> >>> 2: ;
> >>>
> >>> there are others pushl/popl before so I don't think that's the problem
> >>> (in fact the exact same fragment is called just before with different
> >>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
> >>> which would be 0x20e260
> >>> This is in the range:
> >>> (XEN) [0000000000100000, 0000000040068e77] (usable)
> >>> so I can't see why this would be a problem.
> >>>
> >>> Any idea, including how to debug this further, welcome
> >> Even though triple fault's are aborts, they're generally accurate under
> >> virt, so 0x20e268 is most likely where things die.
> > but that's the RIP of the last fault, not the first one, right ?
> > 0x20e268 isn't in the text segment of the kernel, my guess is that the
> > first fault triggers an exception, but the exeption handler isn't set up yet
> > so we end up jumping to some random value.
>
> Double and Triple faults occur when trying to deliver an exception
> generates an exception. So while multiple faults are involved, only one
> instruction typically is.
>
> Is this an Intel or an AMD system? One thing virt can do is break apart
> a triple fault, but the logic to do so is vendor specific.
it's an old intel system:
cpu0: "Intel(R) Xeon(R) CPU X5650 @ 2.67GHz"
cpu0: Intel Xeon 36xx & 56xx, i7, i5 and i3 (686-class), 2667.30 MHz
cpu0: family 0x6 model 0x2c stepping 0x2 (id 0x206c2)
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 10:56 ` Manuel Bouyer
2025-09-02 11:13 ` Andrew Cooper
@ 2025-09-02 12:00 ` Jan Beulich
2025-09-02 12:10 ` Manuel Bouyer
2025-09-02 12:22 ` Juergen Gross
2 siblings, 1 reply; 29+ messages in thread
From: Jan Beulich @ 2025-09-02 12:00 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: xen-devel, Andrew Cooper
On 02.09.2025 12:56, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>> Hello,
>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>> The same NetBSD kernel works fine with Xen 4.18
>>>
>>> The boot options are:
>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>
>>> and the full log from serial console is attached.
>>>
>>> With 4.20 the boot fails with:
>>>
>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>> (XEN) Freed 664kB init memory
>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>> (XEN) CPU: 7
>>> (XEN) RIP: 0008:[<000000000020e268>]
>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>
>>> because of the triple fault the RIP above doens't point to the code.
>>>
>>> I tracked it down to this code:
>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>> je 2f ; \
>>> pushl %ebp ; \
>>> movl RELOC(nox_flag),%ebp ; \
>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>> loop 1b ; \
>>> popl %ebp ; \
>>> 2: ;
>>>
>>> there are others pushl/popl before so I don't think that's the problem
>>> (in fact the exact same fragment is called just before with different
>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>> which would be 0x20e260
>>> This is in the range:
>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>> so I can't see why this would be a problem.
>>>
>>> Any idea, including how to debug this further, welcome
>>
>> Even though triple fault's are aborts, they're generally accurate under
>> virt, so 0x20e268 is most likely where things die.
>
> but that's the RIP of the last fault, not the first one, right ?
> 0x20e268 isn't in the text segment of the kernel, my guess is that the
> first fault triggers an exception, but the exeption handler isn't set up yet
> so we end up jumping to some random value.
Can you perhaps check this guess against the %esp value seen? From the
hypervisor's triple fault handling, you may want to actually log stack
contents as well (in addition to what Andrew suggested), assuming %esp
looks sane to you.
Jan
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:00 ` Jan Beulich
@ 2025-09-02 12:10 ` Manuel Bouyer
2025-09-02 12:54 ` Jan Beulich
0 siblings, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 12:10 UTC (permalink / raw)
To: Jan Beulich; +Cc: xen-devel, Andrew Cooper
On Tue, Sep 02, 2025 at 02:00:35PM +0200, Jan Beulich wrote:
> On 02.09.2025 12:56, Manuel Bouyer wrote:
> > On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
> >> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
> >>> Hello,
> >>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
> >>> The same NetBSD kernel works fine with Xen 4.18
> >>>
> >>> The boot options are:
> >>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
> >>>
> >>> and the full log from serial console is attached.
> >>>
> >>> With 4.20 the boot fails with:
> >>>
> >>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> >>> (XEN) Freed 664kB init memory
> >>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
> >>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
> >>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
> >>> (XEN) CPU: 7
> >>> (XEN) RIP: 0008:[<000000000020e268>]
> >>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
> >>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
> >>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
> >>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
> >>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> >>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> >>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
> >>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> >>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> >>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
> >>>
> >>> because of the triple fault the RIP above doens't point to the code.
> >>>
> >>> I tracked it down to this code:
> >>> cmpl $0,%ecx ; /* zero-sized? */ \
> >>> je 2f ; \
> >>> pushl %ebp ; \
> >>> movl RELOC(nox_flag),%ebp ; \
> >>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
> >>> movl %eax,(%ebx) ; /* store phys addr */ \
> >>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
> >>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
> >>> loop 1b ; \
> >>> popl %ebp ; \
> >>> 2: ;
> >>>
> >>> there are others pushl/popl before so I don't think that's the problem
> >>> (in fact the exact same fragment is called just before with different
> >>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
> >>> which would be 0x20e260
> >>> This is in the range:
> >>> (XEN) [0000000000100000, 0000000040068e77] (usable)
> >>> so I can't see why this would be a problem.
> >>>
> >>> Any idea, including how to debug this further, welcome
> >>
> >> Even though triple fault's are aborts, they're generally accurate under
> >> virt, so 0x20e268 is most likely where things die.
> >
> > but that's the RIP of the last fault, not the first one, right ?
> > 0x20e268 isn't in the text segment of the kernel, my guess is that the
> > first fault triggers an exception, but the exeption handler isn't set up yet
> > so we end up jumping to some random value.
>
> Can you perhaps check this guess against the %esp value seen? From the
> hypervisor's triple fault handling, you may want to actually log stack
> contents as well (in addition to what Andrew suggested), assuming %esp
> looks sane to you.
%esp is sane. I forgot to mention, this happens very early, while we're still
in 32bits real mode. No function call did happen at this point, and on the
stack there's only one 32bit value: the %ebp that we just pushed
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 11:23 ` Manuel Bouyer
@ 2025-09-02 12:14 ` Andrew Cooper
2025-09-02 12:24 ` Manuel Bouyer
0 siblings, 1 reply; 29+ messages in thread
From: Andrew Cooper @ 2025-09-02 12:14 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: xen-devel
On 02/09/2025 12:23 pm, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 12:13:27PM +0100, Andrew Cooper wrote:
>> On 02/09/2025 11:56 am, Manuel Bouyer wrote:
>>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>>>> Hello,
>>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>>>> The same NetBSD kernel works fine with Xen 4.18
>>>>>
>>>>> The boot options are:
>>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>>>
>>>>> and the full log from serial console is attached.
>>>>>
>>>>> With 4.20 the boot fails with:
>>>>>
>>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>>>> (XEN) Freed 664kB init memory
>>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>>>> (XEN) CPU: 7
>>>>> (XEN) RIP: 0008:[<000000000020e268>]
>>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>>>
>>>>> because of the triple fault the RIP above doens't point to the code.
>>>>>
>>>>> I tracked it down to this code:
>>>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>>>> je 2f ; \
>>>>> pushl %ebp ; \
>>>>> movl RELOC(nox_flag),%ebp ; \
>>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>>>> loop 1b ; \
>>>>> popl %ebp ; \
>>>>> 2: ;
>>>>>
>>>>> there are others pushl/popl before so I don't think that's the problem
>>>>> (in fact the exact same fragment is called just before with different
>>>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>>>> which would be 0x20e260
>>>>> This is in the range:
>>>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>>>> so I can't see why this would be a problem.
>>>>>
>>>>> Any idea, including how to debug this further, welcome
>>>> Even though triple fault's are aborts, they're generally accurate under
>>>> virt, so 0x20e268 is most likely where things die.
>>> but that's the RIP of the last fault, not the first one, right ?
>>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
>>> first fault triggers an exception, but the exeption handler isn't set up yet
>>> so we end up jumping to some random value.
>> Double and Triple faults occur when trying to deliver an exception
>> generates an exception. So while multiple faults are involved, only one
>> instruction typically is.
>>
>> Is this an Intel or an AMD system? One thing virt can do is break apart
>> a triple fault, but the logic to do so is vendor specific.
> it's an old intel system:
> cpu0: "Intel(R) Xeon(R) CPU X5650 @ 2.67GHz"
> cpu0: Intel Xeon 36xx & 56xx, i7, i5 and i3 (686-class), 2667.30 MHz
> cpu0: family 0x6 model 0x2c stepping 0x2 (id 0x206c2)
>
Hmm. Westmere EP. Are you running with EPT active, or with Shadow Paging?
~Andrew
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 10:56 ` Manuel Bouyer
2025-09-02 11:13 ` Andrew Cooper
2025-09-02 12:00 ` Jan Beulich
@ 2025-09-02 12:22 ` Juergen Gross
2025-09-02 12:28 ` Juergen Gross
2025-09-02 12:28 ` Manuel Bouyer
2 siblings, 2 replies; 29+ messages in thread
From: Juergen Gross @ 2025-09-02 12:22 UTC (permalink / raw)
To: Manuel Bouyer, Andrew Cooper; +Cc: xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 3757 bytes --]
On 02.09.25 12:56, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>> Hello,
>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>> The same NetBSD kernel works fine with Xen 4.18
>>>
>>> The boot options are:
>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>
>>> and the full log from serial console is attached.
>>>
>>> With 4.20 the boot fails with:
>>>
>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>> (XEN) Freed 664kB init memory
>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>> (XEN) CPU: 7
>>> (XEN) RIP: 0008:[<000000000020e268>]
>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>
>>> because of the triple fault the RIP above doens't point to the code.
>>>
>>> I tracked it down to this code:
>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>> je 2f ; \
>>> pushl %ebp ; \
>>> movl RELOC(nox_flag),%ebp ; \
>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>> loop 1b ; \
>>> popl %ebp ; \
>>> 2: ;
>>>
>>> there are others pushl/popl before so I don't think that's the problem
>>> (in fact the exact same fragment is called just before with different
>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>> which would be 0x20e260
>>> This is in the range:
>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>> so I can't see why this would be a problem.
>>>
>>> Any idea, including how to debug this further, welcome
>>
>> Even though triple fault's are aborts, they're generally accurate under
>> virt, so 0x20e268 is most likely where things die.
>
> but that's the RIP of the last fault, not the first one, right ?
> 0x20e268 isn't in the text segment of the kernel, my guess is that the
> first fault triggers an exception, but the exeption handler isn't set up yet
> so we end up jumping to some random value.
>
What puzzles me is that:
- %cr2 is 0, so probably the first fault wasn't a page fault
- RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
Could it be the code has been moved to this location, or is about to
be moved away afterwards?
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:14 ` Andrew Cooper
@ 2025-09-02 12:24 ` Manuel Bouyer
2025-09-02 12:52 ` Andrew Cooper
0 siblings, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 12:24 UTC (permalink / raw)
To: Andrew Cooper; +Cc: xen-devel
On Tue, Sep 02, 2025 at 01:14:29PM +0100, Andrew Cooper wrote:
> On 02/09/2025 12:23 pm, Manuel Bouyer wrote:
> > On Tue, Sep 02, 2025 at 12:13:27PM +0100, Andrew Cooper wrote:
> >> On 02/09/2025 11:56 am, Manuel Bouyer wrote:
> >>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
> >>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
> >>>>> Hello,
> >>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
> >>>>> The same NetBSD kernel works fine with Xen 4.18
> >>>>>
> >>>>> The boot options are:
> >>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
> >>>>>
> >>>>> and the full log from serial console is attached.
> >>>>>
> >>>>> With 4.20 the boot fails with:
> >>>>>
> >>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> >>>>> (XEN) Freed 664kB init memory
> >>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
> >>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
> >>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
> >>>>> (XEN) CPU: 7
> >>>>> (XEN) RIP: 0008:[<000000000020e268>]
> >>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
> >>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
> >>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
> >>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
> >>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> >>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> >>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
> >>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> >>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> >>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
> >>>>>
> >>>>> because of the triple fault the RIP above doens't point to the code.
> >>>>>
> >>>>> I tracked it down to this code:
> >>>>> cmpl $0,%ecx ; /* zero-sized? */ \
> >>>>> je 2f ; \
> >>>>> pushl %ebp ; \
> >>>>> movl RELOC(nox_flag),%ebp ; \
> >>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
> >>>>> movl %eax,(%ebx) ; /* store phys addr */ \
> >>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
> >>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
> >>>>> loop 1b ; \
> >>>>> popl %ebp ; \
> >>>>> 2: ;
> >>>>>
> >>>>> there are others pushl/popl before so I don't think that's the problem
> >>>>> (in fact the exact same fragment is called just before with different
> >>>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
> >>>>> which would be 0x20e260
> >>>>> This is in the range:
> >>>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
> >>>>> so I can't see why this would be a problem.
> >>>>>
> >>>>> Any idea, including how to debug this further, welcome
> >>>> Even though triple fault's are aborts, they're generally accurate under
> >>>> virt, so 0x20e268 is most likely where things die.
> >>> but that's the RIP of the last fault, not the first one, right ?
> >>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
> >>> first fault triggers an exception, but the exeption handler isn't set up yet
> >>> so we end up jumping to some random value.
> >> Double and Triple faults occur when trying to deliver an exception
> >> generates an exception. So while multiple faults are involved, only one
> >> instruction typically is.
> >>
> >> Is this an Intel or an AMD system? One thing virt can do is break apart
> >> a triple fault, but the logic to do so is vendor specific.
> > it's an old intel system:
> > cpu0: "Intel(R) Xeon(R) CPU X5650 @ 2.67GHz"
> > cpu0: Intel Xeon 36xx & 56xx, i7, i5 and i3 (686-class), 2667.30 MHz
> > cpu0: family 0x6 model 0x2c stepping 0x2 (id 0x206c2)
> >
>
> Hmm. Westmere EP. Are you running with EPT active, or with Shadow Paging?
How do I know ?
Note that the same problem shows up on much newer systems: an i9, and a
Xeon W-2223. Both boots fine with the same NetBSD kernel and Xen 4.18 or 4.15.
I'm using this old Xeon for debug because this one has a serial console.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:22 ` Juergen Gross
@ 2025-09-02 12:28 ` Juergen Gross
2025-09-02 12:49 ` Manuel Bouyer
2025-09-02 13:41 ` Manuel Bouyer
2025-09-02 12:28 ` Manuel Bouyer
1 sibling, 2 replies; 29+ messages in thread
From: Juergen Gross @ 2025-09-02 12:28 UTC (permalink / raw)
To: Manuel Bouyer, Andrew Cooper; +Cc: xen-devel
[-- Attachment #1.1.1: Type: text/plain, Size: 4591 bytes --]
On 02.09.25 14:22, Juergen Gross wrote:
> On 02.09.25 12:56, Manuel Bouyer wrote:
>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>>> Hello,
>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>>> The same NetBSD kernel works fine with Xen 4.18
>>>>
>>>> The boot options are:
>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0
>>>> root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1
>>>> com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64
>>>> sync_console=1 dom0=pvh
>>>>
>>>> and the full log from serial console is attached.
>>>>
>>>> With 4.20 the boot fails with:
>>>>
>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>>> (XEN) Freed 664kB init memory
>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>>> (XEN) CPU: 7
>>>> (XEN) RIP: 0008:[<000000000020e268>]
>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>>
>>>> because of the triple fault the RIP above doens't point to the code.
>>>>
>>>> I tracked it down to this code:
>>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>>> je 2f ; \
>>>> pushl %ebp ; \
>>>> movl RELOC(nox_flag),%ebp ; \
>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>>> loop 1b ; \
>>>> popl %ebp ; \
>>>> 2: ;
>>>>
>>>> there are others pushl/popl before so I don't think that's the problem
>>>> (in fact the exact same fragment is called just before with different
>>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>>> which would be 0x20e260
>>>> This is in the range:
>>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>>> so I can't see why this would be a problem.
>>>>
>>>> Any idea, including how to debug this further, welcome
>>>
>>> Even though triple fault's are aborts, they're generally accurate under
>>> virt, so 0x20e268 is most likely where things die.
>>
>> but that's the RIP of the last fault, not the first one, right ?
>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
>> first fault triggers an exception, but the exeption handler isn't set up yet
>> so we end up jumping to some random value.
>>
>
> What puzzles me is that:
>
> - %cr2 is 0, so probably the first fault wasn't a page fault
> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
>
> Could it be the code has been moved to this location, or is about to
> be moved away afterwards?
And indeed: from the full boot log I can see:
(XEN) virt_base = 0x0
(XEN) elf_paddr_offset = 0x0
(XEN) virt_offset = 0x0
(XEN) virt_kstart = 0x200000
(XEN) virt_kend = 0x17bab90
(XEN) virt_entry = 0x20e4d0
So virt_kentry is very near to the RIP.
Juergen
[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:22 ` Juergen Gross
2025-09-02 12:28 ` Juergen Gross
@ 2025-09-02 12:28 ` Manuel Bouyer
2025-09-02 12:45 ` Jan Beulich
1 sibling, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 12:28 UTC (permalink / raw)
To: Juergen Gross; +Cc: Andrew Cooper, xen-devel
On Tue, Sep 02, 2025 at 02:22:29PM +0200, Juergen Gross wrote:
> On 02.09.25 12:56, Manuel Bouyer wrote:
> > On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
> > > On 02/09/2025 11:17 am, Manuel Bouyer wrote:
> > > > Hello,
> > > > I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
> > > > The same NetBSD kernel works fine with Xen 4.18
> > > >
> > > > The boot options are:
> > > > menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
> > > >
> > > > and the full log from serial console is attached.
> > > >
> > > > With 4.20 the boot fails with:
> > > >
> > > > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> > > > (XEN) Freed 664kB init memory
> > > > (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
> > > > (XEN) *** Dumping Dom0 vcpu#0 state: ***
> > > > (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
> > > > (XEN) CPU: 7
> > > > (XEN) RIP: 0008:[<000000000020e268>]
> > > > (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
> > > > (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
> > > > (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
> > > > (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
> > > > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> > > > (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> > > > (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
> > > > (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> > > > (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> > > > (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
> > > >
> > > > because of the triple fault the RIP above doens't point to the code.
> > > >
> > > > I tracked it down to this code:
> > > > cmpl $0,%ecx ; /* zero-sized? */ \
> > > > je 2f ; \
> > > > pushl %ebp ; \
> > > > movl RELOC(nox_flag),%ebp ; \
> > > > 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
> > > > movl %eax,(%ebx) ; /* store phys addr */ \
> > > > addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
> > > > addl $PAGE_SIZE,%eax ; /* next phys page */ \
> > > > loop 1b ; \
> > > > popl %ebp ; \
> > > > 2: ;
> > > >
> > > > there are others pushl/popl before so I don't think that's the problem
> > > > (in fact the exact same fragment is called just before with different
> > > > inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
> > > > which would be 0x20e260
> > > > This is in the range:
> > > > (XEN) [0000000000100000, 0000000040068e77] (usable)
> > > > so I can't see why this would be a problem.
> > > >
> > > > Any idea, including how to debug this further, welcome
> > >
> > > Even though triple fault's are aborts, they're generally accurate under
> > > virt, so 0x20e268 is most likely where things die.
> >
> > but that's the RIP of the last fault, not the first one, right ?
> > 0x20e268 isn't in the text segment of the kernel, my guess is that the
> > first fault triggers an exception, but the exeption handler isn't set up yet
> > so we end up jumping to some random value.
> >
>
> What puzzles me is that:
>
> - %cr2 is 0, so probably the first fault wasn't a page fault
AFAIK it can't be as we're still in real mode
> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
>
> Could it be the code has been moved to this location, or is about to
> be moved away afterwards?
No. RIP shouldn't end up there in any way. the assembly code is quite simple,
it's just a loop and I'm quite confident that we did enter the loop with
sane values
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:28 ` Manuel Bouyer
@ 2025-09-02 12:45 ` Jan Beulich
2025-09-02 12:54 ` Manuel Bouyer
0 siblings, 1 reply; 29+ messages in thread
From: Jan Beulich @ 2025-09-02 12:45 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On 02.09.2025 14:28, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 02:22:29PM +0200, Juergen Gross wrote:
>> On 02.09.25 12:56, Manuel Bouyer wrote:
>>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>>>> Hello,
>>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>>>> The same NetBSD kernel works fine with Xen 4.18
>>>>>
>>>>> The boot options are:
>>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>>>
>>>>> and the full log from serial console is attached.
>>>>>
>>>>> With 4.20 the boot fails with:
>>>>>
>>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>>>> (XEN) Freed 664kB init memory
>>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>>>> (XEN) CPU: 7
>>>>> (XEN) RIP: 0008:[<000000000020e268>]
>>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>>>
>>>>> because of the triple fault the RIP above doens't point to the code.
>>>>>
>>>>> I tracked it down to this code:
>>>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>>>> je 2f ; \
>>>>> pushl %ebp ; \
>>>>> movl RELOC(nox_flag),%ebp ; \
>>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>>>> loop 1b ; \
>>>>> popl %ebp ; \
>>>>> 2: ;
>>>>>
>>>>> there are others pushl/popl before so I don't think that's the problem
>>>>> (in fact the exact same fragment is called just before with different
>>>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>>>> which would be 0x20e260
>>>>> This is in the range:
>>>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>>>> so I can't see why this would be a problem.
>>>>>
>>>>> Any idea, including how to debug this further, welcome
>>>>
>>>> Even though triple fault's are aborts, they're generally accurate under
>>>> virt, so 0x20e268 is most likely where things die.
>>>
>>> but that's the RIP of the last fault, not the first one, right ?
>>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
>>> first fault triggers an exception, but the exeption handler isn't set up yet
>>> so we end up jumping to some random value.
>>>
>>
>> What puzzles me is that:
>>
>> - %cr2 is 0, so probably the first fault wasn't a page fault
>
> AFAIK it can't be as we're still in real mode
It's protected mode, but with paging still off.
>> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
>>
>> Could it be the code has been moved to this location, or is about to
>> be moved away afterwards?
>
> No. RIP shouldn't end up there in any way. the assembly code is quite simple,
> it's just a loop and I'm quite confident that we did enter the loop with
> sane values
Yet Jürgen has a point - entry point and what is being modified are on the
same page (and despite paging still being off, you writing page tables here
makes pages a relevant unit). Considering
- entry point @ 0x20e4d0
- %ecx = 0xdfeb7
- %ebx = 0x20e260
the loop continuing a little further will overwrite the entry point code.
And with the entry point not at an even (e.g page-aligned) address, other
code (like the one here) could conceivably live immediately ahead of it.
(Of course this overwriting may be intentional, but it looks suspicious in
this context.)
Jan
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:28 ` Juergen Gross
@ 2025-09-02 12:49 ` Manuel Bouyer
2025-09-02 13:41 ` Manuel Bouyer
1 sibling, 0 replies; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 12:49 UTC (permalink / raw)
To: Juergen Gross; +Cc: Andrew Cooper, xen-devel
On Tue, Sep 02, 2025 at 02:28:27PM +0200, Juergen Gross wrote:
> On 02.09.25 14:22, Juergen Gross wrote:
> > On 02.09.25 12:56, Manuel Bouyer wrote:
> > > On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
> > > > On 02/09/2025 11:17 am, Manuel Bouyer wrote:
> > > > > Hello,
> > > > > I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
> > > > > The same NetBSD kernel works fine with Xen 4.18
> > > > >
> > > > > The boot options are:
> > > > > menu=Boot netbsd-current PVH Xen420:dev hd0f:;load
> > > > > /netbsd-PVH console=com0 root=wd0f; multiboot
> > > > > /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1
> > > > > loglvl=all guest_loglvl=all gnttab_max_nr_frames=64
> > > > > sync_console=1 dom0=pvh
> > > > >
> > > > > and the full log from serial console is attached.
> > > > >
> > > > > With 4.20 the boot fails with:
> > > > >
> > > > > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> > > > > (XEN) Freed 664kB init memory
> > > > > (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
> > > > > (XEN) *** Dumping Dom0 vcpu#0 state: ***
> > > > > (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
> > > > > (XEN) CPU: 7
> > > > > (XEN) RIP: 0008:[<000000000020e268>]
> > > > > (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
> > > > > (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
> > > > > (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
> > > > > (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
> > > > > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
> > > > > (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
> > > > > (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
> > > > > (XEN) cr3: 0000000000000000 cr2: 0000000000000000
> > > > > (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
> > > > > (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
> > > > >
> > > > > because of the triple fault the RIP above doens't point to the code.
> > > > >
> > > > > I tracked it down to this code:
> > > > > cmpl $0,%ecx ; /* zero-sized? */ \
> > > > > je 2f ; \
> > > > > pushl %ebp ; \
> > > > > movl RELOC(nox_flag),%ebp ; \
> > > > > 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
> > > > > movl %eax,(%ebx) ; /* store phys addr */ \
> > > > > addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
> > > > > addl $PAGE_SIZE,%eax ; /* next phys page */ \
> > > > > loop 1b ; \
> > > > > popl %ebp ; \
> > > > > 2: ;
> > > > >
> > > > > there are others pushl/popl before so I don't think that's the problem
> > > > > (in fact the exact same fragment is called just before with different
> > > > > inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
> > > > > which would be 0x20e260
> > > > > This is in the range:
> > > > > (XEN) [0000000000100000, 0000000040068e77] (usable)
> > > > > so I can't see why this would be a problem.
> > > > >
> > > > > Any idea, including how to debug this further, welcome
> > > >
> > > > Even though triple fault's are aborts, they're generally accurate under
> > > > virt, so 0x20e268 is most likely where things die.
> > >
> > > but that's the RIP of the last fault, not the first one, right ?
> > > 0x20e268 isn't in the text segment of the kernel, my guess is that the
> > > first fault triggers an exception, but the exeption handler isn't set up yet
> > > so we end up jumping to some random value.
> > >
> >
> > What puzzles me is that:
> >
> > - %cr2 is 0, so probably the first fault wasn't a page fault
> > - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
> >
> > Could it be the code has been moved to this location, or is about to
> > be moved away afterwards?
>
> And indeed: from the full boot log I can see:
>
> (XEN) virt_base = 0x0
> (XEN) elf_paddr_offset = 0x0
> (XEN) virt_offset = 0x0
> (XEN) virt_kstart = 0x200000
> (XEN) virt_kend = 0x17bab90
> (XEN) virt_entry = 0x20e4d0
>
> So virt_kentry is very near to the RIP.
thanks, I missed this point and got my math off by 0x100000 it seems.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:24 ` Manuel Bouyer
@ 2025-09-02 12:52 ` Andrew Cooper
2025-09-02 12:55 ` Andrew Cooper
2025-09-02 12:56 ` Manuel Bouyer
0 siblings, 2 replies; 29+ messages in thread
From: Andrew Cooper @ 2025-09-02 12:52 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: xen-devel
On 02/09/2025 1:24 pm, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 01:14:29PM +0100, Andrew Cooper wrote:
>> On 02/09/2025 12:23 pm, Manuel Bouyer wrote:
>>> On Tue, Sep 02, 2025 at 12:13:27PM +0100, Andrew Cooper wrote:
>>>> On 02/09/2025 11:56 am, Manuel Bouyer wrote:
>>>>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>>>>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>>>>>> Hello,
>>>>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>>>>>> The same NetBSD kernel works fine with Xen 4.18
>>>>>>>
>>>>>>> The boot options are:
>>>>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>>>>>
>>>>>>> and the full log from serial console is attached.
>>>>>>>
>>>>>>> With 4.20 the boot fails with:
>>>>>>>
>>>>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>>>>>> (XEN) Freed 664kB init memory
>>>>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>>>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>>>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>>>>>> (XEN) CPU: 7
>>>>>>> (XEN) RIP: 0008:[<000000000020e268>]
>>>>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>>>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>>>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>>>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>>>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>>>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>>>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>>>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>>>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>>>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>>>>>
>>>>>>> because of the triple fault the RIP above doens't point to the code.
>>>>>>>
>>>>>>> I tracked it down to this code:
>>>>>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>>>>>> je 2f ; \
>>>>>>> pushl %ebp ; \
>>>>>>> movl RELOC(nox_flag),%ebp ; \
>>>>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>>>>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>>>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>>>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>>>>>> loop 1b ; \
>>>>>>> popl %ebp ; \
>>>>>>> 2: ;
>>>>>>>
>>>>>>> there are others pushl/popl before so I don't think that's the problem
>>>>>>> (in fact the exact same fragment is called just before with different
>>>>>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>>>>>> which would be 0x20e260
>>>>>>> This is in the range:
>>>>>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>>>>>> so I can't see why this would be a problem.
>>>>>>>
>>>>>>> Any idea, including how to debug this further, welcome
>>>>>> Even though triple fault's are aborts, they're generally accurate under
>>>>>> virt, so 0x20e268 is most likely where things die.
>>>>> but that's the RIP of the last fault, not the first one, right ?
>>>>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
>>>>> first fault triggers an exception, but the exeption handler isn't set up yet
>>>>> so we end up jumping to some random value.
>>>> Double and Triple faults occur when trying to deliver an exception
>>>> generates an exception. So while multiple faults are involved, only one
>>>> instruction typically is.
>>>>
>>>> Is this an Intel or an AMD system? One thing virt can do is break apart
>>>> a triple fault, but the logic to do so is vendor specific.
>>> it's an old intel system:
>>> cpu0: "Intel(R) Xeon(R) CPU X5650 @ 2.67GHz"
>>> cpu0: Intel Xeon 36xx & 56xx, i7, i5 and i3 (686-class), 2667.30 MHz
>>> cpu0: family 0x6 model 0x2c stepping 0x2 (id 0x206c2)
>>>
>> Hmm. Westmere EP. Are you running with EPT active, or with Shadow Paging?
> How do I know ?
Something like:
HVM: Hardware Assisted Paging (HAP) detected
on boot.
> Note that the same problem shows up on much newer systems: an i9, and a
> Xeon W-2223. Both boots fine with the same NetBSD kernel and Xen 4.18 or 4.15.
>
> I'm using this old Xeon for debug because this one has a serial console.
Sure. It's just that HAP vs Shadow is also relevant to breaking apart a
triple fault.
~Andrew
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:10 ` Manuel Bouyer
@ 2025-09-02 12:54 ` Jan Beulich
0 siblings, 0 replies; 29+ messages in thread
From: Jan Beulich @ 2025-09-02 12:54 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: xen-devel, Andrew Cooper
On 02.09.2025 14:10, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 02:00:35PM +0200, Jan Beulich wrote:
>> On 02.09.2025 12:56, Manuel Bouyer wrote:
>>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>>>> Hello,
>>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>>>> The same NetBSD kernel works fine with Xen 4.18
>>>>>
>>>>> The boot options are:
>>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>>>
>>>>> and the full log from serial console is attached.
>>>>>
>>>>> With 4.20 the boot fails with:
>>>>>
>>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>>>> (XEN) Freed 664kB init memory
>>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>>>> (XEN) CPU: 7
>>>>> (XEN) RIP: 0008:[<000000000020e268>]
>>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>>>
>>>>> because of the triple fault the RIP above doens't point to the code.
>>>>>
>>>>> I tracked it down to this code:
>>>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>>>> je 2f ; \
>>>>> pushl %ebp ; \
>>>>> movl RELOC(nox_flag),%ebp ; \
>>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>>>> loop 1b ; \
>>>>> popl %ebp ; \
>>>>> 2: ;
>>>>>
>>>>> there are others pushl/popl before so I don't think that's the problem
>>>>> (in fact the exact same fragment is called just before with different
>>>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>>>> which would be 0x20e260
>>>>> This is in the range:
>>>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>>>> so I can't see why this would be a problem.
>>>>>
>>>>> Any idea, including how to debug this further, welcome
>>>>
>>>> Even though triple fault's are aborts, they're generally accurate under
>>>> virt, so 0x20e268 is most likely where things die.
>>>
>>> but that's the RIP of the last fault, not the first one, right ?
>>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
>>> first fault triggers an exception, but the exeption handler isn't set up yet
>>> so we end up jumping to some random value.
>>
>> Can you perhaps check this guess against the %esp value seen? From the
>> hypervisor's triple fault handling, you may want to actually log stack
>> contents as well (in addition to what Andrew suggested), assuming %esp
>> looks sane to you.
>
> %esp is sane. I forgot to mention, this happens very early, while we're still
> in 32bits real mode. No function call did happen at this point, and on the
> stack there's only one 32bit value: the %ebp that we just pushed
Which by implication means no earlier exception(s).
Jan
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:45 ` Jan Beulich
@ 2025-09-02 12:54 ` Manuel Bouyer
2025-09-02 12:56 ` Jan Beulich
0 siblings, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 12:54 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On Tue, Sep 02, 2025 at 02:45:04PM +0200, Jan Beulich wrote:
> >> What puzzles me is that:
> >>
> >> - %cr2 is 0, so probably the first fault wasn't a page fault
> >
> > AFAIK it can't be as we're still in real mode
>
> It's protected mode, but with paging still off.
>
> >> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
> >>
> >> Could it be the code has been moved to this location, or is about to
> >> be moved away afterwards?
> >
> > No. RIP shouldn't end up there in any way. the assembly code is quite simple,
> > it's just a loop and I'm quite confident that we did enter the loop with
> > sane values
>
> Yet Jürgen has a point - entry point and what is being modified are on the
> same page (and despite paging still being off, you writing page tables here
> makes pages a relevant unit). Considering
> - entry point @ 0x20e4d0
> - %ecx = 0xdfeb7
> - %ebx = 0x20e260
> the loop continuing a little further will overwrite the entry point code.
> And with the entry point not at an even (e.g page-aligned) address, other
> code (like the one here) could conceivably live immediately ahead of it.
> (Of course this overwriting may be intentional, but it looks suspicious in
> this context.)
Indeed. Now the exact same kernel is booting fine with Xen 4.18 (and also
the same bootstrap is used for domU PVH which works with Xen 4.20).
I guess something changed in the way Xen sets up the dom0 kernel.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:52 ` Andrew Cooper
@ 2025-09-02 12:55 ` Andrew Cooper
2025-09-02 12:56 ` Manuel Bouyer
1 sibling, 0 replies; 29+ messages in thread
From: Andrew Cooper @ 2025-09-02 12:55 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: xen-devel
On 02/09/2025 1:52 pm, Andrew Cooper wrote:
> On 02/09/2025 1:24 pm, Manuel Bouyer wrote:
>> On Tue, Sep 02, 2025 at 01:14:29PM +0100, Andrew Cooper wrote:
>>> On 02/09/2025 12:23 pm, Manuel Bouyer wrote:
>>>> On Tue, Sep 02, 2025 at 12:13:27PM +0100, Andrew Cooper wrote:
>>>>> On 02/09/2025 11:56 am, Manuel Bouyer wrote:
>>>>>> On Tue, Sep 02, 2025 at 11:44:36AM +0100, Andrew Cooper wrote:
>>>>>>> On 02/09/2025 11:17 am, Manuel Bouyer wrote:
>>>>>>>> Hello,
>>>>>>>> I'm trying to boot a NetBSD PVH dom0 on Xen 4.20.
>>>>>>>> The same NetBSD kernel works fine with Xen 4.18
>>>>>>>>
>>>>>>>> The boot options are:
>>>>>>>> menu=Boot netbsd-current PVH Xen420:dev hd0f:;load /netbsd-PVH console=com0 root=wd0f; multiboot /xen420-debug.gz dom0_mem=1024M console=com1 com1=38400,8n1 loglvl=all guest_loglvl=all gnttab_max_nr_frames=64 sync_console=1 dom0=pvh
>>>>>>>>
>>>>>>>> and the full log from serial console is attached.
>>>>>>>>
>>>>>>>> With 4.20 the boot fails with:
>>>>>>>>
>>>>>>>> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
>>>>>>>> (XEN) Freed 664kB init memory
>>>>>>>> (XEN) d0v0 Triple fault - invoking HVM shutdown action 1
>>>>>>>> (XEN) *** Dumping Dom0 vcpu#0 state: ***
>>>>>>>> (XEN) ----[ Xen-4.20.2-pre_20250821nb0 x86_64 debug=y Tainted: C ]----
>>>>>>>> (XEN) CPU: 7
>>>>>>>> (XEN) RIP: 0008:[<000000000020e268>]
>>>>>>>> (XEN) RFLAGS: 0000000000010006 CONTEXT: hvm guest (d0v0)
>>>>>>>> (XEN) rax: 000000002024c003 rbx: 000000000020e260 rcx: 00000000000dfeb7
>>>>>>>> (XEN) rdx: 0000000000100000 rsi: 0000000000103000 rdi: 000000000013e000
>>>>>>>> (XEN) rbp: 0000000080000000 rsp: 00000000014002e4 r8: 0000000000000000
>>>>>>>> (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000
>>>>>>>> (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000
>>>>>>>> (XEN) r15: 0000000000000000 cr0: 0000000000000011 cr4: 0000000000000000
>>>>>>>> (XEN) cr3: 0000000000000000 cr2: 0000000000000000
>>>>>>>> (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000
>>>>>>>> (XEN) ds: 0010 es: 0010 fs: 0000 gs: 0000 ss: 0010 cs: 0008
>>>>>>>>
>>>>>>>> because of the triple fault the RIP above doens't point to the code.
>>>>>>>>
>>>>>>>> I tracked it down to this code:
>>>>>>>> cmpl $0,%ecx ; /* zero-sized? */ \
>>>>>>>> je 2f ; \
>>>>>>>> pushl %ebp ; \
>>>>>>>> movl RELOC(nox_flag),%ebp ; \
>>>>>>>> 1: movl %ebp,(PDE_SIZE-4)(%ebx) ; /* upper 32 bits: NX */ \
>>>>>>>> movl %eax,(%ebx) ; /* store phys addr */ \
>>>>>>>> addl $PDE_SIZE,%ebx ; /* next PTE/PDE */ \
>>>>>>>> addl $PAGE_SIZE,%eax ; /* next phys page */ \
>>>>>>>> loop 1b ; \
>>>>>>>> popl %ebp ; \
>>>>>>>> 2: ;
>>>>>>>>
>>>>>>>> there are others pushl/popl before so I don't think that's the problem
>>>>>>>> (in fact the exact same fragment is called just before with different
>>>>>>>> inputs and it doesn't fault). So the culprit it probably the write to (%ebx),
>>>>>>>> which would be 0x20e260
>>>>>>>> This is in the range:
>>>>>>>> (XEN) [0000000000100000, 0000000040068e77] (usable)
>>>>>>>> so I can't see why this would be a problem.
>>>>>>>>
>>>>>>>> Any idea, including how to debug this further, welcome
>>>>>>> Even though triple fault's are aborts, they're generally accurate under
>>>>>>> virt, so 0x20e268 is most likely where things die.
>>>>>> but that's the RIP of the last fault, not the first one, right ?
>>>>>> 0x20e268 isn't in the text segment of the kernel, my guess is that the
>>>>>> first fault triggers an exception, but the exeption handler isn't set up yet
>>>>>> so we end up jumping to some random value.
>>>>> Double and Triple faults occur when trying to deliver an exception
>>>>> generates an exception. So while multiple faults are involved, only one
>>>>> instruction typically is.
>>>>>
>>>>> Is this an Intel or an AMD system? One thing virt can do is break apart
>>>>> a triple fault, but the logic to do so is vendor specific.
>>>> it's an old intel system:
>>>> cpu0: "Intel(R) Xeon(R) CPU X5650 @ 2.67GHz"
>>>> cpu0: Intel Xeon 36xx & 56xx, i7, i5 and i3 (686-class), 2667.30 MHz
>>>> cpu0: family 0x6 model 0x2c stepping 0x2 (id 0x206c2)
>>>>
>>> Hmm. Westmere EP. Are you running with EPT active, or with Shadow Paging?
>> How do I know ?
> Something like:
>
> HVM: Hardware Assisted Paging (HAP) detected
>
> on boot.
Sorry - i missed the log on the root of the conversation. You do have
HAP on this system.
~Andrew
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:54 ` Manuel Bouyer
@ 2025-09-02 12:56 ` Jan Beulich
2025-09-02 13:05 ` Manuel Bouyer
2025-09-03 15:47 ` Manuel Bouyer
0 siblings, 2 replies; 29+ messages in thread
From: Jan Beulich @ 2025-09-02 12:56 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On 02.09.2025 14:54, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 02:45:04PM +0200, Jan Beulich wrote:
>>>> What puzzles me is that:
>>>>
>>>> - %cr2 is 0, so probably the first fault wasn't a page fault
>>>
>>> AFAIK it can't be as we're still in real mode
>>
>> It's protected mode, but with paging still off.
>>
>>>> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
>>>>
>>>> Could it be the code has been moved to this location, or is about to
>>>> be moved away afterwards?
>>>
>>> No. RIP shouldn't end up there in any way. the assembly code is quite simple,
>>> it's just a loop and I'm quite confident that we did enter the loop with
>>> sane values
>>
>> Yet Jürgen has a point - entry point and what is being modified are on the
>> same page (and despite paging still being off, you writing page tables here
>> makes pages a relevant unit). Considering
>> - entry point @ 0x20e4d0
>> - %ecx = 0xdfeb7
>> - %ebx = 0x20e260
>> the loop continuing a little further will overwrite the entry point code.
>> And with the entry point not at an even (e.g page-aligned) address, other
>> code (like the one here) could conceivably live immediately ahead of it.
>> (Of course this overwriting may be intentional, but it looks suspicious in
>> this context.)
>
> Indeed. Now the exact same kernel is booting fine with Xen 4.18 (and also
> the same bootstrap is used for domU PVH which works with Xen 4.20).
> I guess something changed in the way Xen sets up the dom0 kernel.
Hence why Andrew asked for full logs of both boots, to put side by side.
Jan
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:52 ` Andrew Cooper
2025-09-02 12:55 ` Andrew Cooper
@ 2025-09-02 12:56 ` Manuel Bouyer
1 sibling, 0 replies; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 12:56 UTC (permalink / raw)
To: Andrew Cooper; +Cc: xen-devel
On Tue, Sep 02, 2025 at 01:52:11PM +0100, Andrew Cooper wrote:
> HVM: Hardware Assisted Paging (HAP) detected
> on boot.
So it is:
(XEN) HVM: ASIDs enabled.
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
but the RIP printed by Xen may be right, after all
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:56 ` Jan Beulich
@ 2025-09-02 13:05 ` Manuel Bouyer
2025-09-03 15:47 ` Manuel Bouyer
1 sibling, 0 replies; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 13:05 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel, Juergen Gross
[-- Attachment #1: Type: text/plain, Size: 412 bytes --]
On Tue, Sep 02, 2025 at 02:56:23PM +0200, Jan Beulich wrote:
> Hence why Andrew asked for full logs of both boots, to put side by side.
Sorry I missed this.
Here's both log. 4.20 may be different from the one I posted earlier;
I did another boot to make sure it's the exact same kernel for both logs.
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
[-- Attachment #2: 4.18.log.gz --]
[-- Type: application/octet-stream, Size: 7885 bytes --]
[-- Attachment #3: 4.20.log.gz --]
[-- Type: application/octet-stream, Size: 8806 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:28 ` Juergen Gross
2025-09-02 12:49 ` Manuel Bouyer
@ 2025-09-02 13:41 ` Manuel Bouyer
2025-09-02 13:55 ` Jan Beulich
2025-09-02 13:58 ` Andrew Cooper
1 sibling, 2 replies; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 13:41 UTC (permalink / raw)
To: Juergen Gross; +Cc: Andrew Cooper, xen-devel
On Tue, Sep 02, 2025 at 02:28:27PM +0200, Juergen Gross wrote:
> > What puzzles me is that:
> >
> > - %cr2 is 0, so probably the first fault wasn't a page fault
> > - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
> >
> > Could it be the code has been moved to this location, or is about to
> > be moved away afterwards?
>
> And indeed: from the full boot log I can see:
>
> (XEN) virt_base = 0x0
> (XEN) elf_paddr_offset = 0x0
> (XEN) virt_offset = 0x0
> (XEN) virt_kstart = 0x200000
> (XEN) virt_kend = 0x17bab90
> (XEN) virt_entry = 0x20e4d0
>
> So virt_kentry is very near to the RIP.
thanks to this, I think I found the issue:
with Xen 4.18, the kernel is started with ebx=17bb018; with 4.20 it's
100018.
The bootstrap code assumes that the kernel is after the kernel, and the
kernel symbol table. That seems to be no longer true with Xen 4.20 and a
PVH dom0 (but probably still true in all other cases).
I can deal with that, but with the new layout how do I get the end of the
symbol table ?
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 13:41 ` Manuel Bouyer
@ 2025-09-02 13:55 ` Jan Beulich
2025-09-02 14:06 ` Manuel Bouyer
2025-09-02 13:58 ` Andrew Cooper
1 sibling, 1 reply; 29+ messages in thread
From: Jan Beulich @ 2025-09-02 13:55 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On 02.09.2025 15:41, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 02:28:27PM +0200, Juergen Gross wrote:
>>> What puzzles me is that:
>>>
>>> - %cr2 is 0, so probably the first fault wasn't a page fault
>>> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
>>>
>>> Could it be the code has been moved to this location, or is about to
>>> be moved away afterwards?
>>
>> And indeed: from the full boot log I can see:
>>
>> (XEN) virt_base = 0x0
>> (XEN) elf_paddr_offset = 0x0
>> (XEN) virt_offset = 0x0
>> (XEN) virt_kstart = 0x200000
>> (XEN) virt_kend = 0x17bab90
>> (XEN) virt_entry = 0x20e4d0
>>
>> So virt_kentry is very near to the RIP.
>
> thanks to this, I think I found the issue:
> with Xen 4.18, the kernel is started with ebx=17bb018; with 4.20 it's
> 100018.
>
> The bootstrap code assumes that the kernel is after the kernel, and the
DYM "start info is after the kernel" or some such, seeing that that's what
%ebx is about?
> kernel symbol table. That seems to be no longer true with Xen 4.20 and a
> PVH dom0 (but probably still true in all other cases).
>
> I can deal with that, but with the new layout how do I get the end of the
> symbol table ?
You'll need to handle that internally, I expect, perhaps from properties of
your (ELF) binary.
Jan
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 13:41 ` Manuel Bouyer
2025-09-02 13:55 ` Jan Beulich
@ 2025-09-02 13:58 ` Andrew Cooper
1 sibling, 0 replies; 29+ messages in thread
From: Andrew Cooper @ 2025-09-02 13:58 UTC (permalink / raw)
To: Manuel Bouyer, Juergen Gross; +Cc: xen-devel
On 02/09/2025 2:41 pm, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 02:28:27PM +0200, Juergen Gross wrote:
>>> What puzzles me is that:
>>>
>>> - %cr2 is 0, so probably the first fault wasn't a page fault
>>> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
>>>
>>> Could it be the code has been moved to this location, or is about to
>>> be moved away afterwards?
>> And indeed: from the full boot log I can see:
>>
>> (XEN) virt_base = 0x0
>> (XEN) elf_paddr_offset = 0x0
>> (XEN) virt_offset = 0x0
>> (XEN) virt_kstart = 0x200000
>> (XEN) virt_kend = 0x17bab90
>> (XEN) virt_entry = 0x20e4d0
>>
>> So virt_kentry is very near to the RIP.
> thanks to this, I think I found the issue:
> with Xen 4.18, the kernel is started with ebx=17bb018; with 4.20 it's
> 100018.
>
> The bootstrap code assumes that the kernel is after the kernel,
kernel after kernel?
> and the
> kernel symbol table. That seems to be no longer true with Xen 4.20 and a
> PVH dom0 (but probably still true in all other cases).
>
> I can deal with that, but with the new layout how do I get the end of the
> symbol table ?
ebx points to the PVH info. This is arbitrary, like multiboot info, and
you shouldn't make assumptions about it's position relative to the other
modules loaded.
~Andrew
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 13:55 ` Jan Beulich
@ 2025-09-02 14:06 ` Manuel Bouyer
2025-09-02 14:23 ` Jan Beulich
0 siblings, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 14:06 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On Tue, Sep 02, 2025 at 03:55:14PM +0200, Jan Beulich wrote:
> On 02.09.2025 15:41, Manuel Bouyer wrote:
> > On Tue, Sep 02, 2025 at 02:28:27PM +0200, Juergen Gross wrote:
> >>> What puzzles me is that:
> >>>
> >>> - %cr2 is 0, so probably the first fault wasn't a page fault
> >>> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
> >>>
> >>> Could it be the code has been moved to this location, or is about to
> >>> be moved away afterwards?
> >>
> >> And indeed: from the full boot log I can see:
> >>
> >> (XEN) virt_base = 0x0
> >> (XEN) elf_paddr_offset = 0x0
> >> (XEN) virt_offset = 0x0
> >> (XEN) virt_kstart = 0x200000
> >> (XEN) virt_kend = 0x17bab90
> >> (XEN) virt_entry = 0x20e4d0
> >>
> >> So virt_kentry is very near to the RIP.
> >
> > thanks to this, I think I found the issue:
> > with Xen 4.18, the kernel is started with ebx=17bb018; with 4.20 it's
> > 100018.
> >
> > The bootstrap code assumes that the kernel is after the kernel, and the
>
> DYM "start info is after the kernel" or some such, seeing that that's what
> %ebx is about?
yes, sorry
>
> > kernel symbol table. That seems to be no longer true with Xen 4.20 and a
> > PVH dom0 (but probably still true in all other cases).
> >
> > I can deal with that, but with the new layout how do I get the end of the
> > symbol table ?
>
> You'll need to handle that internally, I expect, perhaps from properties of
> your (ELF) binary.
But I don't have access to the ELF headers from the kernel binary (nor do I
know which kernel was booted).
Hum, maybe a I can hardcode this info in some const of the binary with a
ld trick ?
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 14:06 ` Manuel Bouyer
@ 2025-09-02 14:23 ` Jan Beulich
2025-09-02 14:28 ` Manuel Bouyer
0 siblings, 1 reply; 29+ messages in thread
From: Jan Beulich @ 2025-09-02 14:23 UTC (permalink / raw)
To: Manuel Bouyer; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On 02.09.2025 16:06, Manuel Bouyer wrote:
> On Tue, Sep 02, 2025 at 03:55:14PM +0200, Jan Beulich wrote:
>> On 02.09.2025 15:41, Manuel Bouyer wrote:
>>> On Tue, Sep 02, 2025 at 02:28:27PM +0200, Juergen Gross wrote:
>>>>> What puzzles me is that:
>>>>>
>>>>> - %cr2 is 0, so probably the first fault wasn't a page fault
>>>>> - RIP is %ebx + 8, so maybe the code was just clobbered by the loop?
>>>>>
>>>>> Could it be the code has been moved to this location, or is about to
>>>>> be moved away afterwards?
>>>>
>>>> And indeed: from the full boot log I can see:
>>>>
>>>> (XEN) virt_base = 0x0
>>>> (XEN) elf_paddr_offset = 0x0
>>>> (XEN) virt_offset = 0x0
>>>> (XEN) virt_kstart = 0x200000
>>>> (XEN) virt_kend = 0x17bab90
>>>> (XEN) virt_entry = 0x20e4d0
>>>>
>>>> So virt_kentry is very near to the RIP.
>>>
>>> thanks to this, I think I found the issue:
>>> with Xen 4.18, the kernel is started with ebx=17bb018; with 4.20 it's
>>> 100018.
>>>
>>> The bootstrap code assumes that the kernel is after the kernel, and the
>>
>> DYM "start info is after the kernel" or some such, seeing that that's what
>> %ebx is about?
>
> yes, sorry
>
>>> kernel symbol table. That seems to be no longer true with Xen 4.20 and a
>>> PVH dom0 (but probably still true in all other cases).
>>>
>>> I can deal with that, but with the new layout how do I get the end of the
>>> symbol table ?
>>
>> You'll need to handle that internally, I expect, perhaps from properties of
>> your (ELF) binary.
>
> But I don't have access to the ELF headers from the kernel binary (nor do I
> know which kernel was booted).
>
> Hum, maybe a I can hardcode this info in some const of the binary with a
> ld trick ?
For the symbol table to be loaded, it needs to live in some loadable section.
You should be able to mark that section's end (or the end of the symbol
table in the section, in case there's more stuff there) with a symbol in the
linker script (which I assume you use). If you used the GNU toolchain, you
could also consider using the assembler's .startof. / .sizeof. operators
(producing symbols that the linker then recognizes and resolves accordingly).
Or wait - are you perhaps using the thing we call "bsd_symtab" in our libelf?
Then, as per the scheme in elf_load_bsdsyms(), can't you find the start of
the ELF header from the end of your kernel? At least that's how I understand
it's supposed to work.
Jan
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 14:23 ` Jan Beulich
@ 2025-09-02 14:28 ` Manuel Bouyer
2025-09-02 14:32 ` Manuel Bouyer
0 siblings, 1 reply; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 14:28 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On Tue, Sep 02, 2025 at 04:23:28PM +0200, Jan Beulich wrote:
> >>> kernel symbol table. That seems to be no longer true with Xen 4.20 and a
> >>> PVH dom0 (but probably still true in all other cases).
> >>>
> >>> I can deal with that, but with the new layout how do I get the end of the
> >>> symbol table ?
> >>
> >> You'll need to handle that internally, I expect, perhaps from properties of
> >> your (ELF) binary.
> >
> > But I don't have access to the ELF headers from the kernel binary (nor do I
> > know which kernel was booted).
> >
> > Hum, maybe a I can hardcode this info in some const of the binary with a
> > ld trick ?
>
> For the symbol table to be loaded, it needs to live in some loadable section.
> You should be able to mark that section's end (or the end of the symbol
> table in the section, in case there's more stuff there) with a symbol in the
> linker script (which I assume you use). If you used the GNU toolchain, you
> could also consider using the assembler's .startof. / .sizeof. operators
> (producing symbols that the linker then recognizes and resolves accordingly).
>
> Or wait - are you perhaps using the thing we call "bsd_symtab" in our libelf?
yes, that's it. I'm looking at how it's loaded right now
> Then, as per the scheme in elf_load_bsdsyms(), can't you find the start of
> the ELF header from the end of your kernel? At least that's how I understand
> it's supposed to work.
I can, but at this point I can't easily call C code, and doing it in
assembly doesn't look easy :(
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 14:28 ` Manuel Bouyer
@ 2025-09-02 14:32 ` Manuel Bouyer
0 siblings, 0 replies; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-02 14:32 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel, Juergen Gross
On Tue, Sep 02, 2025 at 04:28:40PM +0200, Manuel Bouyer wrote:
> yes, that's it. I'm looking at how it's loaded right now
>
> > Then, as per the scheme in elf_load_bsdsyms(), can't you find the start of
> > the ELF header from the end of your kernel? At least that's how I understand
> > it's supposed to work.
>
> I can, but at this point I can't easily call C code, and doing it in
> assembly doesn't look easy :(
Oh, it looks like Xen stores the size just before the ELF header.
if that's the case, all good !
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: issue with dom0_pvh on Xen 4.20
2025-09-02 12:56 ` Jan Beulich
2025-09-02 13:05 ` Manuel Bouyer
@ 2025-09-03 15:47 ` Manuel Bouyer
1 sibling, 0 replies; 29+ messages in thread
From: Manuel Bouyer @ 2025-09-03 15:47 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel, Juergen Gross
Hello
with your help I now have a NetBSD PVH dom0 on Xen 4.20. Thanks all !
--
Manuel Bouyer <bouyer@antioche.eu.org>
NetBSD: 26 ans d'experience feront toujours la difference
--
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2025-09-03 15:48 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-02 10:17 issue with dom0_pvh on Xen 4.20 Manuel Bouyer
2025-09-02 10:44 ` Andrew Cooper
2025-09-02 10:56 ` Manuel Bouyer
2025-09-02 11:13 ` Andrew Cooper
2025-09-02 11:23 ` Manuel Bouyer
2025-09-02 12:14 ` Andrew Cooper
2025-09-02 12:24 ` Manuel Bouyer
2025-09-02 12:52 ` Andrew Cooper
2025-09-02 12:55 ` Andrew Cooper
2025-09-02 12:56 ` Manuel Bouyer
2025-09-02 12:00 ` Jan Beulich
2025-09-02 12:10 ` Manuel Bouyer
2025-09-02 12:54 ` Jan Beulich
2025-09-02 12:22 ` Juergen Gross
2025-09-02 12:28 ` Juergen Gross
2025-09-02 12:49 ` Manuel Bouyer
2025-09-02 13:41 ` Manuel Bouyer
2025-09-02 13:55 ` Jan Beulich
2025-09-02 14:06 ` Manuel Bouyer
2025-09-02 14:23 ` Jan Beulich
2025-09-02 14:28 ` Manuel Bouyer
2025-09-02 14:32 ` Manuel Bouyer
2025-09-02 13:58 ` Andrew Cooper
2025-09-02 12:28 ` Manuel Bouyer
2025-09-02 12:45 ` Jan Beulich
2025-09-02 12:54 ` Manuel Bouyer
2025-09-02 12:56 ` Jan Beulich
2025-09-02 13:05 ` Manuel Bouyer
2025-09-03 15:47 ` Manuel Bouyer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.