PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 3781 bytes --]

Hi,

As in the subject, the guest crashes on boot, before kernel output
anything. I've isolated this to the conditions below:
 - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
   without PCI device it works
 - Xen (in KVM) is started through OVMF; with seabios it works
 - nested HVM is disabled in KVM
 - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
   boot (looks like qemu bug, unrelated to this one)

Version info:
 - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
 - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
 - Xen domU: Linux 4.14.13, direct boot

Not sure if relevant, but initially I've tried booting xen.efi /mapbs
/noexitboot and then dom0 kernel crashed saying something about conflict
between e820 and kernel mapping. But now those options are disabled.

The crash message:
(XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
(XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
(XEN) Domain 1 (vcpu#0) crashed on cpu#1:
(XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e033:[<ffffffff826d9156>]
(XEN) RFLAGS: 0000000000000296   EM: 1   CONTEXT: pv guest (d1v0)
(XEN) rax: 0000000000000000   rbx: bdb25197f3daa61a   rcx: 000000000000003f
(XEN) rdx: ffffffff8206f450   rsi: 000000000000003f   rdi: 0000000000000000
(XEN) rbp: ffffffff82203e50   rsp: ffffffff82203d88   r8:  65c74fe852ba23f1
(XEN) r9:  b483505023b6d4a8   r10: c50a553bf60c0435   r11: fb097667f910d8cc
(XEN) r12: 0000000080000000   r13: 0000000000000000   r14: 0000010000000000
(XEN) r15: 000000000007aa00   cr0: 0000000080050033   cr4: 00000000000006e0
(XEN) cr3: 000000002a00a000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: ffffffff826a9000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff82203d88:
(XEN)    000000000000003f fb097667f910d8cc ffffffff826d9156 000000010000e030
(XEN)    0000000000010096 ffffffff82203dc8 000000000000e02b ffffffff826d9156
(XEN)    0add82a0ac2d25fc ffffffff82203e58 0000000001000000 ffffffff00000001
(XEN)    000000007f600000 0000000300000000 0000000019000000 0000000000019000
(XEN)    7ff0ffff82203e68 ffffffff00000017 ffffffff827ef004 0000000000000000
(XEN)    a2cc4720129e68ea 0000000001000000 ffffffff81000000 ffffffff82a66000
(XEN)    ffffffff82203ef8 ffffffff82203e70 ffffffff826e59d6 a76c832b9f537b2a
(XEN)    0000000001000000 ffffffff82203ee8 ffffffff826e13f7 0000000000000000
(XEN)    ffffffff810fbd4d cc49d4ba00000010 ffffffff82203ef8 ffffffff82203eb0
(XEN)    0000000000000000 0000000002a95000 0000000000000000 ec77a7137dd29529
(XEN)    ffffffffffffffff ffffffff82203f54 0000000000000000 0000000000000000
(XEN)    ffffffff82203f38 ffffffff826cd7a6 bb0469b365c5dafd ffffffff82203f20
(XEN)    0000000000000000 8529976ce838b598 ffffffff82203f58 ffffffff82203f54
(XEN)    0000000000000000 0000000000000000 ffffffff82203ff8 ffffffff826db329
(XEN)    00100f4200000000 0000000100000800 000000000789c3f5 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0f00000060c0c748

Any idea where to look?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Andrew Cooper]

On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> Hi,
>
> As in the subject, the guest crashes on boot, before kernel output
> anything. I've isolated this to the conditions below:
>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
>    without PCI device it works
>  - Xen (in KVM) is started through OVMF; with seabios it works
>  - nested HVM is disabled in KVM
>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
>    boot (looks like qemu bug, unrelated to this one)
>
> Version info:
>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
>  - Xen domU: Linux 4.14.13, direct boot
>
> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> /noexitboot and then dom0 kernel crashed saying something about conflict
> between e820 and kernel mapping. But now those options are disabled.
>
> The crash message:
> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> (XEN) CPU:    1
> (XEN) RIP:    e033:[<ffffffff826d9156>]

This is #UD, which is most probably hitting a BUG().  addr2line this ^
to find some code to look at.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 2112 bytes --]

On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> > Hi,
> >
> > As in the subject, the guest crashes on boot, before kernel output
> > anything. I've isolated this to the conditions below:
> >  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
> >    without PCI device it works
> >  - Xen (in KVM) is started through OVMF; with seabios it works
> >  - nested HVM is disabled in KVM
> >  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
> >    boot (looks like qemu bug, unrelated to this one)
> >
> > Version info:
> >  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
> >  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
> >  - Xen domU: Linux 4.14.13, direct boot
> >
> > Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> > /noexitboot and then dom0 kernel crashed saying something about conflict
> > between e820 and kernel mapping. But now those options are disabled.
> >
> > The crash message:
> > (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> > (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
> > (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> > (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> > (XEN) CPU:    1
> > (XEN) RIP:    e033:[<ffffffff826d9156>]
> 
> This is #UD, which is most probably hitting a BUG().  addr2line this ^
> to find some code to look at.

addr2line failed me, but System.map says its xen_memory_setup. And it
looks like the BUG() is the same as I had in dom0 before:
"Xen hypervisor allocated kernel memory conflicts with E820 map".

Disabling e820_host in guest config solved the problem. Thanks!

Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
should be avoided?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Andrew Cooper]

On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
>>> Hi,
>>>
>>> As in the subject, the guest crashes on boot, before kernel output
>>> anything. I've isolated this to the conditions below:
>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
>>>    without PCI device it works
>>>  - Xen (in KVM) is started through OVMF; with seabios it works
>>>  - nested HVM is disabled in KVM
>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
>>>    boot (looks like qemu bug, unrelated to this one)
>>>
>>> Version info:
>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
>>>  - Xen domU: Linux 4.14.13, direct boot
>>>
>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
>>> /noexitboot and then dom0 kernel crashed saying something about conflict
>>> between e820 and kernel mapping. But now those options are disabled.
>>>
>>> The crash message:
>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
>>> (XEN) CPU:    1
>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
>> to find some code to look at.
> addr2line failed me

By default, vmlinux is stripped and compressed.  Ideally you want to
addr2line the vmlinux artefact in the root of your kernel build, which
is the plain elf with debugging symbols.

Alternatively, use scripts/extract-vmlinux on the binary you actually
booted, which might get you somewhere.

> , but System.map says its xen_memory_setup. And it
> looks like the BUG() is the same as I had in dom0 before:
> "Xen hypervisor allocated kernel memory conflicts with E820 map".

Juergen: Is there anything we can do to try and insert some dummy
exception handlers right at PV start, so we could at least print out a
oneliner to the host console which is a little more helpful than Xen
saying "something unknown went wrong" ?

>
> Disabling e820_host in guest config solved the problem. Thanks!
>
> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
> should be avoided?

I don't really know.  e820_host is a gross hack which shouldn't really
be present.  The actually problem is that Linux can't cope with the
memory layout it was given (and I can't recall if there is anything
Linux could potentially to do cope).  OTOH, the toolstack, which knew
about e820_host and chose to lay the guest out in an overlapping way is
probably also at fault.

IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions
at all.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 3789 bytes --]

On Fri, Feb 16, 2018 at 07:02:39PM +0000, Andrew Cooper wrote:
> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
> > On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
> >> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> >>> Hi,
> >>>
> >>> As in the subject, the guest crashes on boot, before kernel output
> >>> anything. I've isolated this to the conditions below:
> >>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
> >>>    without PCI device it works
> >>>  - Xen (in KVM) is started through OVMF; with seabios it works
> >>>  - nested HVM is disabled in KVM
> >>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
> >>>    boot (looks like qemu bug, unrelated to this one)
> >>>
> >>> Version info:
> >>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
> >>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
> >>>  - Xen domU: Linux 4.14.13, direct boot
> >>>
> >>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> >>> /noexitboot and then dom0 kernel crashed saying something about conflict
> >>> between e820 and kernel mapping. But now those options are disabled.
> >>>
> >>> The crash message:
> >>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> >>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
> >>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> >>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> >>> (XEN) CPU:    1
> >>> (XEN) RIP:    e033:[<ffffffff826d9156>]
> >> This is #UD, which is most probably hitting a BUG().  addr2line this ^
> >> to find some code to look at.
> > addr2line failed me
> 
> By default, vmlinux is stripped and compressed.  Ideally you want to
> addr2line the vmlinux artefact in the root of your kernel build, which
> is the plain elf with debugging symbols.

Yes, I've used it on vmlinux. Still got "??:?".

> Alternatively, use scripts/extract-vmlinux on the binary you actually
> booted, which might get you somewhere.

Interestingly, that fails too ("Cannot find vmlinux.").
But I don't care right now.

> > , but System.map says its xen_memory_setup. And it
> > looks like the BUG() is the same as I had in dom0 before:
> > "Xen hypervisor allocated kernel memory conflicts with E820 map".
> 
> Juergen: Is there anything we can do to try and insert some dummy
> exception handlers right at PV start, so we could at least print out a
> oneliner to the host console which is a little more helpful than Xen
> saying "something unknown went wrong" ?

Just before the BUG(), there is a call to xen_raw_console_write(). But
apparently it was too early...

> > Disabling e820_host in guest config solved the problem. Thanks!
> >
> > Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
> > should be avoided?
> 
> I don't really know.  e820_host is a gross hack which shouldn't really
> be present.  The actually problem is that Linux can't cope with the
> memory layout it was given (and I can't recall if there is anything
> Linux could potentially to do cope).  OTOH, the toolstack, which knew
> about e820_host and chose to lay the guest out in an overlapping way is
> probably also at fault.

Yes, probably. But note that the same happened to dom0, when /mapbs is
used. Toolstack wasn't involved there. But /mapbs is also a hack.

> IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions
> at all.
> 
> ~Andrew

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Rich Persaud]

On Feb 16, 2018, at 14:02, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> 
> IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions
> at all.

Would that statement apply to other hypervisors like KVM, VMware ESX or Hyper-V, i.e. are the deficiencies in PCI devices/firmware, IOMMUs, platform firmware?  

If the statement is Xen specific, are there specific issues or known sections of Xen code that would benefit from a rewrite or redesign inspired by other hypervisors?

Rich
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Roger Pau Monné]

On Fri, Feb 16, 2018 at 04:35:14PM -0500, Rich Persaud wrote:
> On Feb 16, 2018, at 14:02, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> > 
> > IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions
> > at all.
> 
> Would that statement apply to other hypervisors like KVM, VMware ESX or Hyper-V, i.e. are the deficiencies in PCI devices/firmware, IOMMUs, platform firmware?  
> 
> If the statement is Xen specific, are there specific issues or known sections of Xen code that would benefit from a rewrite or redesign inspired by other hypervisors?

I personally don't fancy the fact that MSI-X handling for pass-through
to HVM guests is split between QEMU and the hypervisor itself. It
makes tracking down bugs or just trying to figure out the code flow
quite complex IMO.

I have plans to switch HVM to use the pass-through code being added
for PVH:

https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg02042.html

But that's still quite far away from being usable by unprivileged
guests.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Juergen Gross]

On 16/02/18 20:54, Marek Marczykowski-Górecki wrote:
> On Fri, Feb 16, 2018 at 07:02:39PM +0000, Andrew Cooper wrote:
>> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
>>> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
>>>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
>>>>> Hi,
>>>>>
>>>>> As in the subject, the guest crashes on boot, before kernel output
>>>>> anything. I've isolated this to the conditions below:
>>>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
>>>>>    without PCI device it works
>>>>>  - Xen (in KVM) is started through OVMF; with seabios it works
>>>>>  - nested HVM is disabled in KVM
>>>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
>>>>>    boot (looks like qemu bug, unrelated to this one)
>>>>>
>>>>> Version info:
>>>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
>>>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
>>>>>  - Xen domU: Linux 4.14.13, direct boot
>>>>>
>>>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
>>>>> /noexitboot and then dom0 kernel crashed saying something about conflict
>>>>> between e820 and kernel mapping. But now those options are disabled.
>>>>>
>>>>> The crash message:
>>>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
>>>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
>>>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
>>>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
>>>>> (XEN) CPU:    1
>>>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
>>>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
>>>> to find some code to look at.
>>> addr2line failed me
>>
>> By default, vmlinux is stripped and compressed.  Ideally you want to
>> addr2line the vmlinux artefact in the root of your kernel build, which
>> is the plain elf with debugging symbols.
> 
> Yes, I've used it on vmlinux. Still got "??:?".
> 
>> Alternatively, use scripts/extract-vmlinux on the binary you actually
>> booted, which might get you somewhere.
> 
> Interestingly, that fails too ("Cannot find vmlinux.").
> But I don't care right now.
> 
>>> , but System.map says its xen_memory_setup. And it
>>> looks like the BUG() is the same as I had in dom0 before:
>>> "Xen hypervisor allocated kernel memory conflicts with E820 map".
>>
>> Juergen: Is there anything we can do to try and insert some dummy
>> exception handlers right at PV start, so we could at least print out a
>> oneliner to the host console which is a little more helpful than Xen
>> saying "something unknown went wrong" ?
> 
> Just before the BUG(), there is a call to xen_raw_console_write(). But
> apparently it was too early...

Depends.

With a debug enabled hypervisor and appropriate log levels set
(guest_loglvl=all) you should see the message in the hypervisor log.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki]

[-- Attachment #1.1: Type: text/plain, Size: 758 bytes --]

On Mon, Feb 19, 2018 at 06:23:40PM +0100, Juergen Gross wrote:
> On 16/02/18 20:54, Marek Marczykowski-Górecki wrote:
> > Just before the BUG(), there is a call to xen_raw_console_write(). But
> > apparently it was too early...
> 
> Depends.
> 
> With a debug enabled hypervisor and appropriate log levels set
> (guest_loglvl=all) you should see the message in the hypervisor log.

I've tried with guest_loglvl=all, but there was nothing. Wouldn't it
make sense to enable this functionality also in non-debug builds - only
guarded by guest_loglvl=all option?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Juergen Gross]

On 16/02/18 20:02, Andrew Cooper wrote:
> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
>> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
>>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
>>>> Hi,
>>>>
>>>> As in the subject, the guest crashes on boot, before kernel output
>>>> anything. I've isolated this to the conditions below:
>>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
>>>>    without PCI device it works
>>>>  - Xen (in KVM) is started through OVMF; with seabios it works
>>>>  - nested HVM is disabled in KVM
>>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
>>>>    boot (looks like qemu bug, unrelated to this one)
>>>>
>>>> Version info:
>>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
>>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
>>>>  - Xen domU: Linux 4.14.13, direct boot
>>>>
>>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
>>>> /noexitboot and then dom0 kernel crashed saying something about conflict
>>>> between e820 and kernel mapping. But now those options are disabled.
>>>>
>>>> The crash message:
>>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
>>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
>>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
>>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
>>>> (XEN) CPU:    1
>>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
>>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
>>> to find some code to look at.
>> addr2line failed me
> 
> By default, vmlinux is stripped and compressed.  Ideally you want to
> addr2line the vmlinux artefact in the root of your kernel build, which
> is the plain elf with debugging symbols.
> 
> Alternatively, use scripts/extract-vmlinux on the binary you actually
> booted, which might get you somewhere.
> 
>> , but System.map says its xen_memory_setup. And it
>> looks like the BUG() is the same as I had in dom0 before:
>> "Xen hypervisor allocated kernel memory conflicts with E820 map".
> 
> Juergen: Is there anything we can do to try and insert some dummy
> exception handlers right at PV start, so we could at least print out a
> oneliner to the host console which is a little more helpful than Xen
> saying "something unknown went wrong" ?

You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07
added to kernel 4.15?

> 
>>
>> Disabling e820_host in guest config solved the problem. Thanks!
>>
>> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
>> should be avoided?
> 
> I don't really know.  e820_host is a gross hack which shouldn't really
> be present.  The actually problem is that Linux can't cope with the
> memory layout it was given (and I can't recall if there is anything
> Linux could potentially to do cope).  OTOH, the toolstack, which knew
> about e820_host and chose to lay the guest out in an overlapping way is
> probably also at fault.

The kernel can cope with lots of E820 scenarios (e.g. by relocating
initrd or the p2m map), but moving itself out of the way is not
possible.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Juergen Gross]

On 19/02/18 18:29, Marek Marczykowski-Górecki wrote:
> On Mon, Feb 19, 2018 at 06:23:40PM +0100, Juergen Gross wrote:
>> On 16/02/18 20:54, Marek Marczykowski-Górecki wrote:
>>> Just before the BUG(), there is a call to xen_raw_console_write(). But
>>> apparently it was too early...
>>
>> Depends.
>>
>> With a debug enabled hypervisor and appropriate log levels set
>> (guest_loglvl=all) you should see the message in the hypervisor log.
> 
> I've tried with guest_loglvl=all, but there was nothing. Wouldn't it
> make sense to enable this functionality also in non-debug builds - only
> guarded by guest_loglvl=all option?

Hmm, I think this is on purpose to avoid the possibility of a guest
flooding the hypervisor log.

Maybe adding a new option (e.g. guest_messages=on/off) switchable at
runtime only via "xl set-parameters" would avoid the general flooding
problem while enabling diagnosis of such problems?


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Andrew Cooper]

On 19/02/18 17:46, Juergen Gross wrote:
> On 19/02/18 18:29, Marek Marczykowski-Górecki wrote:
>> On Mon, Feb 19, 2018 at 06:23:40PM +0100, Juergen Gross wrote:
>>> On 16/02/18 20:54, Marek Marczykowski-Górecki wrote:
>>>> Just before the BUG(), there is a call to xen_raw_console_write(). But
>>>> apparently it was too early...
>>> Depends.
>>>
>>> With a debug enabled hypervisor and appropriate log levels set
>>> (guest_loglvl=all) you should see the message in the hypervisor log.
>> I've tried with guest_loglvl=all, but there was nothing. Wouldn't it
>> make sense to enable this functionality also in non-debug builds - only
>> guarded by guest_loglvl=all option?
> Hmm, I think this is on purpose to avoid the possibility of a guest
> flooding the hypervisor log.
>
> Maybe adding a new option (e.g. guest_messages=on/off) switchable at
> runtime only via "xl set-parameters" would avoid the general flooding
> problem while enabling diagnosis of such problems?

You want to recompile with CONFIG_VERBOSE, which you need EXPERT mode to
introduce into a release build.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 4520 bytes --]

On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote:
> On 16/02/18 20:02, Andrew Cooper wrote:
> > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
> >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
> >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> >>>> Hi,
> >>>>
> >>>> As in the subject, the guest crashes on boot, before kernel output
> >>>> anything. I've isolated this to the conditions below:
> >>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
> >>>>    without PCI device it works
> >>>>  - Xen (in KVM) is started through OVMF; with seabios it works
> >>>>  - nested HVM is disabled in KVM
> >>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
> >>>>    boot (looks like qemu bug, unrelated to this one)
> >>>>
> >>>> Version info:
> >>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
> >>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
> >>>>  - Xen domU: Linux 4.14.13, direct boot
> >>>>
> >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> >>>> /noexitboot and then dom0 kernel crashed saying something about conflict
> >>>> between e820 and kernel mapping. But now those options are disabled.
> >>>>
> >>>> The crash message:
> >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
> >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> >>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> >>>> (XEN) CPU:    1
> >>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
> >>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
> >>> to find some code to look at.
> >> addr2line failed me
> > 
> > By default, vmlinux is stripped and compressed.  Ideally you want to
> > addr2line the vmlinux artefact in the root of your kernel build, which
> > is the plain elf with debugging symbols.
> > 
> > Alternatively, use scripts/extract-vmlinux on the binary you actually
> > booted, which might get you somewhere.
> > 
> >> , but System.map says its xen_memory_setup. And it
> >> looks like the BUG() is the same as I had in dom0 before:
> >> "Xen hypervisor allocated kernel memory conflicts with E820 map".
> > 
> > Juergen: Is there anything we can do to try and insert some dummy
> > exception handlers right at PV start, so we could at least print out a
> > oneliner to the host console which is a little more helpful than Xen
> > saying "something unknown went wrong" ?
> 
> You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07
> added to kernel 4.15?
> 
> > 
> >>
> >> Disabling e820_host in guest config solved the problem. Thanks!
> >>
> >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
> >> should be avoided?
> > 
> > I don't really know.  e820_host is a gross hack which shouldn't really
> > be present.  The actually problem is that Linux can't cope with the
> > memory layout it was given (and I can't recall if there is anything
> > Linux could potentially to do cope).  OTOH, the toolstack, which knew
> > about e820_host and chose to lay the guest out in an overlapping way is
> > probably also at fault.
> 
> The kernel can cope with lots of E820 scenarios (e.g. by relocating
> initrd or the p2m map), but moving itself out of the way is not
> possible.

I'm afraid I need to resurrect this thread...

With recent kernel (6.6+), the host_e820=0 workaround is not an option
anymore. It makes Linux not initialize xen-swiotlb (due to
f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't
work at all. While I can add yet another layer of workaround (force
xen-swiotlb with iommu=soft), that's getting unwieldy.

Furthermore, I don't get the crash message anymore, even with debug
hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl
dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and
it's considered clean shutdown (I can confirm it by changing various
`on_*` settings (via libvirt) and observing which gets applied).

Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1
already.

This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses
QEMU 8.1.2 + OVMF 202308 to run Xen as L1.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 5667 bytes --]

On Mon, Nov 27, 2023 at 11:20:36AM +0000, Frediano Ziglio wrote:
> On Sun, Nov 26, 2023 at 2:51 PM Marek Marczykowski-Górecki
> <marmarek@invisiblethingslab.com> wrote:
> >
> > On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote:
> > > On 16/02/18 20:02, Andrew Cooper wrote:
> > > > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
> > > >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
> > > >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> > > >>>> Hi,
> > > >>>>
> > > >>>> As in the subject, the guest crashes on boot, before kernel output
> > > >>>> anything. I've isolated this to the conditions below:
> > > >>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
> > > >>>>    without PCI device it works
> > > >>>>  - Xen (in KVM) is started through OVMF; with seabios it works
> > > >>>>  - nested HVM is disabled in KVM
> > > >>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
> > > >>>>    boot (looks like qemu bug, unrelated to this one)
> > > >>>>
> > > >>>> Version info:
> > > >>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
> > > >>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
> > > >>>>  - Xen domU: Linux 4.14.13, direct boot
> > > >>>>
> > > >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> > > >>>> /noexitboot and then dom0 kernel crashed saying something about conflict
> > > >>>> between e820 and kernel mapping. But now those options are disabled.
> > > >>>>
> > > >>>> The crash message:
> > > >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> > > >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
> > > >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> > > >>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> > > >>>> (XEN) CPU:    1
> > > >>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
> > > >>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
> > > >>> to find some code to look at.
> > > >> addr2line failed me
> > > >
> > > > By default, vmlinux is stripped and compressed.  Ideally you want to
> > > > addr2line the vmlinux artefact in the root of your kernel build, which
> > > > is the plain elf with debugging symbols.
> > > >
> > > > Alternatively, use scripts/extract-vmlinux on the binary you actually
> > > > booted, which might get you somewhere.
> > > >
> > > >> , but System.map says its xen_memory_setup. And it
> > > >> looks like the BUG() is the same as I had in dom0 before:
> > > >> "Xen hypervisor allocated kernel memory conflicts with E820 map".
> > > >
> > > > Juergen: Is there anything we can do to try and insert some dummy
> > > > exception handlers right at PV start, so we could at least print out a
> > > > oneliner to the host console which is a little more helpful than Xen
> > > > saying "something unknown went wrong" ?
> > >
> > > You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07
> > > added to kernel 4.15?
> > >
> > > >
> > > >>
> > > >> Disabling e820_host in guest config solved the problem. Thanks!
> > > >>
> > > >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
> > > >> should be avoided?
> > > >
> > > > I don't really know.  e820_host is a gross hack which shouldn't really
> > > > be present.  The actually problem is that Linux can't cope with the
> > > > memory layout it was given (and I can't recall if there is anything
> > > > Linux could potentially to do cope).  OTOH, the toolstack, which knew
> > > > about e820_host and chose to lay the guest out in an overlapping way is
> > > > probably also at fault.
> > >
> > > The kernel can cope with lots of E820 scenarios (e.g. by relocating
> > > initrd or the p2m map), but moving itself out of the way is not
> > > possible.
> >
> > I'm afraid I need to resurrect this thread...
> >
> > With recent kernel (6.6+), the host_e820=0 workaround is not an option
> > anymore. It makes Linux not initialize xen-swiotlb (due to
> > f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't
> > work at all. While I can add yet another layer of workaround (force
> > xen-swiotlb with iommu=soft), that's getting unwieldy.
> >
> > Furthermore, I don't get the crash message anymore, even with debug
> > hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl
> > dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and
> > it's considered clean shutdown (I can confirm it by changing various
> > `on_*` settings (via libvirt) and observing which gets applied).
> >
> > Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1
> > already.
> >
> > This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses
> > QEMU 8.1.2 + OVMF 202308 to run Xen as L1.
> >
> 
> So basically you start the domain and it looks like it's shutting down
> cleanly from logs.
> Can you see anything from the guest? Can you turn on some more
> debugging at guest level?

No, it crashes before printing anything to the console, also with
earlyprintk=xen.

> I tried to get some more information from the initial crash but I
> could not understand which guest code triggered the bug.

I'm not sure which one is it this time (because I don't have Xen
reporting guest crash...) but last time it was here:
https://github.com/torvalds/linux/blob/master/arch/x86/xen/setup.c#L873-L874

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Jason Andryuk]

On Mon, Nov 27, 2023 at 6:27 AM Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:
>
> On Mon, Nov 27, 2023 at 11:20:36AM +0000, Frediano Ziglio wrote:
> > On Sun, Nov 26, 2023 at 2:51 PM Marek Marczykowski-Górecki
> > <marmarek@invisiblethingslab.com> wrote:
> > >
> > > On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote:
> > > > On 16/02/18 20:02, Andrew Cooper wrote:
> > > > > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
> > > > >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
> > > > >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> > > > >>>> Hi,
> > > > >>>>
> > > > >>>> As in the subject, the guest crashes on boot, before kernel output
> > > > >>>> anything. I've isolated this to the conditions below:
> > > > >>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
> > > > >>>>    without PCI device it works
> > > > >>>>  - Xen (in KVM) is started through OVMF; with seabios it works
> > > > >>>>  - nested HVM is disabled in KVM
> > > > >>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
> > > > >>>>    boot (looks like qemu bug, unrelated to this one)
> > > > >>>>
> > > > >>>> Version info:
> > > > >>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
> > > > >>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
> > > > >>>>  - Xen domU: Linux 4.14.13, direct boot
> > > > >>>>
> > > > >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> > > > >>>> /noexitboot and then dom0 kernel crashed saying something about conflict
> > > > >>>> between e820 and kernel mapping. But now those options are disabled.
> > > > >>>>
> > > > >>>> The crash message:
> > > > >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> > > > >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
> > > > >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> > > > >>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> > > > >>>> (XEN) CPU:    1
> > > > >>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
> > > > >>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
> > > > >>> to find some code to look at.
> > > > >> addr2line failed me
> > > > >
> > > > > By default, vmlinux is stripped and compressed.  Ideally you want to
> > > > > addr2line the vmlinux artefact in the root of your kernel build, which
> > > > > is the plain elf with debugging symbols.
> > > > >
> > > > > Alternatively, use scripts/extract-vmlinux on the binary you actually
> > > > > booted, which might get you somewhere.
> > > > >
> > > > >> , but System.map says its xen_memory_setup. And it
> > > > >> looks like the BUG() is the same as I had in dom0 before:
> > > > >> "Xen hypervisor allocated kernel memory conflicts with E820 map".
> > > > >
> > > > > Juergen: Is there anything we can do to try and insert some dummy
> > > > > exception handlers right at PV start, so we could at least print out a
> > > > > oneliner to the host console which is a little more helpful than Xen
> > > > > saying "something unknown went wrong" ?
> > > >
> > > > You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07
> > > > added to kernel 4.15?
> > > >
> > > > >
> > > > >>
> > > > >> Disabling e820_host in guest config solved the problem. Thanks!
> > > > >>
> > > > >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
> > > > >> should be avoided?
> > > > >
> > > > > I don't really know.  e820_host is a gross hack which shouldn't really
> > > > > be present.  The actually problem is that Linux can't cope with the
> > > > > memory layout it was given (and I can't recall if there is anything
> > > > > Linux could potentially to do cope).  OTOH, the toolstack, which knew
> > > > > about e820_host and chose to lay the guest out in an overlapping way is
> > > > > probably also at fault.
> > > >
> > > > The kernel can cope with lots of E820 scenarios (e.g. by relocating
> > > > initrd or the p2m map), but moving itself out of the way is not
> > > > possible.
> > >
> > > I'm afraid I need to resurrect this thread...
> > >
> > > With recent kernel (6.6+), the host_e820=0 workaround is not an option
> > > anymore. It makes Linux not initialize xen-swiotlb (due to
> > > f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't
> > > work at all. While I can add yet another layer of workaround (force
> > > xen-swiotlb with iommu=soft), that's getting unwieldy.
> > >
> > > Furthermore, I don't get the crash message anymore, even with debug
> > > hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl
> > > dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and
> > > it's considered clean shutdown (I can confirm it by changing various
> > > `on_*` settings (via libvirt) and observing which gets applied).
> > >
> > > Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1
> > > already.
> > >
> > > This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses
> > > QEMU 8.1.2 + OVMF 202308 to run Xen as L1.
> > >
> >
> > So basically you start the domain and it looks like it's shutting down
> > cleanly from logs.
> > Can you see anything from the guest? Can you turn on some more
> > debugging at guest level?
>
> No, it crashes before printing anything to the console, also with
> earlyprintk=xen.
>
> > I tried to get some more information from the initial crash but I
> > could not understand which guest code triggered the bug.
>
> I'm not sure which one is it this time (because I don't have Xen
> reporting guest crash...) but last time it was here:
> https://github.com/torvalds/linux/blob/master/arch/x86/xen/setup.c#L873-L874

Hi Marek,

I too have run into this "Xen hypervisor allocated kernel memory
conflicts with E820 map" error when running Xen under KVM & OVMF with
SecureBoot.  OVMF built without SecureBoot did not trip over the
issue.  It was a little while back - I have some notes though.

Non-SecureBoot
(XEN)  [0000000000810000, 00000000008fffff] (ACPI NVS)
(XEN)  [0000000000900000, 000000007f8eefff] (usable)

SecureBoot
(XEN)  [0000000000810000, 000000000170ffff] (ACPI NVS)
(XEN)  [0000000001710000, 000000007f0edfff] (usable)

Linux (under Xen) is checking that _pa(_text) (= 0x1000000) is RAM,
but it is not.  Looking at the E820 map, there is type 4, NVS, region
defined:
[0000000000810000, 000000000170ffff] (ACPI NVS)

When OVMF is built with SMM (for SecureBoot) and S3Supported is true,
the memory range 0x900000-0x170ffff is additionally marked ACPI NVS
and Linux trips over this.  It becomes usable RAM under Non-SecureBoot
so Linux boots fine.

What I don't understand is why there is even a check that _pa(_text)
is RAM.  Xen logs that it places dom0 way up high in memory, so the
physical address of the kernel pages are much higher than 0x1000000.
The value 0x1000000 for _pa(_text) doesn't match reality.  Maybe there
are some expectations for the ACPI NVS and other reserved regions to
be 1-1 mapped?  I tried removing the BUG mentioned above, but it still
failed to boot.  I think I also removed a second BUG, but
unfortunately I don't have notes on either.

The other thing I noticed is _pa() uses phys_base to shift its output,
but phys_base is not updated under Xen.  If _pa() is supposed to
reference the high memory addresses Xen assigned, then maybe that
should be updated?

Booting under SecureBoot was tangential to my goal at the time, so I
didn't pursue it further.

Regards,
Jason

Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 7598 bytes --]

On 27.11.23 16:56, Jason Andryuk wrote:
> On Mon, Nov 27, 2023 at 6:27 AM Marek Marczykowski-Górecki
> <marmarek@invisiblethingslab.com> wrote:
>>
>> On Mon, Nov 27, 2023 at 11:20:36AM +0000, Frediano Ziglio wrote:
>>> On Sun, Nov 26, 2023 at 2:51 PM Marek Marczykowski-Górecki
>>> <marmarek@invisiblethingslab.com> wrote:
>>>>
>>>> On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote:
>>>>> On 16/02/18 20:02, Andrew Cooper wrote:
>>>>>> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
>>>>>>> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
>>>>>>>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> As in the subject, the guest crashes on boot, before kernel output
>>>>>>>>> anything. I've isolated this to the conditions below:
>>>>>>>>>   - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
>>>>>>>>>     without PCI device it works
>>>>>>>>>   - Xen (in KVM) is started through OVMF; with seabios it works
>>>>>>>>>   - nested HVM is disabled in KVM
>>>>>>>>>   - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
>>>>>>>>>     boot (looks like qemu bug, unrelated to this one)
>>>>>>>>>
>>>>>>>>> Version info:
>>>>>>>>>   - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
>>>>>>>>>   - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
>>>>>>>>>   - Xen domU: Linux 4.14.13, direct boot
>>>>>>>>>
>>>>>>>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
>>>>>>>>> /noexitboot and then dom0 kernel crashed saying something about conflict
>>>>>>>>> between e820 and kernel mapping. But now those options are disabled.
>>>>>>>>>
>>>>>>>>> The crash message:
>>>>>>>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
>>>>>>>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
>>>>>>>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
>>>>>>>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
>>>>>>>>> (XEN) CPU:    1
>>>>>>>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
>>>>>>>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
>>>>>>>> to find some code to look at.
>>>>>>> addr2line failed me
>>>>>>
>>>>>> By default, vmlinux is stripped and compressed.  Ideally you want to
>>>>>> addr2line the vmlinux artefact in the root of your kernel build, which
>>>>>> is the plain elf with debugging symbols.
>>>>>>
>>>>>> Alternatively, use scripts/extract-vmlinux on the binary you actually
>>>>>> booted, which might get you somewhere.
>>>>>>
>>>>>>> , but System.map says its xen_memory_setup. And it
>>>>>>> looks like the BUG() is the same as I had in dom0 before:
>>>>>>> "Xen hypervisor allocated kernel memory conflicts with E820 map".
>>>>>>
>>>>>> Juergen: Is there anything we can do to try and insert some dummy
>>>>>> exception handlers right at PV start, so we could at least print out a
>>>>>> oneliner to the host console which is a little more helpful than Xen
>>>>>> saying "something unknown went wrong" ?
>>>>>
>>>>> You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07
>>>>> added to kernel 4.15?
>>>>>
>>>>>>
>>>>>>>
>>>>>>> Disabling e820_host in guest config solved the problem. Thanks!
>>>>>>>
>>>>>>> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
>>>>>>> should be avoided?
>>>>>>
>>>>>> I don't really know.  e820_host is a gross hack which shouldn't really
>>>>>> be present.  The actually problem is that Linux can't cope with the
>>>>>> memory layout it was given (and I can't recall if there is anything
>>>>>> Linux could potentially to do cope).  OTOH, the toolstack, which knew
>>>>>> about e820_host and chose to lay the guest out in an overlapping way is
>>>>>> probably also at fault.
>>>>>
>>>>> The kernel can cope with lots of E820 scenarios (e.g. by relocating
>>>>> initrd or the p2m map), but moving itself out of the way is not
>>>>> possible.
>>>>
>>>> I'm afraid I need to resurrect this thread...
>>>>
>>>> With recent kernel (6.6+), the host_e820=0 workaround is not an option
>>>> anymore. It makes Linux not initialize xen-swiotlb (due to
>>>> f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't
>>>> work at all. While I can add yet another layer of workaround (force
>>>> xen-swiotlb with iommu=soft), that's getting unwieldy.
>>>>
>>>> Furthermore, I don't get the crash message anymore, even with debug
>>>> hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl
>>>> dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and
>>>> it's considered clean shutdown (I can confirm it by changing various
>>>> `on_*` settings (via libvirt) and observing which gets applied).
>>>>
>>>> Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1
>>>> already.
>>>>
>>>> This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses
>>>> QEMU 8.1.2 + OVMF 202308 to run Xen as L1.
>>>>
>>>
>>> So basically you start the domain and it looks like it's shutting down
>>> cleanly from logs.
>>> Can you see anything from the guest? Can you turn on some more
>>> debugging at guest level?
>>
>> No, it crashes before printing anything to the console, also with
>> earlyprintk=xen.
>>
>>> I tried to get some more information from the initial crash but I
>>> could not understand which guest code triggered the bug.
>>
>> I'm not sure which one is it this time (because I don't have Xen
>> reporting guest crash...) but last time it was here:
>> https://github.com/torvalds/linux/blob/master/arch/x86/xen/setup.c#L873-L874
> 
> Hi Marek,
> 
> I too have run into this "Xen hypervisor allocated kernel memory
> conflicts with E820 map" error when running Xen under KVM & OVMF with
> SecureBoot.  OVMF built without SecureBoot did not trip over the
> issue.  It was a little while back - I have some notes though.
> 
> Non-SecureBoot
> (XEN)  [0000000000810000, 00000000008fffff] (ACPI NVS)
> (XEN)  [0000000000900000, 000000007f8eefff] (usable)
> 
> SecureBoot
> (XEN)  [0000000000810000, 000000000170ffff] (ACPI NVS)
> (XEN)  [0000000001710000, 000000007f0edfff] (usable)
> 
> Linux (under Xen) is checking that _pa(_text) (= 0x1000000) is RAM,
> but it is not.  Looking at the E820 map, there is type 4, NVS, region
> defined:
> [0000000000810000, 000000000170ffff] (ACPI NVS)
> 
> When OVMF is built with SMM (for SecureBoot) and S3Supported is true,
> the memory range 0x900000-0x170ffff is additionally marked ACPI NVS
> and Linux trips over this.  It becomes usable RAM under Non-SecureBoot
> so Linux boots fine.
> 
> What I don't understand is why there is even a check that _pa(_text)
> is RAM.  Xen logs that it places dom0 way up high in memory, so the
> physical address of the kernel pages are much higher than 0x1000000.
> The value 0x1000000 for _pa(_text) doesn't match reality.  Maybe there
> are some expectations for the ACPI NVS and other reserved regions to
> be 1-1 mapped?  I tried removing the BUG mentioned above, but it still
> failed to boot.  I think I also removed a second BUG, but
> unfortunately I don't have notes on either.

The _guest_ physical address is what matters here.

With using the host E820 map the PV-kernel tries to rearrange its guest
physical memory layout to match the E820 map. And a non-RAM GPA for the
location where the kernel is located triggers the BUG.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]