Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

From: Juergen Gross <jgross@suse.com>
To: "Marek Marczykowski-Górecki" <marmarek@invisiblethingslab.com>,
	"Andrew Cooper" <andrew.cooper3@citrix.com>
Cc: xen-devel <xen-devel@lists.xen.org>
Subject: Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF
Date: Mon, 19 Feb 2018 18:23:40 +0100	[thread overview]
Message-ID: <1f3c44fb-e557-9b5c-6e9f-2ac40a8e0562@suse.com> (raw)
In-Reply-To: <20180216195415.GK2084@mail-itl>

On 16/02/18 20:54, Marek Marczykowski-Górecki wrote:
> On Fri, Feb 16, 2018 at 07:02:39PM +0000, Andrew Cooper wrote:
>> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
>>> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
>>>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
>>>>> Hi,
>>>>>
>>>>> As in the subject, the guest crashes on boot, before kernel output
>>>>> anything. I've isolated this to the conditions below:
>>>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this case),
>>>>>    without PCI device it works
>>>>>  - Xen (in KVM) is started through OVMF; with seabios it works
>>>>>  - nested HVM is disabled in KVM
>>>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
>>>>>    boot (looks like qemu bug, unrelated to this one)
>>>>>
>>>>> Version info:
>>>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
>>>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
>>>>>  - Xen domU: Linux 4.14.13, direct boot
>>>>>
>>>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
>>>>> /noexitboot and then dom0 kernel crashed saying something about conflict
>>>>> between e820 and kernel mapping. But now those options are disabled.
>>>>>
>>>>> The crash message:
>>>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
>>>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146
>>>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
>>>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
>>>>> (XEN) CPU:    1
>>>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
>>>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
>>>> to find some code to look at.
>>> addr2line failed me
>>
>> By default, vmlinux is stripped and compressed.  Ideally you want to
>> addr2line the vmlinux artefact in the root of your kernel build, which
>> is the plain elf with debugging symbols.
> 
> Yes, I've used it on vmlinux. Still got "??:?".
> 
>> Alternatively, use scripts/extract-vmlinux on the binary you actually
>> booted, which might get you somewhere.
> 
> Interestingly, that fails too ("Cannot find vmlinux.").
> But I don't care right now.
> 
>>> , but System.map says its xen_memory_setup. And it
>>> looks like the BUG() is the same as I had in dom0 before:
>>> "Xen hypervisor allocated kernel memory conflicts with E820 map".
>>
>> Juergen: Is there anything we can do to try and insert some dummy
>> exception handlers right at PV start, so we could at least print out a
>> oneliner to the host console which is a little more helpful than Xen
>> saying "something unknown went wrong" ?
> 
> Just before the BUG(), there is a call to xen_raw_console_write(). But
> apparently it was too early...

Depends.

With a debug enabled hypervisor and appropriate log levels set
(guest_loglvl=all) you should see the message in the hypervisor log.

Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel