* PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF @ 2018-02-16 17:48 Marek Marczykowski-Górecki 2018-02-16 17:52 ` Andrew Cooper 0 siblings, 1 reply; 16+ messages in thread From: Marek Marczykowski-Górecki @ 2018-02-16 17:48 UTC (permalink / raw) To: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3781 bytes --] Hi, As in the subject, the guest crashes on boot, before kernel output anything. I've isolated this to the conditions below: - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), without PCI device it works - Xen (in KVM) is started through OVMF; with seabios it works - nested HVM is disabled in KVM - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on boot (looks like qemu bug, unrelated to this one) Version info: - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 - Xen domU: Linux 4.14.13, direct boot Not sure if relevant, but initially I've tried booting xen.efi /mapbs /noexitboot and then dom0 kernel crashed saying something about conflict between e820 and kernel mapping. But now those options are disabled. The crash message: (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 (XEN) Domain 1 (vcpu#0) crashed on cpu#1: (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- (XEN) CPU: 1 (XEN) RIP: e033:[<ffffffff826d9156>] (XEN) RFLAGS: 0000000000000296 EM: 1 CONTEXT: pv guest (d1v0) (XEN) rax: 0000000000000000 rbx: bdb25197f3daa61a rcx: 000000000000003f (XEN) rdx: ffffffff8206f450 rsi: 000000000000003f rdi: 0000000000000000 (XEN) rbp: ffffffff82203e50 rsp: ffffffff82203d88 r8: 65c74fe852ba23f1 (XEN) r9: b483505023b6d4a8 r10: c50a553bf60c0435 r11: fb097667f910d8cc (XEN) r12: 0000000080000000 r13: 0000000000000000 r14: 0000010000000000 (XEN) r15: 000000000007aa00 cr0: 0000000080050033 cr4: 00000000000006e0 (XEN) cr3: 000000002a00a000 cr2: 0000000000000000 (XEN) fsb: 0000000000000000 gsb: ffffffff826a9000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033 (XEN) Guest stack trace from rsp=ffffffff82203d88: (XEN) 000000000000003f fb097667f910d8cc ffffffff826d9156 000000010000e030 (XEN) 0000000000010096 ffffffff82203dc8 000000000000e02b ffffffff826d9156 (XEN) 0add82a0ac2d25fc ffffffff82203e58 0000000001000000 ffffffff00000001 (XEN) 000000007f600000 0000000300000000 0000000019000000 0000000000019000 (XEN) 7ff0ffff82203e68 ffffffff00000017 ffffffff827ef004 0000000000000000 (XEN) a2cc4720129e68ea 0000000001000000 ffffffff81000000 ffffffff82a66000 (XEN) ffffffff82203ef8 ffffffff82203e70 ffffffff826e59d6 a76c832b9f537b2a (XEN) 0000000001000000 ffffffff82203ee8 ffffffff826e13f7 0000000000000000 (XEN) ffffffff810fbd4d cc49d4ba00000010 ffffffff82203ef8 ffffffff82203eb0 (XEN) 0000000000000000 0000000002a95000 0000000000000000 ec77a7137dd29529 (XEN) ffffffffffffffff ffffffff82203f54 0000000000000000 0000000000000000 (XEN) ffffffff82203f38 ffffffff826cd7a6 bb0469b365c5dafd ffffffff82203f20 (XEN) 0000000000000000 8529976ce838b598 ffffffff82203f58 ffffffff82203f54 (XEN) 0000000000000000 0000000000000000 ffffffff82203ff8 ffffffff826db329 (XEN) 00100f4200000000 0000000100000800 000000000789c3f5 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000000000000 0000000000000000 0f00000060c0c748 Any idea where to look? -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? [-- Attachment #1.2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] [-- Attachment #2: Type: text/plain, Size: 157 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 17:48 PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF Marek Marczykowski-Górecki @ 2018-02-16 17:52 ` Andrew Cooper 2018-02-16 18:51 ` Marek Marczykowski-Górecki 0 siblings, 1 reply; 16+ messages in thread From: Andrew Cooper @ 2018-02-16 17:52 UTC (permalink / raw) To: Marek Marczykowski-Górecki, xen-devel On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: > Hi, > > As in the subject, the guest crashes on boot, before kernel output > anything. I've isolated this to the conditions below: > - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), > without PCI device it works > - Xen (in KVM) is started through OVMF; with seabios it works > - nested HVM is disabled in KVM > - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on > boot (looks like qemu bug, unrelated to this one) > > Version info: > - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD > - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 > - Xen domU: Linux 4.14.13, direct boot > > Not sure if relevant, but initially I've tried booting xen.efi /mapbs > /noexitboot and then dom0 kernel crashed saying something about conflict > between e820 and kernel mapping. But now those options are disabled. > > The crash message: > (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] > (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 > (XEN) Domain 1 (vcpu#0) crashed on cpu#1: > (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- > (XEN) CPU: 1 > (XEN) RIP: e033:[<ffffffff826d9156>] This is #UD, which is most probably hitting a BUG(). addr2line this ^ to find some code to look at. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 17:52 ` Andrew Cooper @ 2018-02-16 18:51 ` Marek Marczykowski-Górecki 2018-02-16 19:02 ` Andrew Cooper 0 siblings, 1 reply; 16+ messages in thread From: Marek Marczykowski-Górecki @ 2018-02-16 18:51 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel [-- Attachment #1.1: Type: text/plain, Size: 2112 bytes --] On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: > On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: > > Hi, > > > > As in the subject, the guest crashes on boot, before kernel output > > anything. I've isolated this to the conditions below: > > - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), > > without PCI device it works > > - Xen (in KVM) is started through OVMF; with seabios it works > > - nested HVM is disabled in KVM > > - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on > > boot (looks like qemu bug, unrelated to this one) > > > > Version info: > > - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD > > - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 > > - Xen domU: Linux 4.14.13, direct boot > > > > Not sure if relevant, but initially I've tried booting xen.efi /mapbs > > /noexitboot and then dom0 kernel crashed saying something about conflict > > between e820 and kernel mapping. But now those options are disabled. > > > > The crash message: > > (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] > > (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 > > (XEN) Domain 1 (vcpu#0) crashed on cpu#1: > > (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- > > (XEN) CPU: 1 > > (XEN) RIP: e033:[<ffffffff826d9156>] > > This is #UD, which is most probably hitting a BUG(). addr2line this ^ > to find some code to look at. addr2line failed me, but System.map says its xen_memory_setup. And it looks like the BUG() is the same as I had in dom0 before: "Xen hypervisor allocated kernel memory conflicts with E820 map". Disabling e820_host in guest config solved the problem. Thanks! Is this some bug in Xen or OVMF, or is it expected behavior and e820_host should be avoided? -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? [-- Attachment #1.2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] [-- Attachment #2: Type: text/plain, Size: 157 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 18:51 ` Marek Marczykowski-Górecki @ 2018-02-16 19:02 ` Andrew Cooper 2018-02-16 19:54 ` Marek Marczykowski-Górecki ` (2 more replies) 0 siblings, 3 replies; 16+ messages in thread From: Andrew Cooper @ 2018-02-16 19:02 UTC (permalink / raw) To: Marek Marczykowski-Górecki; +Cc: Juergen Gross, xen-devel On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: > On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: >> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: >>> Hi, >>> >>> As in the subject, the guest crashes on boot, before kernel output >>> anything. I've isolated this to the conditions below: >>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), >>> without PCI device it works >>> - Xen (in KVM) is started through OVMF; with seabios it works >>> - nested HVM is disabled in KVM >>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on >>> boot (looks like qemu bug, unrelated to this one) >>> >>> Version info: >>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD >>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 >>> - Xen domU: Linux 4.14.13, direct boot >>> >>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs >>> /noexitboot and then dom0 kernel crashed saying something about conflict >>> between e820 and kernel mapping. But now those options are disabled. >>> >>> The crash message: >>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] >>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 >>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: >>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- >>> (XEN) CPU: 1 >>> (XEN) RIP: e033:[<ffffffff826d9156>] >> This is #UD, which is most probably hitting a BUG(). addr2line this ^ >> to find some code to look at. > addr2line failed me By default, vmlinux is stripped and compressed. Ideally you want to addr2line the vmlinux artefact in the root of your kernel build, which is the plain elf with debugging symbols. Alternatively, use scripts/extract-vmlinux on the binary you actually booted, which might get you somewhere. > , but System.map says its xen_memory_setup. And it > looks like the BUG() is the same as I had in dom0 before: > "Xen hypervisor allocated kernel memory conflicts with E820 map". Juergen: Is there anything we can do to try and insert some dummy exception handlers right at PV start, so we could at least print out a oneliner to the host console which is a little more helpful than Xen saying "something unknown went wrong" ? > > Disabling e820_host in guest config solved the problem. Thanks! > > Is this some bug in Xen or OVMF, or is it expected behavior and e820_host > should be avoided? I don't really know. e820_host is a gross hack which shouldn't really be present. The actually problem is that Linux can't cope with the memory layout it was given (and I can't recall if there is anything Linux could potentially to do cope). OTOH, the toolstack, which knew about e820_host and chose to lay the guest out in an overlapping way is probably also at fault. IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions at all. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 19:02 ` Andrew Cooper @ 2018-02-16 19:54 ` Marek Marczykowski-Górecki 2018-02-19 17:23 ` Juergen Gross 2018-02-16 21:35 ` Rich Persaud 2018-02-19 17:30 ` Juergen Gross 2 siblings, 1 reply; 16+ messages in thread From: Marek Marczykowski-Górecki @ 2018-02-16 19:54 UTC (permalink / raw) To: Andrew Cooper; +Cc: Juergen Gross, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3789 bytes --] On Fri, Feb 16, 2018 at 07:02:39PM +0000, Andrew Cooper wrote: > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: > > On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: > >> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: > >>> Hi, > >>> > >>> As in the subject, the guest crashes on boot, before kernel output > >>> anything. I've isolated this to the conditions below: > >>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), > >>> without PCI device it works > >>> - Xen (in KVM) is started through OVMF; with seabios it works > >>> - nested HVM is disabled in KVM > >>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on > >>> boot (looks like qemu bug, unrelated to this one) > >>> > >>> Version info: > >>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD > >>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 > >>> - Xen domU: Linux 4.14.13, direct boot > >>> > >>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs > >>> /noexitboot and then dom0 kernel crashed saying something about conflict > >>> between e820 and kernel mapping. But now those options are disabled. > >>> > >>> The crash message: > >>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] > >>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 > >>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: > >>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- > >>> (XEN) CPU: 1 > >>> (XEN) RIP: e033:[<ffffffff826d9156>] > >> This is #UD, which is most probably hitting a BUG(). addr2line this ^ > >> to find some code to look at. > > addr2line failed me > > By default, vmlinux is stripped and compressed. Ideally you want to > addr2line the vmlinux artefact in the root of your kernel build, which > is the plain elf with debugging symbols. Yes, I've used it on vmlinux. Still got "??:?". > Alternatively, use scripts/extract-vmlinux on the binary you actually > booted, which might get you somewhere. Interestingly, that fails too ("Cannot find vmlinux."). But I don't care right now. > > , but System.map says its xen_memory_setup. And it > > looks like the BUG() is the same as I had in dom0 before: > > "Xen hypervisor allocated kernel memory conflicts with E820 map". > > Juergen: Is there anything we can do to try and insert some dummy > exception handlers right at PV start, so we could at least print out a > oneliner to the host console which is a little more helpful than Xen > saying "something unknown went wrong" ? Just before the BUG(), there is a call to xen_raw_console_write(). But apparently it was too early... > > Disabling e820_host in guest config solved the problem. Thanks! > > > > Is this some bug in Xen or OVMF, or is it expected behavior and e820_host > > should be avoided? > > I don't really know. e820_host is a gross hack which shouldn't really > be present. The actually problem is that Linux can't cope with the > memory layout it was given (and I can't recall if there is anything > Linux could potentially to do cope). OTOH, the toolstack, which knew > about e820_host and chose to lay the guest out in an overlapping way is > probably also at fault. Yes, probably. But note that the same happened to dom0, when /mapbs is used. Toolstack wasn't involved there. But /mapbs is also a hack. > IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions > at all. > > ~Andrew -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? [-- Attachment #1.2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] [-- Attachment #2: Type: text/plain, Size: 157 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 19:54 ` Marek Marczykowski-Górecki @ 2018-02-19 17:23 ` Juergen Gross 2018-02-19 17:29 ` Marek Marczykowski-Górecki 0 siblings, 1 reply; 16+ messages in thread From: Juergen Gross @ 2018-02-19 17:23 UTC (permalink / raw) To: Marek Marczykowski-Górecki, Andrew Cooper; +Cc: xen-devel On 16/02/18 20:54, Marek Marczykowski-Górecki wrote: > On Fri, Feb 16, 2018 at 07:02:39PM +0000, Andrew Cooper wrote: >> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: >>> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: >>>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: >>>>> Hi, >>>>> >>>>> As in the subject, the guest crashes on boot, before kernel output >>>>> anything. I've isolated this to the conditions below: >>>>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), >>>>> without PCI device it works >>>>> - Xen (in KVM) is started through OVMF; with seabios it works >>>>> - nested HVM is disabled in KVM >>>>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on >>>>> boot (looks like qemu bug, unrelated to this one) >>>>> >>>>> Version info: >>>>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD >>>>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 >>>>> - Xen domU: Linux 4.14.13, direct boot >>>>> >>>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs >>>>> /noexitboot and then dom0 kernel crashed saying something about conflict >>>>> between e820 and kernel mapping. But now those options are disabled. >>>>> >>>>> The crash message: >>>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] >>>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 >>>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: >>>>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- >>>>> (XEN) CPU: 1 >>>>> (XEN) RIP: e033:[<ffffffff826d9156>] >>>> This is #UD, which is most probably hitting a BUG(). addr2line this ^ >>>> to find some code to look at. >>> addr2line failed me >> >> By default, vmlinux is stripped and compressed. Ideally you want to >> addr2line the vmlinux artefact in the root of your kernel build, which >> is the plain elf with debugging symbols. > > Yes, I've used it on vmlinux. Still got "??:?". > >> Alternatively, use scripts/extract-vmlinux on the binary you actually >> booted, which might get you somewhere. > > Interestingly, that fails too ("Cannot find vmlinux."). > But I don't care right now. > >>> , but System.map says its xen_memory_setup. And it >>> looks like the BUG() is the same as I had in dom0 before: >>> "Xen hypervisor allocated kernel memory conflicts with E820 map". >> >> Juergen: Is there anything we can do to try and insert some dummy >> exception handlers right at PV start, so we could at least print out a >> oneliner to the host console which is a little more helpful than Xen >> saying "something unknown went wrong" ? > > Just before the BUG(), there is a call to xen_raw_console_write(). But > apparently it was too early... Depends. With a debug enabled hypervisor and appropriate log levels set (guest_loglvl=all) you should see the message in the hypervisor log. Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-19 17:23 ` Juergen Gross @ 2018-02-19 17:29 ` Marek Marczykowski-Górecki 2018-02-19 17:46 ` Juergen Gross 0 siblings, 1 reply; 16+ messages in thread From: Marek Marczykowski-Górecki @ 2018-02-19 17:29 UTC (permalink / raw) To: Juergen Gross; +Cc: Andrew Cooper, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 758 bytes --] On Mon, Feb 19, 2018 at 06:23:40PM +0100, Juergen Gross wrote: > On 16/02/18 20:54, Marek Marczykowski-Górecki wrote: > > Just before the BUG(), there is a call to xen_raw_console_write(). But > > apparently it was too early... > > Depends. > > With a debug enabled hypervisor and appropriate log levels set > (guest_loglvl=all) you should see the message in the hypervisor log. I've tried with guest_loglvl=all, but there was nothing. Wouldn't it make sense to enable this functionality also in non-debug builds - only guarded by guest_loglvl=all option? -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? [-- Attachment #1.2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] [-- Attachment #2: Type: text/plain, Size: 157 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-19 17:29 ` Marek Marczykowski-Górecki @ 2018-02-19 17:46 ` Juergen Gross 2018-02-19 17:49 ` Andrew Cooper 0 siblings, 1 reply; 16+ messages in thread From: Juergen Gross @ 2018-02-19 17:46 UTC (permalink / raw) To: Marek Marczykowski-Górecki; +Cc: Andrew Cooper, xen-devel On 19/02/18 18:29, Marek Marczykowski-Górecki wrote: > On Mon, Feb 19, 2018 at 06:23:40PM +0100, Juergen Gross wrote: >> On 16/02/18 20:54, Marek Marczykowski-Górecki wrote: >>> Just before the BUG(), there is a call to xen_raw_console_write(). But >>> apparently it was too early... >> >> Depends. >> >> With a debug enabled hypervisor and appropriate log levels set >> (guest_loglvl=all) you should see the message in the hypervisor log. > > I've tried with guest_loglvl=all, but there was nothing. Wouldn't it > make sense to enable this functionality also in non-debug builds - only > guarded by guest_loglvl=all option? Hmm, I think this is on purpose to avoid the possibility of a guest flooding the hypervisor log. Maybe adding a new option (e.g. guest_messages=on/off) switchable at runtime only via "xl set-parameters" would avoid the general flooding problem while enabling diagnosis of such problems? Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-19 17:46 ` Juergen Gross @ 2018-02-19 17:49 ` Andrew Cooper 0 siblings, 0 replies; 16+ messages in thread From: Andrew Cooper @ 2018-02-19 17:49 UTC (permalink / raw) To: Juergen Gross, Marek Marczykowski-Górecki; +Cc: xen-devel On 19/02/18 17:46, Juergen Gross wrote: > On 19/02/18 18:29, Marek Marczykowski-Górecki wrote: >> On Mon, Feb 19, 2018 at 06:23:40PM +0100, Juergen Gross wrote: >>> On 16/02/18 20:54, Marek Marczykowski-Górecki wrote: >>>> Just before the BUG(), there is a call to xen_raw_console_write(). But >>>> apparently it was too early... >>> Depends. >>> >>> With a debug enabled hypervisor and appropriate log levels set >>> (guest_loglvl=all) you should see the message in the hypervisor log. >> I've tried with guest_loglvl=all, but there was nothing. Wouldn't it >> make sense to enable this functionality also in non-debug builds - only >> guarded by guest_loglvl=all option? > Hmm, I think this is on purpose to avoid the possibility of a guest > flooding the hypervisor log. > > Maybe adding a new option (e.g. guest_messages=on/off) switchable at > runtime only via "xl set-parameters" would avoid the general flooding > problem while enabling diagnosis of such problems? You want to recompile with CONFIG_VERBOSE, which you need EXPERT mode to introduce into a release build. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 19:02 ` Andrew Cooper 2018-02-16 19:54 ` Marek Marczykowski-Górecki @ 2018-02-16 21:35 ` Rich Persaud 2018-02-19 17:13 ` Roger Pau Monné 2018-02-19 17:30 ` Juergen Gross 2 siblings, 1 reply; 16+ messages in thread From: Rich Persaud @ 2018-02-16 21:35 UTC (permalink / raw) To: Andrew Cooper; +Cc: Juergen Gross, Marek Marczykowski-Górecki, xen-devel On Feb 16, 2018, at 14:02, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > > IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions > at all. Would that statement apply to other hypervisors like KVM, VMware ESX or Hyper-V, i.e. are the deficiencies in PCI devices/firmware, IOMMUs, platform firmware? If the statement is Xen specific, are there specific issues or known sections of Xen code that would benefit from a rewrite or redesign inspired by other hypervisors? Rich _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 21:35 ` Rich Persaud @ 2018-02-19 17:13 ` Roger Pau Monné 0 siblings, 0 replies; 16+ messages in thread From: Roger Pau Monné @ 2018-02-19 17:13 UTC (permalink / raw) To: Rich Persaud Cc: Juergen Gross, Andrew Cooper, Marek Marczykowski-Górecki, xen-devel On Fri, Feb 16, 2018 at 04:35:14PM -0500, Rich Persaud wrote: > On Feb 16, 2018, at 14:02, Andrew Cooper <andrew.cooper3@citrix.com> wrote: > > > > IMO, PCI Passthrough is a trainwreck, and it is a miracle it functions > > at all. > > Would that statement apply to other hypervisors like KVM, VMware ESX or Hyper-V, i.e. are the deficiencies in PCI devices/firmware, IOMMUs, platform firmware? > > If the statement is Xen specific, are there specific issues or known sections of Xen code that would benefit from a rewrite or redesign inspired by other hypervisors? I personally don't fancy the fact that MSI-X handling for pass-through to HVM guests is split between QEMU and the hypervisor itself. It makes tracking down bugs or just trying to figure out the code flow quite complex IMO. I have plans to switch HVM to use the pass-through code being added for PVH: https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg02042.html But that's still quite far away from being usable by unprivileged guests. Roger. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-16 19:02 ` Andrew Cooper 2018-02-16 19:54 ` Marek Marczykowski-Górecki 2018-02-16 21:35 ` Rich Persaud @ 2018-02-19 17:30 ` Juergen Gross 2023-11-26 14:51 ` [Xen-devel] " Marek Marczykowski-Górecki 2 siblings, 1 reply; 16+ messages in thread From: Juergen Gross @ 2018-02-19 17:30 UTC (permalink / raw) To: Andrew Cooper, Marek Marczykowski-Górecki; +Cc: xen-devel On 16/02/18 20:02, Andrew Cooper wrote: > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: >>>> Hi, >>>> >>>> As in the subject, the guest crashes on boot, before kernel output >>>> anything. I've isolated this to the conditions below: >>>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), >>>> without PCI device it works >>>> - Xen (in KVM) is started through OVMF; with seabios it works >>>> - nested HVM is disabled in KVM >>>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on >>>> boot (looks like qemu bug, unrelated to this one) >>>> >>>> Version info: >>>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD >>>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 >>>> - Xen domU: Linux 4.14.13, direct boot >>>> >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs >>>> /noexitboot and then dom0 kernel crashed saying something about conflict >>>> between e820 and kernel mapping. But now those options are disabled. >>>> >>>> The crash message: >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: >>>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- >>>> (XEN) CPU: 1 >>>> (XEN) RIP: e033:[<ffffffff826d9156>] >>> This is #UD, which is most probably hitting a BUG(). addr2line this ^ >>> to find some code to look at. >> addr2line failed me > > By default, vmlinux is stripped and compressed. Ideally you want to > addr2line the vmlinux artefact in the root of your kernel build, which > is the plain elf with debugging symbols. > > Alternatively, use scripts/extract-vmlinux on the binary you actually > booted, which might get you somewhere. > >> , but System.map says its xen_memory_setup. And it >> looks like the BUG() is the same as I had in dom0 before: >> "Xen hypervisor allocated kernel memory conflicts with E820 map". > > Juergen: Is there anything we can do to try and insert some dummy > exception handlers right at PV start, so we could at least print out a > oneliner to the host console which is a little more helpful than Xen > saying "something unknown went wrong" ? You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07 added to kernel 4.15? > >> >> Disabling e820_host in guest config solved the problem. Thanks! >> >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host >> should be avoided? > > I don't really know. e820_host is a gross hack which shouldn't really > be present. The actually problem is that Linux can't cope with the > memory layout it was given (and I can't recall if there is anything > Linux could potentially to do cope). OTOH, the toolstack, which knew > about e820_host and chose to lay the guest out in an overlapping way is > probably also at fault. The kernel can cope with lots of E820 scenarios (e.g. by relocating initrd or the p2m map), but moving itself out of the way is not possible. Juergen _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2018-02-19 17:30 ` Juergen Gross @ 2023-11-26 14:51 ` Marek Marczykowski-Górecki [not found] ` <CACHz=ZiWufUenyw_wg+QuK86+gU5RZNkuJNzX9-K1UM5P3m8+Q@mail.gmail.com> 0 siblings, 1 reply; 16+ messages in thread From: Marek Marczykowski-Górecki @ 2023-11-26 14:51 UTC (permalink / raw) To: Juergen Gross; +Cc: Andrew Cooper, xen-devel [-- Attachment #1: Type: text/plain, Size: 4520 bytes --] On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote: > On 16/02/18 20:02, Andrew Cooper wrote: > > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: > >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: > >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: > >>>> Hi, > >>>> > >>>> As in the subject, the guest crashes on boot, before kernel output > >>>> anything. I've isolated this to the conditions below: > >>>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), > >>>> without PCI device it works > >>>> - Xen (in KVM) is started through OVMF; with seabios it works > >>>> - nested HVM is disabled in KVM > >>>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on > >>>> boot (looks like qemu bug, unrelated to this one) > >>>> > >>>> Version info: > >>>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD > >>>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 > >>>> - Xen domU: Linux 4.14.13, direct boot > >>>> > >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs > >>>> /noexitboot and then dom0 kernel crashed saying something about conflict > >>>> between e820 and kernel mapping. But now those options are disabled. > >>>> > >>>> The crash message: > >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] > >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 > >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: > >>>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- > >>>> (XEN) CPU: 1 > >>>> (XEN) RIP: e033:[<ffffffff826d9156>] > >>> This is #UD, which is most probably hitting a BUG(). addr2line this ^ > >>> to find some code to look at. > >> addr2line failed me > > > > By default, vmlinux is stripped and compressed. Ideally you want to > > addr2line the vmlinux artefact in the root of your kernel build, which > > is the plain elf with debugging symbols. > > > > Alternatively, use scripts/extract-vmlinux on the binary you actually > > booted, which might get you somewhere. > > > >> , but System.map says its xen_memory_setup. And it > >> looks like the BUG() is the same as I had in dom0 before: > >> "Xen hypervisor allocated kernel memory conflicts with E820 map". > > > > Juergen: Is there anything we can do to try and insert some dummy > > exception handlers right at PV start, so we could at least print out a > > oneliner to the host console which is a little more helpful than Xen > > saying "something unknown went wrong" ? > > You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07 > added to kernel 4.15? > > > > >> > >> Disabling e820_host in guest config solved the problem. Thanks! > >> > >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host > >> should be avoided? > > > > I don't really know. e820_host is a gross hack which shouldn't really > > be present. The actually problem is that Linux can't cope with the > > memory layout it was given (and I can't recall if there is anything > > Linux could potentially to do cope). OTOH, the toolstack, which knew > > about e820_host and chose to lay the guest out in an overlapping way is > > probably also at fault. > > The kernel can cope with lots of E820 scenarios (e.g. by relocating > initrd or the p2m map), but moving itself out of the way is not > possible. I'm afraid I need to resurrect this thread... With recent kernel (6.6+), the host_e820=0 workaround is not an option anymore. It makes Linux not initialize xen-swiotlb (due to f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't work at all. While I can add yet another layer of workaround (force xen-swiotlb with iommu=soft), that's getting unwieldy. Furthermore, I don't get the crash message anymore, even with debug hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and it's considered clean shutdown (I can confirm it by changing various `on_*` settings (via libvirt) and observing which gets applied). Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1 already. This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses QEMU 8.1.2 + OVMF 202308 to run Xen as L1. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
[parent not found: <CACHz=ZiWufUenyw_wg+QuK86+gU5RZNkuJNzX9-K1UM5P3m8+Q@mail.gmail.com>]
* Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF [not found] ` <CACHz=ZiWufUenyw_wg+QuK86+gU5RZNkuJNzX9-K1UM5P3m8+Q@mail.gmail.com> @ 2023-11-27 11:26 ` Marek Marczykowski-Górecki 2023-11-27 15:56 ` Jason Andryuk 0 siblings, 1 reply; 16+ messages in thread From: Marek Marczykowski-Górecki @ 2023-11-27 11:26 UTC (permalink / raw) To: Frediano Ziglio; +Cc: xen-devel [-- Attachment #1: Type: text/plain, Size: 5667 bytes --] On Mon, Nov 27, 2023 at 11:20:36AM +0000, Frediano Ziglio wrote: > On Sun, Nov 26, 2023 at 2:51 PM Marek Marczykowski-Górecki > <marmarek@invisiblethingslab.com> wrote: > > > > On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote: > > > On 16/02/18 20:02, Andrew Cooper wrote: > > > > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: > > > >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: > > > >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: > > > >>>> Hi, > > > >>>> > > > >>>> As in the subject, the guest crashes on boot, before kernel output > > > >>>> anything. I've isolated this to the conditions below: > > > >>>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), > > > >>>> without PCI device it works > > > >>>> - Xen (in KVM) is started through OVMF; with seabios it works > > > >>>> - nested HVM is disabled in KVM > > > >>>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on > > > >>>> boot (looks like qemu bug, unrelated to this one) > > > >>>> > > > >>>> Version info: > > > >>>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD > > > >>>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 > > > >>>> - Xen domU: Linux 4.14.13, direct boot > > > >>>> > > > >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs > > > >>>> /noexitboot and then dom0 kernel crashed saying something about conflict > > > >>>> between e820 and kernel mapping. But now those options are disabled. > > > >>>> > > > >>>> The crash message: > > > >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] > > > >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 > > > >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: > > > >>>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- > > > >>>> (XEN) CPU: 1 > > > >>>> (XEN) RIP: e033:[<ffffffff826d9156>] > > > >>> This is #UD, which is most probably hitting a BUG(). addr2line this ^ > > > >>> to find some code to look at. > > > >> addr2line failed me > > > > > > > > By default, vmlinux is stripped and compressed. Ideally you want to > > > > addr2line the vmlinux artefact in the root of your kernel build, which > > > > is the plain elf with debugging symbols. > > > > > > > > Alternatively, use scripts/extract-vmlinux on the binary you actually > > > > booted, which might get you somewhere. > > > > > > > >> , but System.map says its xen_memory_setup. And it > > > >> looks like the BUG() is the same as I had in dom0 before: > > > >> "Xen hypervisor allocated kernel memory conflicts with E820 map". > > > > > > > > Juergen: Is there anything we can do to try and insert some dummy > > > > exception handlers right at PV start, so we could at least print out a > > > > oneliner to the host console which is a little more helpful than Xen > > > > saying "something unknown went wrong" ? > > > > > > You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07 > > > added to kernel 4.15? > > > > > > > > > > >> > > > >> Disabling e820_host in guest config solved the problem. Thanks! > > > >> > > > >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host > > > >> should be avoided? > > > > > > > > I don't really know. e820_host is a gross hack which shouldn't really > > > > be present. The actually problem is that Linux can't cope with the > > > > memory layout it was given (and I can't recall if there is anything > > > > Linux could potentially to do cope). OTOH, the toolstack, which knew > > > > about e820_host and chose to lay the guest out in an overlapping way is > > > > probably also at fault. > > > > > > The kernel can cope with lots of E820 scenarios (e.g. by relocating > > > initrd or the p2m map), but moving itself out of the way is not > > > possible. > > > > I'm afraid I need to resurrect this thread... > > > > With recent kernel (6.6+), the host_e820=0 workaround is not an option > > anymore. It makes Linux not initialize xen-swiotlb (due to > > f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't > > work at all. While I can add yet another layer of workaround (force > > xen-swiotlb with iommu=soft), that's getting unwieldy. > > > > Furthermore, I don't get the crash message anymore, even with debug > > hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl > > dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and > > it's considered clean shutdown (I can confirm it by changing various > > `on_*` settings (via libvirt) and observing which gets applied). > > > > Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1 > > already. > > > > This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses > > QEMU 8.1.2 + OVMF 202308 to run Xen as L1. > > > > So basically you start the domain and it looks like it's shutting down > cleanly from logs. > Can you see anything from the guest? Can you turn on some more > debugging at guest level? No, it crashes before printing anything to the console, also with earlyprintk=xen. > I tried to get some more information from the initial crash but I > could not understand which guest code triggered the bug. I'm not sure which one is it this time (because I don't have Xen reporting guest crash...) but last time it was here: https://github.com/torvalds/linux/blob/master/arch/x86/xen/setup.c#L873-L874 -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2023-11-27 11:26 ` Marek Marczykowski-Górecki @ 2023-11-27 15:56 ` Jason Andryuk 2023-11-27 16:05 ` Juergen Gross 0 siblings, 1 reply; 16+ messages in thread From: Jason Andryuk @ 2023-11-27 15:56 UTC (permalink / raw) To: Marek Marczykowski-Górecki; +Cc: Frediano Ziglio, xen-devel On Mon, Nov 27, 2023 at 6:27 AM Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> wrote: > > On Mon, Nov 27, 2023 at 11:20:36AM +0000, Frediano Ziglio wrote: > > On Sun, Nov 26, 2023 at 2:51 PM Marek Marczykowski-Górecki > > <marmarek@invisiblethingslab.com> wrote: > > > > > > On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote: > > > > On 16/02/18 20:02, Andrew Cooper wrote: > > > > > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: > > > > >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: > > > > >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: > > > > >>>> Hi, > > > > >>>> > > > > >>>> As in the subject, the guest crashes on boot, before kernel output > > > > >>>> anything. I've isolated this to the conditions below: > > > > >>>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), > > > > >>>> without PCI device it works > > > > >>>> - Xen (in KVM) is started through OVMF; with seabios it works > > > > >>>> - nested HVM is disabled in KVM > > > > >>>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on > > > > >>>> boot (looks like qemu bug, unrelated to this one) > > > > >>>> > > > > >>>> Version info: > > > > >>>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD > > > > >>>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 > > > > >>>> - Xen domU: Linux 4.14.13, direct boot > > > > >>>> > > > > >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs > > > > >>>> /noexitboot and then dom0 kernel crashed saying something about conflict > > > > >>>> between e820 and kernel mapping. But now those options are disabled. > > > > >>>> > > > > >>>> The crash message: > > > > >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] > > > > >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 > > > > >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: > > > > >>>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- > > > > >>>> (XEN) CPU: 1 > > > > >>>> (XEN) RIP: e033:[<ffffffff826d9156>] > > > > >>> This is #UD, which is most probably hitting a BUG(). addr2line this ^ > > > > >>> to find some code to look at. > > > > >> addr2line failed me > > > > > > > > > > By default, vmlinux is stripped and compressed. Ideally you want to > > > > > addr2line the vmlinux artefact in the root of your kernel build, which > > > > > is the plain elf with debugging symbols. > > > > > > > > > > Alternatively, use scripts/extract-vmlinux on the binary you actually > > > > > booted, which might get you somewhere. > > > > > > > > > >> , but System.map says its xen_memory_setup. And it > > > > >> looks like the BUG() is the same as I had in dom0 before: > > > > >> "Xen hypervisor allocated kernel memory conflicts with E820 map". > > > > > > > > > > Juergen: Is there anything we can do to try and insert some dummy > > > > > exception handlers right at PV start, so we could at least print out a > > > > > oneliner to the host console which is a little more helpful than Xen > > > > > saying "something unknown went wrong" ? > > > > > > > > You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07 > > > > added to kernel 4.15? > > > > > > > > > > > > > >> > > > > >> Disabling e820_host in guest config solved the problem. Thanks! > > > > >> > > > > >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host > > > > >> should be avoided? > > > > > > > > > > I don't really know. e820_host is a gross hack which shouldn't really > > > > > be present. The actually problem is that Linux can't cope with the > > > > > memory layout it was given (and I can't recall if there is anything > > > > > Linux could potentially to do cope). OTOH, the toolstack, which knew > > > > > about e820_host and chose to lay the guest out in an overlapping way is > > > > > probably also at fault. > > > > > > > > The kernel can cope with lots of E820 scenarios (e.g. by relocating > > > > initrd or the p2m map), but moving itself out of the way is not > > > > possible. > > > > > > I'm afraid I need to resurrect this thread... > > > > > > With recent kernel (6.6+), the host_e820=0 workaround is not an option > > > anymore. It makes Linux not initialize xen-swiotlb (due to > > > f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't > > > work at all. While I can add yet another layer of workaround (force > > > xen-swiotlb with iommu=soft), that's getting unwieldy. > > > > > > Furthermore, I don't get the crash message anymore, even with debug > > > hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl > > > dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and > > > it's considered clean shutdown (I can confirm it by changing various > > > `on_*` settings (via libvirt) and observing which gets applied). > > > > > > Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1 > > > already. > > > > > > This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses > > > QEMU 8.1.2 + OVMF 202308 to run Xen as L1. > > > > > > > So basically you start the domain and it looks like it's shutting down > > cleanly from logs. > > Can you see anything from the guest? Can you turn on some more > > debugging at guest level? > > No, it crashes before printing anything to the console, also with > earlyprintk=xen. > > > I tried to get some more information from the initial crash but I > > could not understand which guest code triggered the bug. > > I'm not sure which one is it this time (because I don't have Xen > reporting guest crash...) but last time it was here: > https://github.com/torvalds/linux/blob/master/arch/x86/xen/setup.c#L873-L874 Hi Marek, I too have run into this "Xen hypervisor allocated kernel memory conflicts with E820 map" error when running Xen under KVM & OVMF with SecureBoot. OVMF built without SecureBoot did not trip over the issue. It was a little while back - I have some notes though. Non-SecureBoot (XEN) [0000000000810000, 00000000008fffff] (ACPI NVS) (XEN) [0000000000900000, 000000007f8eefff] (usable) SecureBoot (XEN) [0000000000810000, 000000000170ffff] (ACPI NVS) (XEN) [0000000001710000, 000000007f0edfff] (usable) Linux (under Xen) is checking that _pa(_text) (= 0x1000000) is RAM, but it is not. Looking at the E820 map, there is type 4, NVS, region defined: [0000000000810000, 000000000170ffff] (ACPI NVS) When OVMF is built with SMM (for SecureBoot) and S3Supported is true, the memory range 0x900000-0x170ffff is additionally marked ACPI NVS and Linux trips over this. It becomes usable RAM under Non-SecureBoot so Linux boots fine. What I don't understand is why there is even a check that _pa(_text) is RAM. Xen logs that it places dom0 way up high in memory, so the physical address of the kernel pages are much higher than 0x1000000. The value 0x1000000 for _pa(_text) doesn't match reality. Maybe there are some expectations for the ACPI NVS and other reserved regions to be 1-1 mapped? I tried removing the BUG mentioned above, but it still failed to boot. I think I also removed a second BUG, but unfortunately I don't have notes on either. The other thing I noticed is _pa() uses phys_base to shift its output, but phys_base is not updated under Xen. If _pa() is supposed to reference the high memory addresses Xen assigned, then maybe that should be updated? Booting under SecureBoot was tangential to my goal at the time, so I didn't pursue it further. Regards, Jason ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF 2023-11-27 15:56 ` Jason Andryuk @ 2023-11-27 16:05 ` Juergen Gross 0 siblings, 0 replies; 16+ messages in thread From: Juergen Gross @ 2023-11-27 16:05 UTC (permalink / raw) To: xen-devel [-- Attachment #1.1.1: Type: text/plain, Size: 7598 bytes --] On 27.11.23 16:56, Jason Andryuk wrote: > On Mon, Nov 27, 2023 at 6:27 AM Marek Marczykowski-Górecki > <marmarek@invisiblethingslab.com> wrote: >> >> On Mon, Nov 27, 2023 at 11:20:36AM +0000, Frediano Ziglio wrote: >>> On Sun, Nov 26, 2023 at 2:51 PM Marek Marczykowski-Górecki >>> <marmarek@invisiblethingslab.com> wrote: >>>> >>>> On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote: >>>>> On 16/02/18 20:02, Andrew Cooper wrote: >>>>>> On 16/02/18 18:51, Marek Marczykowski-Górecki wrote: >>>>>>> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote: >>>>>>>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> As in the subject, the guest crashes on boot, before kernel output >>>>>>>>> anything. I've isolated this to the conditions below: >>>>>>>>> - PV guest have PCI device assigned (e1000e emulated by QEMU in this case), >>>>>>>>> without PCI device it works >>>>>>>>> - Xen (in KVM) is started through OVMF; with seabios it works >>>>>>>>> - nested HVM is disabled in KVM >>>>>>>>> - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on >>>>>>>>> boot (looks like qemu bug, unrelated to this one) >>>>>>>>> >>>>>>>>> Version info: >>>>>>>>> - KVM host: OpenSUSE 42.3, qemu 2.9.1, ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD >>>>>>>>> - Xen host: Xen 4.8.3, dom0: Linux 4.14.13 >>>>>>>>> - Xen domU: Linux 4.14.13, direct boot >>>>>>>>> >>>>>>>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs >>>>>>>>> /noexitboot and then dom0 kernel crashed saying something about conflict >>>>>>>>> between e820 and kernel mapping. But now those options are disabled. >>>>>>>>> >>>>>>>>> The crash message: >>>>>>>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000] >>>>>>>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 entry.o#create_bounce_frame+0x137/0x146 >>>>>>>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1: >>>>>>>>> (XEN) ----[ Xen-4.8.3 x86_64 debug=n Not tainted ]---- >>>>>>>>> (XEN) CPU: 1 >>>>>>>>> (XEN) RIP: e033:[<ffffffff826d9156>] >>>>>>>> This is #UD, which is most probably hitting a BUG(). addr2line this ^ >>>>>>>> to find some code to look at. >>>>>>> addr2line failed me >>>>>> >>>>>> By default, vmlinux is stripped and compressed. Ideally you want to >>>>>> addr2line the vmlinux artefact in the root of your kernel build, which >>>>>> is the plain elf with debugging symbols. >>>>>> >>>>>> Alternatively, use scripts/extract-vmlinux on the binary you actually >>>>>> booted, which might get you somewhere. >>>>>> >>>>>>> , but System.map says its xen_memory_setup. And it >>>>>>> looks like the BUG() is the same as I had in dom0 before: >>>>>>> "Xen hypervisor allocated kernel memory conflicts with E820 map". >>>>>> >>>>>> Juergen: Is there anything we can do to try and insert some dummy >>>>>> exception handlers right at PV start, so we could at least print out a >>>>>> oneliner to the host console which is a little more helpful than Xen >>>>>> saying "something unknown went wrong" ? >>>>> >>>>> You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07 >>>>> added to kernel 4.15? >>>>> >>>>>> >>>>>>> >>>>>>> Disabling e820_host in guest config solved the problem. Thanks! >>>>>>> >>>>>>> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host >>>>>>> should be avoided? >>>>>> >>>>>> I don't really know. e820_host is a gross hack which shouldn't really >>>>>> be present. The actually problem is that Linux can't cope with the >>>>>> memory layout it was given (and I can't recall if there is anything >>>>>> Linux could potentially to do cope). OTOH, the toolstack, which knew >>>>>> about e820_host and chose to lay the guest out in an overlapping way is >>>>>> probably also at fault. >>>>> >>>>> The kernel can cope with lots of E820 scenarios (e.g. by relocating >>>>> initrd or the p2m map), but moving itself out of the way is not >>>>> possible. >>>> >>>> I'm afraid I need to resurrect this thread... >>>> >>>> With recent kernel (6.6+), the host_e820=0 workaround is not an option >>>> anymore. It makes Linux not initialize xen-swiotlb (due to >>>> f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't >>>> work at all. While I can add yet another layer of workaround (force >>>> xen-swiotlb with iommu=soft), that's getting unwieldy. >>>> >>>> Furthermore, I don't get the crash message anymore, even with debug >>>> hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl >>>> dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and >>>> it's considered clean shutdown (I can confirm it by changing various >>>> `on_*` settings (via libvirt) and observing which gets applied). >>>> >>>> Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1 >>>> already. >>>> >>>> This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses >>>> QEMU 8.1.2 + OVMF 202308 to run Xen as L1. >>>> >>> >>> So basically you start the domain and it looks like it's shutting down >>> cleanly from logs. >>> Can you see anything from the guest? Can you turn on some more >>> debugging at guest level? >> >> No, it crashes before printing anything to the console, also with >> earlyprintk=xen. >> >>> I tried to get some more information from the initial crash but I >>> could not understand which guest code triggered the bug. >> >> I'm not sure which one is it this time (because I don't have Xen >> reporting guest crash...) but last time it was here: >> https://github.com/torvalds/linux/blob/master/arch/x86/xen/setup.c#L873-L874 > > Hi Marek, > > I too have run into this "Xen hypervisor allocated kernel memory > conflicts with E820 map" error when running Xen under KVM & OVMF with > SecureBoot. OVMF built without SecureBoot did not trip over the > issue. It was a little while back - I have some notes though. > > Non-SecureBoot > (XEN) [0000000000810000, 00000000008fffff] (ACPI NVS) > (XEN) [0000000000900000, 000000007f8eefff] (usable) > > SecureBoot > (XEN) [0000000000810000, 000000000170ffff] (ACPI NVS) > (XEN) [0000000001710000, 000000007f0edfff] (usable) > > Linux (under Xen) is checking that _pa(_text) (= 0x1000000) is RAM, > but it is not. Looking at the E820 map, there is type 4, NVS, region > defined: > [0000000000810000, 000000000170ffff] (ACPI NVS) > > When OVMF is built with SMM (for SecureBoot) and S3Supported is true, > the memory range 0x900000-0x170ffff is additionally marked ACPI NVS > and Linux trips over this. It becomes usable RAM under Non-SecureBoot > so Linux boots fine. > > What I don't understand is why there is even a check that _pa(_text) > is RAM. Xen logs that it places dom0 way up high in memory, so the > physical address of the kernel pages are much higher than 0x1000000. > The value 0x1000000 for _pa(_text) doesn't match reality. Maybe there > are some expectations for the ACPI NVS and other reserved regions to > be 1-1 mapped? I tried removing the BUG mentioned above, but it still > failed to boot. I think I also removed a second BUG, but > unfortunately I don't have notes on either. The _guest_ physical address is what matters here. With using the host E820 map the PV-kernel tries to rearrange its guest physical memory layout to match the E820 map. And a non-RAM GPA for the location where the kernel is located triggers the BUG. Juergen [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 3743 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 495 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2023-11-27 16:06 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2018-02-16 17:48 PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF Marek Marczykowski-Górecki
2018-02-16 17:52 ` Andrew Cooper
2018-02-16 18:51 ` Marek Marczykowski-Górecki
2018-02-16 19:02 ` Andrew Cooper
2018-02-16 19:54 ` Marek Marczykowski-Górecki
2018-02-19 17:23 ` Juergen Gross
2018-02-19 17:29 ` Marek Marczykowski-Górecki
2018-02-19 17:46 ` Juergen Gross
2018-02-19 17:49 ` Andrew Cooper
2018-02-16 21:35 ` Rich Persaud
2018-02-19 17:13 ` Roger Pau Monné
2018-02-19 17:30 ` Juergen Gross
2023-11-26 14:51 ` [Xen-devel] " Marek Marczykowski-Górecki
[not found] ` <CACHz=ZiWufUenyw_wg+QuK86+gU5RZNkuJNzX9-K1UM5P3m8+Q@mail.gmail.com>
2023-11-27 11:26 ` Marek Marczykowski-Górecki
2023-11-27 15:56 ` Jason Andryuk
2023-11-27 16:05 ` Juergen Gross
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).