From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alvin Starr Subject: Re: Nested virtualization off VMware vSphere 6.0 with EL6 guests crashes on Xen 4.6 Date: Tue, 12 Jan 2016 09:18:40 -0500 Message-ID: <56950B40.2000301@netvel.net> References: <20160112033844.GB15551@char.us.oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20160112033844.GB15551@char.us.oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org Insure that memory and maxmem are set to the same value. On 01/11/2016 10:38 PM, Konrad Rzeszutek Wilk wrote: > Hey, > > The machine is an X5-2 which is a Haswell based E5-2699 v3. > > We are trying to launch to use the nested virtualization. The > guest is a simple VMware vSphere 6.0 with 32GB, 8 CPUs. > > The guest than that is launched within VMware is a 2 VCPU 2GB Linux > (OEL6 to be exact). During its bootup Xen crashes with this assert. > > Oddly enough if this is repeated on a workstation Ivy Bridge CPU (i5-3570) > it works fine. > > Disabling APICv (apicv=0) on the Xen command line did not help. > > I added some debug code to see if the vapic_pg is bad and what > the p2mt type is [read below] > > > Serial console started. To stop, type ESC ( > (XEN) Assertion 'vapic_pg && !p2m_is_paging(p2mt)' failed at vvmx.c:698 > (XEN) ----[ Xen-4.6.0 x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 39 > (XEN) RIP: e008:[] virtual_vmentry+0x487/0xac9 > (XEN) RFLAGS: 0000000000010246 CONTEXT: hypervisor (d1v3) > (XEN) rax: 0000000000000000 rbx: ffff83007786c000 rcx: 0000000000000000 > (XEN) rdx: 0000000000000e00 rsi: 000fffffffffffff rdi: ffff83407f81e010 > (XEN) rbp: ffff834008a47ea8 rsp: ffff834008a47e38 r8: 0000000000000000 > (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 > (XEN) r12: 0000000000000000 r13: ffff82c000341000 r14: ffff834008a47f18 > (XEN) r15: ffff83407f7c4000 cr0: 0000000080050033 cr4: 00000000001526e0 > (XEN) cr3: 000000407fb22000 cr2: 0000000000000000 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 > (XEN) Xen stack trace from rsp=ffff834008a47e38: > (XEN) ffff834008a47e68 ffff82d0801d2cde ffff834008a47e68 0000000000000d00 > (XEN) 0000000000000000 0000000000000000 ffff834008a47e88 00000004801cc30e > (XEN) ffff83007786c000 ffff83007786c000 ffff834008a40000 0000000000000000 > (XEN) ffff834008a47f18 0000000000000000 ffff834008a47f08 ffff82d0801edf94 > (XEN) ffff834008a47ef8 0000000000000000 ffff834008f62000 ffff834008a47f18 > (XEN) 000000ae8c99eb8d ffff83007786c000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 ffff82d0801ee2ab > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 0000000000000000 0000000000000000 > (XEN) 00000000078bfbff 0000000000000000 0000000000000000 0000beef0000beef > (XEN) fffffffffc4b3440 000000bf0000beef 0000000000040046 fffffffffc607f00 > (XEN) 000000000000beef 000000000000beef 000000000000beef 000000000000beef > (XEN) 000000000000beef 0000000000000027 ffff83007786c000 0000006f88716300 > (XEN) 0000000000000000 > (XEN) Xen call trace: > (XEN) [] virtual_vmentry+0x487/0xac9 > (XEN) [] nvmx_switch_guest+0x8ff/0x915 > (XEN) [] vmx_asm_vmexit_handler+0x4b/0xc0 > (XEN) > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 39: > (XEN) Assertion 'vapic_pg && !p2m_is_paging(p2mt)' failed at vvmx.c:698 > (XEN) **************************************** > (XEN) > > ..and then to my surprise the hypervisor stopped hitting this. Instead > I started getting an even more bizzare crash: > > > (d1) enter handle_19: > (d1) NULL > (d1) Booting from Hard Disk... > (d1) Booting from 0000:7c00 > (XEN) stdvga.c:151:d1v0 leaving stdvga mode > (XEN) stdvga.c:147:d1v0 entering stdvga and caching modes > (XEN) stdvga.c:520:d1v0 leaving caching mode > (XEN) ----[ Xen-4.6.0 x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 3 > (XEN) RIP: e008:[] vmx_cpu_up+0xacc/0xba5 > (XEN) RFLAGS: 0000000000010242 CONTEXT: hypervisor (d1v1) > (XEN) rax: 0000000000000000 rbx: ffff830077877000 rcx: ffff834077e54000 > (XEN) rdx: ffff834007dc8000 rsi: 0000000000002000 rdi: ffff830077877000 > (XEN) rbp: ffff834007dcfc48 rsp: ffff834007dcfc38 r8: 0000000004040000 > (XEN) r9: 000ffffffffff000 r10: 0000000000000000 r11: fffffffffc423f1e > (XEN) r12: 0000000000002000 r13: 0000000000000000 r14: 0000000000000000 > (XEN) r15: 0000000000000000 cr0: 0000000080050033 cr4: 00000000001526e0 > (XEN) cr3: 0000004000763000 cr2: 0000000000000000 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 > (XEN) Xen stack trace from rsp=ffff834007dcfc38: > (XEN) ffff834007dcfc98 0000000000000000 ffff834007dcfc68 ffff82d0801e2533 > (XEN) ffff830077877000 0000000000002000 ffff834007dcfc78 ffff82d0801ea933 > (XEN) ffff834007dcfca8 ffff82d0801eaae4 0000000000000000 ffff830077877000 > (XEN) 0000000000000000 ffff834007dcff18 ffff834007dcfd08 ffff82d0801eb983 > (XEN) ffff834000000001 000000013692c000 ffff834000000000 fffffffffc607f28 > (XEN) 0000000000000008 ffff834000000006 ffff834007dcff18 ffff830077877000 > (XEN) 0000000000000015 0000000000000000 ffff834007dcff08 ffff82d0801e8c8d > (XEN) ffff834007763000 ffff8300778c2000 ffff8340007c3000 ffff834007dcfd50 > (XEN) ffff82d0801e120b ffff834007dcfd50 ffff830077877000 ffff834007dcfdf0 > (XEN) 0000000000000000 0000000000000000 ffff82d08012fe0b ffff834007dfcac0 > (XEN) ffff834007dd30e8 0000000000000086 ffff834007dcfda0 ffff82d08012d4c2 > (XEN) ffff834000000003 0000000000000008 0000000000000000 0000000000000000 > (XEN) 0000000000000000 ffff834007dcfdf0 ffff8300778c2000 ffff830077877000 > (XEN) ffff834007dd30c8 00000083aa72fdd8 0000000000000001 ffff834007dcfe90 > (XEN) 0000000000000286 ffff834007dcfe18 ffff82d08012d4c2 ffff830077877000 > (XEN) ffff834007dcfe88 ffff82d0801d67b2 92e004e300000002 ffff830077877560 > (XEN) ffff834007dcfe68 ffff82d0801d2cbe ffff834007dcfe68 ffff830077877000 > (XEN) ffff8340007c3000 0000439115b27100 ffff834007dcfe88 ffff82d0801cc2ee > (XEN) ffff830077877000 0000000000000100 ffff834007dcff08 ffff82d0801dfd2a > (XEN) ffff834007dcff18 ffff830077877000 ffff834007dcff08 ffff82d0801e6f09 > (XEN) Xen call trace: > (XEN) [] vmx_cpu_up+0xacc/0xba5 > (XEN) [] virtual_vmcs_vmread+0x1c/0x3f > (XEN) [] get_vvmcs_real+0x9/0xb > (XEN) [] _map_io_bitmap+0x5a/0x9f > (XEN) [] nvmx_handle_vmptrld+0xd5/0x201 > (XEN) [] vmx_vmexit_handler+0x1253/0x19d4 > (XEN) [] vmx_asm_vmexit_handler+0x41/0xc0 > (XEN) > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 3: > (XEN) FATAL TRAP: vector = 6 (invalid opcode) > (XEN) **************************************** > (XEN) > (XEN) Manual reset required ('noreboot' specified) > > With the stack and gdb and following it I see: > (gdb) x/20i virtual_vmcs_vmread > 0xffff82d0801e2517 : push %rbp > 0xffff82d0801e2518 : mov %rsp,%rbp > 0xffff82d0801e251b : sub $0x10,%rsp > 0xffff82d0801e251f : mov %rbx,(%rsp) > 0xffff82d0801e2523 : mov %r12,0x8(%rsp) > 0xffff82d0801e2528 : mov %rdi,%rbx > 0xffff82d0801e252b : mov %esi,%r12d > 0xffff82d0801e252e : callq 0xffff82d0801e03f9 > 0xffff82d0801e2533 : mov %r12d,%r12d > 0xffff82d0801e2536 : vmread %r12,%r12 > 0xffff82d0801e253a : jbe 0xffff82d0801e3df3 > 0xffff82d0801e2540 : mov %rbx,%rdi > 0xffff82d0801e2543 : callq 0xffff82d0801e23f2 > 0xffff82d0801e2548 : mov %r12,%rax > 0xffff82d0801e254b : mov (%rsp),%rbx > 0xffff82d0801e254f : mov 0x8(%rsp),%r12 > 0xffff82d0801e2554 : leaveq > 0xffff82d0801e2555 : retq > 0xffff82d0801e2556 : push %rbp > 0xffff82d0801e2557 : mov %rsp,%rbp > (gdb) > (gdb) x/20i 0xffff82d0801e03f9 > 0xffff82d0801e03f9 : push %rbp > 0xffff82d0801e03fa : mov %rsp,%rbp > 0xffff82d0801e03fd : sub $0x10,%rsp > 0xffff82d0801e0401 : mov 0x5c8(%rdi),%rax > 0xffff82d0801e0408 : mov %rax,-0x8(%rbp) > 0xffff82d0801e040c : vmptrld -0x8(%rbp) > 0xffff82d0801e0410 : jbe 0xffff82d0801e3dc7 > 0xffff82d0801e0416 : leaveq > 0xffff82d0801e0417 : retq > (gdb) x/20i 0xffff82d0801e3dc7 > 0xffff82d0801e3dc7: ud2a > 0xffff82d0801e3dc9: ud2a > > static inline void __vmptrld(u64 addr) > { > asm volatile ( > #ifdef HAVE_GAS_VMX > "vmptrld %0\n" > #else > VMPTRLD_OPCODE MODRM_EAX_06 > #endif > /* CF==1 or ZF==1 --> crash (ud2) */ > UNLIKELY_START(be, vmptrld) > "\tud2\n" > UNLIKELY_END_SECTION > : > #ifdef HAVE_GAS_VMX > : "m" (addr) > #else > : "a" (&addr) > #endif > : "memory"); > } > > Thoughts? > The guest config is quite simple: > > hap=1 > nestedhvm=1 > cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'] > disk = [ 'file:/home/xen/esx2/esx2.img,hda,w','phy:/dev/mapper/vg_caex01db04-esx2,hdb,w'] > memory=32000 > vcpus=8 > name="esx2" > vif = [ 'type=ioemu,bridge=virbr0,model=vmxnet3','type=ioemu,bridge=intbr0,model=vmxnet3' ] > builder = "hvm" > device_model = "/usr/lib/xen/bin/qemu-dm" > vnc=1 > vncunused=1 > vnclisten="10.68.50.68" > apic=1 > acpi=1 > pae=1 > serial = "pty" # enable serial console > on_reboot = 'restart' > on_crash = 'restart' > > The cpuid is borrowed from: > http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel -- Alvin Starr || voice: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||