From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: Nested virtualization instabilities Date: Mon, 16 Dec 2013 13:51:27 +0000 Message-ID: <52AF055F.5070908@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2592498268020957905==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Etzion Bar-Noy , xen-devel@lists.xen.org Cc: eddie.dong@intel.com, jun.nakajima@intel.com List-Id: xen-devel@lists.xenproject.org --===============2592498268020957905== Content-Type: multipart/alternative; boundary="------------000704040604040902020409" --------------000704040604040902020409 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit On 16/12/2013 11:05, Etzion Bar-Noy wrote: > Hi. > > I am using the unstable build-from-source of Xen community. It reports > as 4.4, and was pulled from Git few days ago. The problem I will > describe below appears on version 4.3.1, compiled from source as well. > In both cases - I used spec file to create my own RPMs, because it's > the right method of doing it. > > Problem: When VM running under HVM which is a hypervisor (XenServer > 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the > following happens, 100% reproducible on this server. > XenServer: domain crash. XenServer restarts itself. It doesn't happen > for PV on XenServer VM > VMware: An "unknown failure", which leads me to think that the nested > virtualization properties are not forwarded correctly to DomU > > > (XEN) error code 7 > (XEN) domain_crash_sync called from vmcs.c:1293 > (XEN) Domain 8 (vcpu#0) crashed on cpu#4: > (XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]---- > (XEN) CPU: 4 > (XEN) RIP: 0000:[<0000000000000000>] > (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest > (XEN) rax: 0000000000000000 rbx: ffff830824f98000 rcx: > ffff830824f9ff80 > (XEN) rdx: ffff82d0801d0cf9 rsi: 0000000000000000 rdi: > ffff82d0801de7fc > (XEN) rbp: ffff82d080105ab1 rsp: 0000000000000000 r8: > 0000000000000004 > (XEN) r9: ffff82d080105b20 r10: ffff830824f9ff70 r11: > 0000000000000000 > (XEN) r12: ffff830824f9ff50 r13: ffff830654f9e000 r14: > ffff830824f9ff30 > (XEN) r15: ffff82d080187e43 cr0: 0000000000000039 cr4: > 0000000000002050 > (XEN) cr3: 00000000feffa000 cr2: 0000000000000000 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: 0000 This is the problem. Error code 7 is defined as "VM entry with invalid control field(s)" At a guess, I would think that the L1 hypervisor is probing the control fields (as that seems to be the only way of detecting whether a feature is available or not), and this error might need bouncing back to the VM. Otherwise, the L1 hypervisor has decided it can use a particular control, and Xen has insufficient validation if the L1 state. Either way, it would be useful to know if there is a way for the hardware to identify which control field it has an issue with... ~Andrew --------------000704040604040902020409 Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit

On 16/12/2013 11:05, Etzion Bar-Noy wrote:

Hi.

I am using the unstable build-from-source of Xen community. It reports as 4.4, and was pulled from Git few days ago. The problem I will describe below appears on version 4.3.1, compiled from source as well. In both cases - I used spec file to create my own RPMs, because it's the right method of doing it.

Problem: When VM running under HVM which is a hypervisor (XenServer 6.2 or VMware ESXi5.5) attempts to start a (HVM) virtual machine, the following happens, 100% reproducible on this server.

XenServer: domain crash. XenServer restarts itself. It doesn't happen for PV on XenServer VM

VMware: An "unknown failure", which leads me to think that the nested virtualization properties are not forwarded correctly to DomU

<snip>

(XEN) <vm_launch_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1293
(XEN) Domain 8 (vcpu#0) crashed on cpu#4:
(XEN) ----[ Xen-4.4-unstable x86_64 debug=y Not tainted ]----
(XEN) CPU:    4
(XEN) RIP:    0000:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest
(XEN) rax: 0000000000000000   rbx: ffff830824f98000   rcx: ffff830824f9ff80
(XEN) rdx: ffff82d0801d0cf9   rsi: 0000000000000000   rdi: ffff82d0801de7fc
(XEN) rbp: ffff82d080105ab1   rsp: 0000000000000000   r8: 0000000000000004
(XEN) r9: ffff82d080105b20   r10: ffff830824f9ff70   r11: 0000000000000000
(XEN) r12: ffff830824f9ff50   r13: ffff830654f9e000   r14: ffff830824f9ff30
(XEN) r15: ffff82d080187e43   cr0: 0000000000000039   cr4: 0000000000002050
(XEN) cr3: 00000000feffa000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000

This is the problem.

Error code 7 is defined as "VM entry with invalid control field(s)"

At a guess, I would think that the L1 hypervisor is probing the control fields (as that seems to be the only way of detecting whether a feature is available or not), and this error might need bouncing back to the VM.

Otherwise, the L1 hypervisor has decided it can use a particular control, and Xen has insufficient validation if the L1 state.

Either way, it would be useful to know if there is a way for the hardware to identify which control field it has an issue with...

~Andrew
--------------000704040604040902020409-- --===============2592498268020957905== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2592498268020957905==--