From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Nested virtualization off VMware vSphere 6.0 with EL6 guests crashes on Xen 4.6 Date: Wed, 3 Feb 2016 10:07:27 -0500 Message-ID: <20160203150727.GC20732@char.us.oracle.com> References: <20160112033844.GB15551@char.us.oracle.com> <5694D3CB02000078000C5D00@prv-mh.provo.novell.com> <20160115213958.GA16118@char.us.oracle.com> <569CC17002000078000C7D91@prv-mh.provo.novell.com> <20160202220545.GA9915@char.us.oracle.com> <56B1D7C702000078000CDDAA@prv-mh.provo.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1aQz1t-0002Ke-F1 for xen-devel@lists.xenproject.org; Wed, 03 Feb 2016 15:07:37 +0000 Content-Disposition: inline In-Reply-To: <56B1D7C702000078000CDDAA@prv-mh.provo.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: andrew.cooper3@citrix.com, kevin.tian@intel.com, wim.coekaerts@oracle.com, jun.nakajima@intel.com, xen-devel List-Id: xen-devel@lists.xenproject.org On Wed, Feb 03, 2016 at 02:34:47AM -0700, Jan Beulich wrote: > >>> On 02.02.16 at 23:05, wrote: > > This is getting more and more bizzare. > > > > I realized that this machine has VMCS shadowing so Xen does not trap on > > any vmwrite or vmread. Unless I update the VMCS shadowing bitmap - which > > I did for vmwrite and vmread to get a better view of this. It never > > traps on VIRTUAL_APIC_PAGE_ADDR accesses. It does trap on: > > VIRTUAL_PROCESSOR_ID, > > VM_EXIT_MSR_LOAD_ADDR and GUEST_[ES,DS,FS,GS,TR]_SELECTORS. > > > > (It may also trap on IO_BITMAP_A,B but I didn't print that out). > > > > To confirm that the VMCS that will be given to the L2 guest is correct > > I added some printking of some states that ought to be pretty OK such > > as HOST_RIP or HOST_RSP - which are all 0! > > But did you also check what the field of interest starts out as? I will do that. > > > If I let the nvmx_update_virtual_apic_address keep on going without > > modifying the VIRTUAL_APIC_PAGE_ADDR it later on crashes the nested > > guest: > > > > EN) nvmx_handle_vmwrite: 0 > > The missing characters at the beginning may just be a copy-and- > paste mistake, but they could also indicate a truncated log. Can > you clarify which of the two it is? Just an copy-n-paste error. Nothing of interest before there: (d1) NULL (d1) Booting from Hard Disk... (d1) Booting from 0000:7c00 (XEN) nvmx_handle_vmwrite: 0 (XEN) nvmx_handle_vmwrite: 0 .. > > > (XEN) nvmx_handle_vmwrite: 0 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 0 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 0 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 2008 > > (XEN) nvmx_handle_vmwrite: 800 > > (XEN) nvmx_handle_vmwrite: 804 > > (XEN) nvmx_handle_vmwrite: 806 > > (XEN) nvmx_handle_vmwrite: 80a > > (XEN) nvmx_handle_vmwrite: 80e > > (XEN) nvmx_update_virtual_apic_address: vCPU1 0xffffffffffffffff(vAPIC) 0x0(APIC), 0x0(TPR) ctrl=b5b9effe sec=0 > > Assuming the field starts out as other than all ones, could you check > its value on each of the intercepted VMWRITEs, to at least narrow > when it changes. Yes of course. > > Kevin, Jun - are there any cases where the hardware would alter > this field's value? Like during some guest side LAPIC manipulations? > (The same monitoring as suggested during VMWRITEs could of > course also be added to LAPIC accesses visible to the hypervisor, > but I guess there won't be too many of those.) > > Jan >