Re: Nested virtualization off VMware vSphere 6.0 with EL6 guests crashes on Xen 4.6

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Feb 03, 2016 at 02:34:47AM -0700, Jan Beulich wrote:
> >>> On 02.02.16 at 23:05, <konrad.wilk@oracle.com> wrote:
> > This is getting more and more bizzare.
> > 
> > I realized that this machine has VMCS shadowing so Xen does not trap on
> > any vmwrite or vmread. Unless I update the VMCS shadowing bitmap - which
> > I did for vmwrite and vmread to get a better view of this. It never
> > traps on VIRTUAL_APIC_PAGE_ADDR accesses. It does trap on: 
> > VIRTUAL_PROCESSOR_ID,
> > VM_EXIT_MSR_LOAD_ADDR and GUEST_[ES,DS,FS,GS,TR]_SELECTORS.
> > 
> > (It may also trap on IO_BITMAP_A,B but I didn't print that out).
> > 
> > To confirm that the VMCS that will be given to the L2 guest is correct
> > I added some printking of some states that ought to be pretty OK such
> > as HOST_RIP or HOST_RSP - which are all 0!
> 
> But did you also check what the field of interest starts out as?

I will do that.
> 
> > If I let the nvmx_update_virtual_apic_address keep on going without
> > modifying the VIRTUAL_APIC_PAGE_ADDR it later on crashes the nested
> > guest:
> > 
> > EN) nvmx_handle_vmwrite: 0                                                   
> 
> The missing characters at the beginning may just be a copy-and-
> paste mistake, but they could also indicate a truncated log. Can
> you clarify which of the two it is?

Just an copy-n-paste error. Nothing of interest before there:
(d1)   NULL                                                                        
(d1) Booting from Hard Disk...                                                     
(d1) Booting from 0000:7c00                                                        
(XEN) nvmx_handle_vmwrite: 0                                                       
(XEN) nvmx_handle_vmwrite: 0                
..
> 
> > (XEN) nvmx_handle_vmwrite: 0                                                 
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 0                                                 
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 0                                                 
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 2008                                              
> > (XEN) nvmx_handle_vmwrite: 800                                               
> > (XEN) nvmx_handle_vmwrite: 804                                               
> > (XEN) nvmx_handle_vmwrite: 806                                               
> > (XEN) nvmx_handle_vmwrite: 80a                                               
> > (XEN) nvmx_handle_vmwrite: 80e                                               
> > (XEN) nvmx_update_virtual_apic_address: vCPU1 0xffffffffffffffff(vAPIC) 0x0(APIC), 0x0(TPR) ctrl=b5b9effe sec=0 
> 
> Assuming the field starts out as other than all ones, could you check
> its value on each of the intercepted VMWRITEs, to at least narrow
> when it changes.

Yes of course.
> 
> Kevin, Jun - are there any cases where the hardware would alter
> this field's value? Like during some guest side LAPIC manipulations?
> (The same monitoring as suggested during VMWRITEs could of
> course also be added to LAPIC accesses visible to the hypervisor,
> but I guess there won't be too many of those.)
> 
> Jan
>