Xen-4.4-rc2 - Some Nested Virt testing

Xen-4.4-rc2 - Some Nested Virt testing

[Andrew Cooper]

Hello,

I have been giving nested virt a try, and have my first bug to report. 
This is still ongoing, and is by no means complete yet.

Setup:
Each reference to XenServer is a trunk XenServer based on 4.4-rc2

Single Intel Haswell SDP (Grantley platform):
Native hypervisor: XenServer

Two L1 guests:
  XenServer (running with EPT)
  XenServer (running with shadow)


When attempting to create an L2 EPT HVM domain under an L1 shadow
domain, the L1 shadow domain is killed with:

(XEN) <vm_launch_fail> error code 7
(XEN) domain_crash_sync called from vmcs.c:1293
(XEN) Domain 16 (vcpu#3) crashed on cpu#2:
(XEN) ----[ Xen-4.4.0-xs82349-d  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    2
(XEN) RIP:    0000:[<0000000000000000>]
(XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest
(XEN) rax: 0000000000000000   rbx: ffff83043cad8000   rcx: ffff83043cadff80
(XEN) rdx: ffff82d0801d6ea0   rsi: 0000000000000000   rdi: ffff82d0801e2e8c
(XEN) rbp: ffff82d080105680   rsp: 0000000000000000   r8:  ffff830064100000
(XEN) r9:  ffff82d0801056ee   r10: ffff83043cadff70   r11: 0000000000000000
(XEN) r12: ffff83043cadff50   r13: ffff830441e42000   r14: ffff830064100000
(XEN) r15: ffff82d080189425   cr0: 0000000000000039   cr4: 0000000000002050
(XEN) cr3: 0000000000000000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000


I am continuing experiments with different VMs under each L1 hypervisor,
to see what else breaks.

~Andrew

Re: Xen-4.4-rc2 - Some Nested Virt testing

[Andrew Cooper]

On 21/01/2014 18:12, Andrew Cooper wrote:
> Hello,
>
> I have been giving nested virt a try, and have my first bug to report. 
> This is still ongoing, and is by no means complete yet.
>
> Setup:
> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
>
> Single Intel Haswell SDP (Grantley platform):
> Native hypervisor: XenServer
>
> Two L1 guests:
>   XenServer (running with EPT)
>   XenServer (running with shadow)
>
>
> When attempting to create an L2 EPT HVM domain under an L1 shadow
> domain, the L1 shadow domain is killed with:
>
> (XEN) <vm_launch_fail> error code 7
> (XEN) domain_crash_sync called from vmcs.c:1293
> (XEN) Domain 16 (vcpu#3) crashed on cpu#2:
> (XEN) ----[ Xen-4.4.0-xs82349-d  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    2
> (XEN) RIP:    0000:[<0000000000000000>]
> (XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest
> (XEN) rax: 0000000000000000   rbx: ffff83043cad8000   rcx: ffff83043cadff80
> (XEN) rdx: ffff82d0801d6ea0   rsi: 0000000000000000   rdi: ffff82d0801e2e8c
> (XEN) rbp: ffff82d080105680   rsp: 0000000000000000   r8:  ffff830064100000
> (XEN) r9:  ffff82d0801056ee   r10: ffff83043cadff70   r11: 0000000000000000
> (XEN) r12: ffff83043cadff50   r13: ffff830441e42000   r14: ffff830064100000
> (XEN) r15: ffff82d080189425   cr0: 0000000000000039   cr4: 0000000000002050
> (XEN) cr3: 0000000000000000   cr2: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000

I suppose it is worth adding that `xl dmesg` from the L1 shadow
XenServer shows:

(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 4 CPUs

Which indicates that EPT is available even in a shadow L1 domain.  I
can't think of a technical reason why it wouldn't work.

>
>
> I am continuing experiments with different VMs under each L1 hypervisor,
> to see what else breaks.
>
> ~Andrew
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen-4.4-rc2 - Some Nested Virt testing

[Jan Beulich]

>>> On 21.01.14 at 19:12, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> I have been giving nested virt a try, and have my first bug to report. 
> This is still ongoing, and is by no means complete yet.
> 
> Setup:
> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
> 
> Single Intel Haswell SDP (Grantley platform):
> Native hypervisor: XenServer
> 
> Two L1 guests:
>   XenServer (running with EPT)
>   XenServer (running with shadow)
> 
> 
> When attempting to create an L2 EPT HVM domain under an L1 shadow
> domain, the L1 shadow domain is killed with:
> 
> (XEN) <vm_launch_fail> error code 7

Considering that 7 is "VM entry with invalid control field(s)", I think
it would be quite helpful if we enhanced the error handling here to
dump the VMCS.

Also - did you perhaps mean to Cc VMX folks on your original mail?
Chances that they see your report without doing so are - according
to my experience - rather slim...

Jan

Re: Xen-4.4-rc2 - Some Nested Virt testing

[Andrew Cooper]

On 22/01/14 09:38, Jan Beulich wrote:
>>>> On 21.01.14 at 19:12, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> I have been giving nested virt a try, and have my first bug to report. 
>> This is still ongoing, and is by no means complete yet.
>>
>> Setup:
>> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
>>
>> Single Intel Haswell SDP (Grantley platform):
>> Native hypervisor: XenServer
>>
>> Two L1 guests:
>>   XenServer (running with EPT)
>>   XenServer (running with shadow)
>>
>>
>> When attempting to create an L2 EPT HVM domain under an L1 shadow
>> domain, the L1 shadow domain is killed with:
>>
>> (XEN) <vm_launch_fail> error code 7
> Considering that 7 is "VM entry with invalid control field(s)", I think
> it would be quite helpful if we enhanced the error handling here to
> dump the VMCS.

Agreed.  I cannot find any further help from hardware to identify which
control field(s) is(are) invalid, so the best we appear to be able to
know is "At least one of these bits are wrong in the current context".

>
> Also - did you perhaps mean to Cc VMX folks on your original mail?
> Chances that they see your report without doing so are - according
> to my experience - rather slim...
>
> Jan
>
>

I wasn't really thinking that much - I had hoped to also try out
nested-virt on AMD, but have completely run out of time.

After 4.4 gets released, I will try to automate the environment setup,
and start investigating/reporting the encountered issues properly.

Until then sadly, I have more important issues to work on in the meantime.

~Andrew

Re: Xen-4.4-rc2 - Some Nested Virt testing

[George Dunlap]

Cc'ing the guy working on nested virt...

On Tue, Jan 21, 2014 at 6:12 PM, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> Hello,
>
> I have been giving nested virt a try, and have my first bug to report.
> This is still ongoing, and is by no means complete yet.
>
> Setup:
> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
>
> Single Intel Haswell SDP (Grantley platform):
> Native hypervisor: XenServer
>
> Two L1 guests:
>   XenServer (running with EPT)
>   XenServer (running with shadow)
>
>
> When attempting to create an L2 EPT HVM domain under an L1 shadow
> domain, the L1 shadow domain is killed with:
>
> (XEN) <vm_launch_fail> error code 7
> (XEN) domain_crash_sync called from vmcs.c:1293
> (XEN) Domain 16 (vcpu#3) crashed on cpu#2:
> (XEN) ----[ Xen-4.4.0-xs82349-d  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    2
> (XEN) RIP:    0000:[<0000000000000000>]
> (XEN) RFLAGS: 0000000000000002   CONTEXT: hvm guest
> (XEN) rax: 0000000000000000   rbx: ffff83043cad8000   rcx: ffff83043cadff80
> (XEN) rdx: ffff82d0801d6ea0   rsi: 0000000000000000   rdi: ffff82d0801e2e8c
> (XEN) rbp: ffff82d080105680   rsp: 0000000000000000   r8:  ffff830064100000
> (XEN) r9:  ffff82d0801056ee   r10: ffff83043cadff70   r11: 0000000000000000
> (XEN) r12: ffff83043cadff50   r13: ffff830441e42000   r14: ffff830064100000
> (XEN) r15: ffff82d080189425   cr0: 0000000000000039   cr4: 0000000000002050
> (XEN) cr3: 0000000000000000   cr2: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000
>
>
> I am continuing experiments with different VMs under each L1 hypervisor,
> to see what else breaks.
>
> ~Andrew
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Xen-4.4-rc2 - Some Nested Virt testing

[George Dunlap]

On Tue, Jan 21, 2014 at 6:12 PM, Andrew Cooper
<andrew.cooper3@citrix.com> wrote:
> Hello,
>
> I have been giving nested virt a try, and have my first bug to report.
> This is still ongoing, and is by no means complete yet.
>
> Setup:
> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
>
> Single Intel Haswell SDP (Grantley platform):
> Native hypervisor: XenServer
>
> Two L1 guests:
>   XenServer (running with EPT)
>   XenServer (running with shadow)
>
>
> When attempting to create an L2 EPT HVM domain under an L1 shadow
> domain, the L1 shadow domain is killed with:

Is EPT-on-shadow actually meant to work?  I wouldn't be surprised if
the L2 HAP stuff assumed that L1 was HAP as well.

In which case, if an L1 guest is started in shadow mode, then EPT
should not be advertised.

 -George

Re: Xen-4.4-rc2 - Some Nested Virt testing

[Zhang, Yang Z]

George Dunlap wrote on 2014-01-25:
> On Tue, Jan 21, 2014 at 6:12 PM, Andrew Cooper
> <andrew.cooper3@citrix.com> wrote:
>> Hello,
>> 
>> I have been giving nested virt a try, and have my first bug to report.
>> This is still ongoing, and is by no means complete yet.
>> 
>> Setup:
>> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
>> 
>> Single Intel Haswell SDP (Grantley platform):
>> Native hypervisor: XenServer
>> 
>> Two L1 guests:
>>   XenServer (running with EPT)
>>   XenServer (running with shadow)
>> 
>> When attempting to create an L2 EPT HVM domain under an L1 shadow
>> domain, the L1 shadow domain is killed with:
> 
> Is EPT-on-shadow actually meant to work?  I wouldn't be surprised if
> the L2 HAP stuff assumed that L1 was HAP as well.
> 
> In which case, if an L1 guest is started in shadow mode, then EPT
> should not be advertised.

AFAK, EPT-on-shadow is not supported. Shadow-on-shadow is buggy (Actually, I never tried it successfully from the first day I start working on nested stuff). Shadow-on-EPT and EPT-on-EPT are working in my box. But I recommended using EPT on EPT if possible. Because it is really a pain to run L2 guest on shadow on shadow mode due to the poor performance.

> 
>  -George


Best regards,
Yang

Re: Xen-4.4-rc2 - Some Nested Virt testing

[Tim Deegan]

B0;278;0cAt 08:29 +0000 on 26 Jan (1390721344), Zhang, Yang Z wrote:
> George Dunlap wrote on 2014-01-25:
> > On Tue, Jan 21, 2014 at 6:12 PM, Andrew Cooper
> > <andrew.cooper3@citrix.com> wrote:
> >> Hello,
> >> 
> >> I have been giving nested virt a try, and have my first bug to report.
> >> This is still ongoing, and is by no means complete yet.
> >> 
> >> Setup:
> >> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
> >> 
> >> Single Intel Haswell SDP (Grantley platform):
> >> Native hypervisor: XenServer
> >> 
> >> Two L1 guests:
> >>   XenServer (running with EPT)
> >>   XenServer (running with shadow)
> >> 
> >> When attempting to create an L2 EPT HVM domain under an L1 shadow
> >> domain, the L1 shadow domain is killed with:
> > 
> > Is EPT-on-shadow actually meant to work?  I wouldn't be surprised if
> > the L2 HAP stuff assumed that L1 was HAP as well.
> > 
> > In which case, if an L1 guest is started in shadow mode, then EPT
> > should not be advertised.
> 
> AFAK, EPT-on-shadow is not supported. Shadow-on-shadow is buggy
> (Actually, I never tried it successfully from the first day I start
> working on nested stuff).

Fair enough.  That needs to be documented, and those modes (which I
guess means nested-on-shadow in general) need to be disabled in the
hypervisor, with a sensible error message.

> Shadow-on-EPT and EPT-on-EPT are working
> in my box. But I recommended using EPT on EPT if possible. Because
> it is really a pain to run L2 guest on shadow on shadow mode due to
> the poor performance.

Yeah, I think it's generally accepted that having shadow pagetables
anywhere in that stack is going to hurt.  Sadly, there's no way for
the L0 admin to stop the L1 hypervisor from using shadow pagetables,
so shadow-on-EPT ought to at least work correctly, even if performance
sucks.

Tim.

Re: Xen-4.4-rc2 - Some Nested Virt testing

[Zhang, Yang Z]

Tim Deegan wrote on 2014-01-27:
> B0;278;0cAt 08:29 +0000 on 26 Jan (1390721344), Zhang, Yang Z wrote:
>> George Dunlap wrote on 2014-01-25:
>>> On Tue, Jan 21, 2014 at 6:12 PM, Andrew Cooper
>>> <andrew.cooper3@citrix.com> wrote:
>>>> Hello,
>>>> 
>>>> I have been giving nested virt a try, and have my first bug to report.
>>>> This is still ongoing, and is by no means complete yet.
>>>> 
>>>> Setup:
>>>> Each reference to XenServer is a trunk XenServer based on 4.4-rc2
>>>> 
>>>> Single Intel Haswell SDP (Grantley platform):
>>>> Native hypervisor: XenServer
>>>> 
>>>> Two L1 guests:
>>>>   XenServer (running with EPT)
>>>>   XenServer (running with shadow)
>>>> When attempting to create an L2 EPT HVM domain under an L1 shadow
>>>> domain, the L1 shadow domain is killed with:
>>> 
>>> Is EPT-on-shadow actually meant to work?  I wouldn't be surprised
>>> if the L2 HAP stuff assumed that L1 was HAP as well.
>>> 
>>> In which case, if an L1 guest is started in shadow mode, then EPT
>>> should not be advertised.
>> 
>> AFAK, EPT-on-shadow is not supported. Shadow-on-shadow is buggy
>> (Actually, I never tried it successfully from the first day I start
>> working on nested stuff).
> 
> Fair enough.  That needs to be documented, and those modes (which I
> guess means nested-on-shadow in general) need to be disabled in the
> hypervisor, with a sensible error message.
> 

Yes, I am working on writing the wiki page.

>> Shadow-on-EPT and EPT-on-EPT are working in my box. But I
>> recommended using EPT on EPT if possible. Because it is really a
>> pain to run L2 guest on shadow on shadow mode due to the poor performance.
> 
> Yeah, I think it's generally accepted that having shadow pagetables
> anywhere in that stack is going to hurt.  Sadly, there's no way for
> the L0 admin to stop the L1 hypervisor from using shadow pagetables,
> so shadow-on-EPT ought to at least work correctly, even if performance sucks.
> 
> Tim.


Best regards,
Yang