Xen 4.4 development update

Xen 4.4 development update

[Ian Campbell]

I'm filing in for George while he is on vacation and travelling to a
conference etc. I'm still coming up to speed wrt what is going on with
this release so please do correct me when I'm wrong. George will be
back on 20 January.

This information will be mirrored on the Xen 4.4 Roadmap wiki page:
 http://wiki.xen.org/wiki/Xen_Roadmap/4.4

We tagged 4.4.0-rc1 on 19 December. Based on the conversation had last
time and on George's final comments in [1] I think this means that PVH
dom0 support has not made the cut for 4.4, which is a shame but there
is plenty of good functionality (including PVH domU support) in there.

[1] http://bugs.xenproject.org/xen/mid/%3C52B05C0A.4040404@eu.citrix.com%3E

= Timeline =

Here is our current timeline based on a 6-month release:

* Feature freeze: 18 October 2013 
* Code freezing point: 18 November 2013
* First RCs: 6 December 2013  <== WE ARE HERE
* Release: 21 January 2014

Last updated: 8 January 2014

== Completed ==

* Event channel scalability (FIFO event channels)

* Non-udev scripts for driver domains (non-Linux driver domains)

* Multi-vector PCI MSI (Hypervisor side)

* Improved Spice support on libxl
 - Added Spice vdagent support
 - Added Spice clipboard sharing support
 - Spice usbredirection support for upstream qemu 

* PHV domU (experimental only)

* pvgrub2 checked into grub upstream

* ARM64 guest

* Guest EFI booting (tianocore)

* kexec

* Testing: Xen on ARM

* Update to SeaBIOS 1.7.3.1

* Update to qemu 1.6

* SWIOTLB (in Linux 3.13)

* Disk: indirect descriptors (in 3.11)

* Reworked ocaml bindings 

== Resolved since last update ==

== Open ==

* xl support for vnc and vnclisten options with PV guests
 > http://bugs.xenproject.org/xen/bug/25
 status: V4 patch posted. Should go in.
 Blocker?

* libxl / xl does not handle failure of remote qemu gracefully
  > Related to http://bugs.xenproject.org/xen/bug/29
  > Easiest way to reproduce: 
  >  - set "vncunused=0" and do a local migrate
  >  - The "remote" qemu will fail because the vnc port is in use
  > The failure isn't the problem, but everything being stuck afterwards is
 Ian J investigating

* xl needs to disallow PoD with PCI passthrough
  >see http://xen.1045712.n5.nabble.com/PATCH-VT-d-Dis-allow-PCI-device-assignment-if-PoD-is-enabled-td2547788.html

* qemu-upstream not freeing pirq 
 > http://www.gossamer-threads.com/lists/xen/devel/281498
 > http://marc.info/?l=xen-devel&m=137265766424502
 status: patches posted; latest patches need testing
 Not a blocker.

* Race in PV shutdown between tool detection and shutdown watch
 > http://www.gossamer-threads.com/lists/xen/devel/282467
 > Nothing to do with ACPI
 status: Patches posted
 Not a blocker.

* xl does not support specifying virtual function for passthrough device
 > http://bugs.xenproject.org/xen/bug/22
 Too much work to be a blocker.

* xl does not handle migrate interruption gracefully
  > If you start a localhost migrate, and press "Ctrl-C" in the middle,
  > you get two hung domains
 Ian J investigated -- can of worms, too big to be a blocker for 4.4

* Win2k3 SP2 RTC infinite loops
   > Regression introduced late in Xen-4.3 development
   owner: andrew.cooper@citrix
   status: patches posted, undergoing review. ( v2 ID
1386241748-9617-1-git-send-email-andrew.cooper3@citrix.com )

  > andyhhp: my proposed RTC fixes break migrate from older versions of
  > Xen, so I have to redesign it from scratch. no way it is going to
  > be ready for 4.4

* HPET interrupt stack overflow (when using hpet_broadcast mode and MSI
capable HPETs)
  owner: andyh@citrix
  status: patches posted, undergoing review iteration.

  > andyhhp: I have more work to do on the HPET series
  > andyhhp: no way it is going to be ready or safe for 4.4

* PCI hole resize support hvmloader/qemu-traditional/qemu-upstream with PCI/GPU passthrough
  > http://bugs.xenproject.org/xen/bug/28
  > http://lists.xen.org/archives/html/xen-devel/2013-05/msg02813.html
  > Where Stefano writes:
  > 2) for Xen 4.4 rework the two patches above and improve
  > i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
  > enough, it also needs to be able to resize the system memory region
  > (xen.ram) to make room for the bigger pci_hole

  status: not going to be fixed for 4.4 either. Created bug #28.

* qemu memory leak?
  > http://lists.xen.org/archives/html/xen-users/2013-03/msg00276.html

* qemu-* parses "008" as octal in USB bus.addr format
  > http://bugs.xenproject.org/xen/bug/15
  > just needs documenting
  Anthony Perard to patch docs

* osstest windows-install failures
  > http://bugs.xenproject.org/xen/bug/29
  > http://www.chiark.greenend.org.uk/~xensrcts/logs/24250/
  Anthony and/or Jan investigating

=== Big ticket items ===

* PVH dom0 (w/ Linux) 
  blocker
  owner: mukesh@oracle, george@citrix
  status (Linux): Acked, waiting for ABI to be nailed down
  status (Xen): v6 posted; no longer considered a blocker

* libvirt/libxl integration (external)
 - owner: jfehlig@suse, dario@citrix
 - patches posted (should be released before 4.4)
  - migration
  - PCI pass-through
 - In progress
  - integration w/ libvirt's lock manager
  - improved concurrency

qemu-upstream not freeing pirq (Was: Re: Xen 4.4 development update)

[Ian Campbell]

On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> 
> * qemu-upstream not freeing pirq 
>  > http://www.gossamer-threads.com/lists/xen/devel/281498
>  > http://marc.info/?l=xen-devel&m=137265766424502
>  status: patches posted; latest patches need testing
>  Not a blocker.

I had it in my mind that this was fixed -- true?

Ian.

Race in PV shutdown between tool detection and shutdown watch (Was: Re: Xen 4.4 development update)

[Ian Campbell]

On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:

> * Race in PV shutdown between tool detection and shutdown watch
>  > http://www.gossamer-threads.com/lists/xen/devel/282467
>  > Nothing to do with ACPI
>  status: Patches posted
>  Not a blocker.

This is a Linux issue, I think? Did those patches go in?

Ian.

Re: Race in PV shutdown between tool detection and shutdown watch (Was: Re: Xen 4.4 development update)

[David Vrabel]

On 08/01/14 13:19, Ian Campbell wrote:
> On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> 
>> * Race in PV shutdown between tool detection and shutdown watch
>>  > http://www.gossamer-threads.com/lists/xen/devel/282467
>>  > Nothing to do with ACPI
>>  status: Patches posted
>>  Not a blocker.
> 
> This is a Linux issue, I think? Did those patches go in?

Konrad had a series.  I don't recall their status.  I think they needed
some more work.

David

Re: qemu-upstream not freeing pirq (Was: Re: Xen 4.4 development update)

[Stefano Stabellini]

On Wed, 8 Jan 2014, Ian Campbell wrote:
> On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> > 
> > * qemu-upstream not freeing pirq 
> >  > http://www.gossamer-threads.com/lists/xen/devel/281498
> >  > http://marc.info/?l=xen-devel&m=137265766424502
> >  status: patches posted; latest patches need testing
> >  Not a blocker.
> 
> I had it in my mind that this was fixed -- true?

It was fixed on qemu-traditional. We have a patch for upstream qemu but
it hasn't been tested because of the other passthrough issues.

Re: Xen 4.4 development update

[Ian Campbell]

On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> 
> * libxl / xl does not handle failure of remote qemu gracefully
>   > Related to http://bugs.xenproject.org/xen/bug/29

This should be http://bugs.xenproject.org/xen/bug/30

http://bugs.xenproject.org/xen/bug/29 is

* Windows install failures/BSOD
 > http://bugs.xenproject.org/xen/bug/29
 status: Anthony attempting to reproduce
 Blocker

which I forgot to list.

Re: Xen 4.4 development update

[Ian Campbell]

On Wed, 2014-01-08 at 13:29 +0000, Ian Campbell wrote:
> On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> > 
> > * libxl / xl does not handle failure of remote qemu gracefully
> >   > Related to http://bugs.xenproject.org/xen/bug/29
> 
> This should be http://bugs.xenproject.org/xen/bug/30
> 
> http://bugs.xenproject.org/xen/bug/29 is
> 
> * Windows install failures/BSOD
>  > http://bugs.xenproject.org/xen/bug/29
>  status: Anthony attempting to reproduce
>  Blocker
> 
> which I forgot to list.

Except I didn't -- it was at the end...

Re: qemu-upstream not freeing pirq (Was: Re: Xen 4.4 development update)

[Pasi Kärkkäinen]

On Wed, Jan 08, 2014 at 01:22:09PM +0000, Stefano Stabellini wrote:
> On Wed, 8 Jan 2014, Ian Campbell wrote:
> > On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> > > 
> > > * qemu-upstream not freeing pirq 
> > >  > http://www.gossamer-threads.com/lists/xen/devel/281498
> > >  > http://marc.info/?l=xen-devel&m=137265766424502
> > >  status: patches posted; latest patches need testing
> > >  Not a blocker.
> > 
> > I had it in my mind that this was fixed -- true?
> 
> It was fixed on qemu-traditional. We have a patch for upstream qemu but
> it hasn't been tested because of the other passthrough issues.
> 

Hmm, what other passthrough issues? Should those be added to the list aswell?

-- Pasi

Re: Xen 4.4 development update

[Sander Eikelenboom]

Wednesday, January 8, 2014, 2:16:24 PM, you wrote:

> I'm filing in for George while he is on vacation and travelling to a
> conference etc. I'm still coming up to speed wrt what is going on with
> this release so please do correct me when I'm wrong. George will be
> back on 20 January.

> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>  http://wiki.xen.org/wiki/Xen_Roadmap/4.4

> We tagged 4.4.0-rc1 on 19 December. Based on the conversation had last
> time and on George's final comments in [1] I think this means that PVH
> dom0 support has not made the cut for 4.4, which is a shame but there
> is plenty of good functionality (including PVH domU support) in there.

> [1] http://bugs.xenproject.org/xen/mid/%3C52B05C0A.4040404@eu.citrix.com%3E

> = Timeline =

> Here is our current timeline based on a 6-month release:

> * Feature freeze: 18 October 2013 
> * Code freezing point: 18 November 2013
> * First RCs: 6 December 2013  <== WE ARE HERE
> * Release: 21 January 2014

> Last updated: 8 January 2014

> == Completed ==

<snip>

> * PHV domU (experimental only)

<snip>

> === Big ticket items ===

> * PVH dom0 (w/ Linux) 
>   blocker
>   owner: mukesh@oracle, george@citrix
>   status (Linux): Acked, waiting for ABI to be nailed down
>   status (Xen): v6 posted; no longer considered a blocker

Perhaps worth noting as a separate non-blocking (for 4.5) item:
* PVH support for AMD (/SVM)

And also mention this "lack" of support in the announcement of the new experimental feature PVH.

Re: Xen 4.4 development update

[Ian Campbell]

On Wed, 2014-01-08 at 15:21 +0100, Sander Eikelenboom wrote:
> We
> Perhaps worth noting as a separate non-blocking (for 4.5) item:
> * PVH support for AMD (/SVM)

I leave that up to the 4.5 RM.

> And also mention this "lack" of support in the announcement of the new experimental feature PVH.

It is certainly worth making the point that it is Intel only in the
Release Notes or announcements etc.

Ian.

Re: qemu-upstream not freeing pirq (Was: Re: Xen 4.4 development update)

[Konrad Rzeszutek Wilk]

On Wed, Jan 08, 2014 at 03:55:25PM +0200, Pasi Kärkkäinen wrote:
> On Wed, Jan 08, 2014 at 01:22:09PM +0000, Stefano Stabellini wrote:
> > On Wed, 8 Jan 2014, Ian Campbell wrote:
> > > On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> > > > 
> > > > * qemu-upstream not freeing pirq 
> > > >  > http://www.gossamer-threads.com/lists/xen/devel/281498
> > > >  > http://marc.info/?l=xen-devel&m=137265766424502
> > > >  status: patches posted; latest patches need testing
> > > >  Not a blocker.
> > > 
> > > I had it in my mind that this was fixed -- true?
> > 
> > It was fixed on qemu-traditional. We have a patch for upstream qemu but
> > it hasn't been tested because of the other passthrough issues.
> > 
> 
> Hmm, what other passthrough issues? Should those be added to the list aswell?

http://www.gossamer-threads.com/lists/xen/devel/309476


> 
> -- Pasi
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Race in PV shutdown between tool detection and shutdown watch (Was: Re: Xen 4.4 development update)

[Konrad Rzeszutek Wilk]

On Wed, Jan 08, 2014 at 01:22:03PM +0000, David Vrabel wrote:
> On 08/01/14 13:19, Ian Campbell wrote:
> > On Wed, 2014-01-08 at 13:16 +0000, Ian Campbell wrote:
> > 
> >> * Race in PV shutdown between tool detection and shutdown watch
> >>  > http://www.gossamer-threads.com/lists/xen/devel/282467
> >>  > Nothing to do with ACPI
> >>  status: Patches posted
> >>  Not a blocker.
> > 
> > This is a Linux issue, I think? Did those patches go in?
> 
> Konrad had a series.  I don't recall their status.  I think they needed
> some more work.

<nods> David pointed out some improvements - but I had my head buried
in PVH so hadn't been able to make any progress.

They will be stalled for some time.
> 
> David

Re: Xen 4.4 development update

[Wei Liu]

On Wed, Jan 08, 2014 at 01:16:24PM +0000, Ian Campbell wrote:
[...]
> == Open ==
> 
> * xl support for vnc and vnclisten options with PV guests
>  > http://bugs.xenproject.org/xen/bug/25
>  status: V4 patch posted. Should go in.
>  Blocker?
> 

Konrad (the reporter) confirmed this is not a blocker.

Wei.

Re: Xen 4.4 development update

[Zhang, Yang Z]

Ian Campbell wrote on 2014-01-08:
> I'm filing in for George while he is on vacation and travelling to a conference etc.
> I'm still coming up to speed wrt what is going on with this release so
> please do correct me when I'm wrong. George will be back on 20 January.
> 
> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>  http://wiki.xen.org/wiki/Xen_Roadmap/4.4
> We tagged 4.4.0-rc1 on 19 December. Based on the conversation had last
> time and on George's final comments in [1] I think this means that PVH
> dom0 support has not made the cut for 4.4, which is a shame but there
> is plenty of good functionality (including PVH domU support) in there.
> 
> [1]
> http://bugs.xenproject.org/xen/mid/%3C52B05C0A.4040404@eu.citrix.com%3 E
> 
> = Timeline =
> 
> Here is our current timeline based on a 6-month release:
> 
> * Feature freeze: 18 October 2013
> * Code freezing point: 18 November 2013
> * First RCs: 6 December 2013  <== WE ARE HERE
> * Release: 21 January 2014
> 
> Last updated: 8 January 2014
> 
> == Completed ==
> 
> * Event channel scalability (FIFO event channels)
> 
> * Non-udev scripts for driver domains (non-Linux driver domains)
> 
> * Multi-vector PCI MSI (Hypervisor side)
> 
> * Improved Spice support on libxl
>  - Added Spice vdagent support
>  - Added Spice clipboard sharing support
>  - Spice usbredirection support for upstream qemu
> * PHV domU (experimental only)
> 
> * pvgrub2 checked into grub upstream
> 
> * ARM64 guest
> 
> * Guest EFI booting (tianocore)
> 
> * kexec
> 
> * Testing: Xen on ARM
> 
> * Update to SeaBIOS 1.7.3.1
> 
> * Update to qemu 1.6
> 
> * SWIOTLB (in Linux 3.13)
> 
> * Disk: indirect descriptors (in 3.11)
> 
> * Reworked ocaml bindings

Can I say nested virtualization also is good supported in Xen 4.4?

> 
> == Resolved since last update ==
> 
> == Open ==
> 
> * xl support for vnc and vnclisten options with PV guests  >
> http://bugs.xenproject.org/xen/bug/25
>  status: V4 patch posted. Should go in.
>  Blocker?
> * libxl / xl does not handle failure of remote qemu gracefully
>> Related to http://bugs.xenproject.org/xen/bug/29
>> Easiest way to reproduce:
>>  - set "vncunused=0" and do a local migrate
>>  - The "remote" qemu will fail because the vnc port is in use
>> The failure isn't the problem, but everything being stuck
> afterwards is  Ian J investigating
> 
> * xl needs to disallow PoD with PCI passthrough
>> see
> http://xen.1045712.n5.nabble.com/PATCH-VT-d-Dis-allow-PCI-device-assig
> nme nt-if-PoD-is-enabled-td2547788.html
> 
> * qemu-upstream not freeing pirq
>> http://www.gossamer-threads.com/lists/xen/devel/281498
>> http://marc.info/?l=xen-devel&m=137265766424502
>> status: patches posted; latest patches need testing  Not a blocker.
> * Race in PV shutdown between tool detection and shutdown watch  >
> http://www.gossamer-threads.com/lists/xen/devel/282467
>> Nothing to do with ACPI
>> status: Patches posted
>> Not a blocker.
> * xl does not support specifying virtual function for passthrough
> device  >
> http://bugs.xenproject.org/xen/bug/22
>  Too much work to be a blocker.
> * xl does not handle migrate interruption gracefully
>> If you start a localhost migrate, and press "Ctrl-C" in the middle,
>> you get two hung domains
>> Ian J investigated -- can of worms, too big to be a blocker for 4.4
> * Win2k3 SP2 RTC infinite loops
>> Regression introduced late in Xen-4.3 development
>    owner: andrew.cooper@citrix
>    status: patches posted, undergoing review. ( v2 ID
> 1386241748-9617-1-git-send-email-andrew.cooper3@citrix.com )
> 
>> andyhhp: my proposed RTC fixes break migrate from older versions of
>> Xen, so I have to redesign it from scratch. no way it is going to
>> be ready for 4.4
> 
> * HPET interrupt stack overflow (when using hpet_broadcast mode and
> MSI capable HPETs)
>   owner: andyh@citrix
>   status: patches posted, undergoing review iteration.
>> andyhhp: I have more work to do on the HPET series
>> andyhhp: no way it is going to be ready or safe for 4.4
> 
> * PCI hole resize support hvmloader/qemu-traditional/qemu-upstream
> with PCI/GPU passthrough
>> http://bugs.xenproject.org/xen/bug/28
>> http://lists.xen.org/archives/html/xen-devel/2013-05/msg02813.html
>> Where Stefano writes:
>> 2) for Xen 4.4 rework the two patches above and improve
>> i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
>> enough, it also needs to be able to resize the system memory region
>> (xen.ram) to make room for the bigger pci_hole
> 
>   status: not going to be fixed for 4.4 either. Created bug #28.
> * qemu memory leak?
>> http://lists.xen.org/archives/html/xen-users/2013-03/msg00276.html
> 
> * qemu-* parses "008" as octal in USB bus.addr format
>> http://bugs.xenproject.org/xen/bug/15
>> just needs documenting
>   Anthony Perard to patch docs
> * osstest windows-install failures
>> http://bugs.xenproject.org/xen/bug/29
>> http://www.chiark.greenend.org.uk/~xensrcts/logs/24250/
>   Anthony and/or Jan investigating
> === Big ticket items ===
> 
> * PVH dom0 (w/ Linux)
>   blocker
>   owner: mukesh@oracle, george@citrix
>   status (Linux): Acked, waiting for ABI to be nailed down
>   status (Xen): v6 posted; no longer considered a blocker
> * libvirt/libxl integration (external)
>  - owner: jfehlig@suse, dario@citrix
>  - patches posted (should be released before 4.4)
>   - migration - PCI pass-through - In progress - integration w/
>   libvirt's lock manager - improved concurrency
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


Best regards,
Yang

Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Ian Campbell]

On Thu, 2014-01-16 at 06:54 +0000, Zhang, Yang Z wrote:
> Can I say nested virtualization also is good supported in Xen 4.4? 

Can you enumerate the scenarios which have been tested and which you
consider are "supported"?

What do the hypervisor side maintainers think?

ISTR that not so long ago there were some quirks wrt not exposing the
feature to guests and crashing if they used it, and another one a while
back relating to a guest being able to enable nested virt on itself
regardless of the host administrator's settings. I suppose they are both
fixed but I wonder if the "command and control" side of nested virt has
had the same level of consideration and testing as the actual
functionality.

Ian.

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Andrew Cooper]

On 16/01/14 09:45, Ian Campbell wrote:
> On Thu, 2014-01-16 at 06:54 +0000, Zhang, Yang Z wrote:
>> Can I say nested virtualization also is good supported in Xen 4.4? 
> Can you enumerate the scenarios which have been tested and which you
> consider are "supported"?
>
> What do the hypervisor side maintainers think?

Absolutely still experimental.

>
> ISTR that not so long ago there were some quirks wrt not exposing the
> feature to guests and crashing if they used it, and another one a while
> back relating to a guest being able to enable nested virt on itself
> regardless of the host administrator's settings. I suppose they are both
> fixed but I wonder if the "command and control" side of nested virt has
> had the same level of consideration and testing as the actual
> functionality.
>
> Ian.

To the best of my knowledge, there are still deadlocks cases in the mm
code for interesting combinations of L1 and L2 memory styles such as
PoD/Paging.

See some of the concernes in the thread including Message-ID:
4A59A236-A272-471D-A061-A960E0CEFAAD@gridcentric.ca

~Andrew

Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Jan Beulich]

>>> On 16.01.14 at 10:45, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2014-01-16 at 06:54 +0000, Zhang, Yang Z wrote:
>> Can I say nested virtualization also is good supported in Xen 4.4? 
> 
> Can you enumerate the scenarios which have been tested and which you
> consider are "supported"?
> 
> What do the hypervisor side maintainers think?

Indeed I'm not sure we're there yet, not the least considering
the recent discussion between SVM and VMX folks about how
to deal with a certain problem, where a regression on the SVM
side was expected if the code would have got committed as is.

But in the end I think the VMX and SVM maintainers should
have the final say on nVMX and nSVM support state.

Jan

> ISTR that not so long ago there were some quirks wrt not exposing the
> feature to guests and crashing if they used it, and another one a while
> back relating to a guest being able to enable nested virt on itself
> regardless of the host administrator's settings. I suppose they are both
> fixed but I wonder if the "command and control" side of nested virt has
> had the same level of consideration and testing as the actual
> functionality.
> 
> Ian.

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Zhang, Yang Z]

Jan Beulich wrote on 2014-01-16:
>>>> On 16.01.14 at 10:45, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> On Thu, 2014-01-16 at 06:54 +0000, Zhang, Yang Z wrote:
>>> Can I say nested virtualization also is good supported in Xen 4.4?
>> 
>> Can you enumerate the scenarios which have been tested and which you
>> consider are "supported"?

As you know, it hard to cover all nested testings. So currently, will only cover the following L1 hypervisors: Xen, KVM, VMware workstation, VMware ESX, Win7 xp Mode, Hyper-v and L2 guest are: xp_x86, xp_x64, win7_x86, win7_x64, win8_x86, win8_x64, rhel6_x86, rhel6_x64.
For L1 Xen, KVM, VMware workstation, VMware ESX, Win7 xp Mode, I can boot up all L2 guests.
And for Hyperv, there are some known issues. But I can boot all L2 guests with my workaround patch.
Here are two topic that are discussing in mailing list which will block hyper-v
http://www.gossamer-threads.com/lists/xen/devel/310569?do=post_view_threaded
http://www.gossamer-threads.com/lists/xen/devel/299986
Also there are two another issues which I am working on cooking patches.

BTW, all my testing is based on L2 EPT. It seems there are some issues with L2 shadow page table that need deep investigatation.

As Andrew said, nested still in experimental stage, because there are still lots of scenarios I am not covered in my testing. So it may not accurate to say it is good supported. But I hope people know that the nested is ready to use now. And encourage them to try it and report bug to us to push nested move forward.

>> 
>> What do the hypervisor side maintainers think?
> 
> Indeed I'm not sure we're there yet, not the least considering the
> recent discussion between SVM and VMX folks about how to deal with a
> certain problem, where a regression on the SVM side was expected if
> the code would have got committed as is.
> 
> But in the end I think the VMX and SVM maintainers should have the
> final say on nVMX and nSVM support state.
> 
> Jan
> 
>> ISTR that not so long ago there were some quirks wrt not exposing
>> the feature to guests and crashing if they used it, and another one
>> a while back relating to a guest being able to enable nested virt on
>> itself regardless of the host administrator's settings. I suppose
>> they are both fixed but I wonder if the "command and control" side
>> of nested virt has had the same level of consideration and testing
>> as the actual functionality.
>> 
>> Ian.
> 
>


Best regards,
Yang

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Ian Campbell]

On Fri, 2014-01-17 at 02:16 +0000, Zhang, Yang Z wrote:
> As Andrew said, nested still in experimental stage, because there are
> still lots of scenarios I am not covered in my testing. So it may not
> accurate to say it is good supported. But I hope people know that the
> nested is ready to use now. And encourage them to try it and report
> bug to us to push nested move forward. 

Perhaps we could say it is "tech preview" rather than "experimental"?

What would really help is some clear guidance (i.e. docs, wiki page, a
blog post or more than one of these) on:
      * what configurations (L1 hyp/L2 guest) are supposed to work, are
        tested and are considered Supported by you guys.
      * what scenarios are expected to work, but have not been tested
        (bug/success reports welcome etc)
      * what scenarios are expected not to work but which we would like
        to support at some point or are actively working on adding
        support for
      * what scenarios are not expected to work and no one is working on

I think documenting the things toward the top end of that list would be
most valuable, the last one could be quite a long list ("everything
else") and is maybe a bit silly to try and enumerate.

Some docs and a blog post to raise awareness of the feature would likely
help push things forwards, do you think you could put something
together? For the docs either send a docs patch or write something on
the wiki (email me your wiki userid if you need to be granted write
access), for a blog post contact the publicity@ list.

You could also consider writing some instructions for test days so that
the community can do some testing, it also a good way to gain visibility
for a feature. It's probably too late to get that done for the one on
Monday[0] but you should coordinate with Russ & Dario about a future one
(a test day dedicated to nested virt is even a possibility)

Ian.

[0] http://wiki.xen.org/wiki/Xen_Test_Days  
        ttp://www.xenproject.org/about/events/viewevent/90-xen-test-day-for-4-4-rc2.html
    http://wiki.xen.org/wiki/Xen_4.4_RC2_test_instructions

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Zhang, Yang Z]

Ian Campbell wrote on 2014-01-17:
> On Fri, 2014-01-17 at 02:16 +0000, Zhang, Yang Z wrote:
>> As Andrew said, nested still in experimental stage, because there
>> are still lots of scenarios I am not covered in my testing. So it
>> may not accurate to say it is good supported. But I hope people know
>> that the nested is ready to use now. And encourage them to try it
>> and report bug to us to push nested move forward.
> 
> Perhaps we could say it is "tech preview" rather than "experimental"?
> 
> What would really help is some clear guidance (i.e. docs, wiki page, a
> blog post or more than one of these) on:
>       * what configurations (L1 hyp/L2 guest) are supposed to work, are
>         tested and are considered Supported by you guys. * what
>         scenarios are expected to work, but have not been tested
>         (bug/success reports welcome etc) * what scenarios are expected
>         not to work but which we would like to support at some point or
>         are actively working on adding support for
>       * what scenarios are not expected to work and no one is working
> on
> 
> I think documenting the things toward the top end of that list would
> be most valuable, the last one could be quite a long list ("everything
> else") and is maybe a bit silly to try and enumerate.
> 
> Some docs and a blog post to raise awareness of the feature would
> likely help push things forwards, do you think you could put something
> together? For the docs either send a docs patch or write something on
> the wiki (email me your wiki userid if you need to be granted write
> access), for a blog post contact the publicity@ list.
> 
> You could also consider writing some instructions for test days so that
> the community can do some testing, it also a good way to gain visibility
> for a feature. It's probably too late to get that done for the one on
> Monday[0] but you should coordinate with Russ & Dario about a future one
> (a test day dedicated to nested virt is even a possibility)
> 

I have a draft document, but it seems I don't have the access to add wiki page. Do you know how to get it?

> Ian.
> 
> [0] http://wiki.xen.org/wiki/Xen_Test_Days
> 
> ttp://www.xenproject.org/about/events/viewevent/90-xen-test-day-for-4-
> 4-rc2 .html
>     http://wiki.xen.org/wiki/Xen_4.4_RC2_test_instructions


Best regards,
Yang

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Ian Campbell]

On Fri, 2014-01-24 at 07:59 +0000, Zhang, Yang Z wrote:
> I have a draft document, but it seems I don't have the access to add
> wiki page. Do you know how to get it?

Either fill out the form linked from the front page of the wiki, or just
send me your wiki login name privately.

Ian.

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[George Dunlap]

On 01/17/2014 09:40 AM, Ian Campbell wrote:
> On Fri, 2014-01-17 at 02:16 +0000, Zhang, Yang Z wrote:
>> As Andrew said, nested still in experimental stage, because there are
>> still lots of scenarios I am not covered in my testing. So it may not
>> accurate to say it is good supported. But I hope people know that the
>> nested is ready to use now. And encourage them to try it and report
>> bug to us to push nested move forward.
> Perhaps we could say it is "tech preview" rather than "experimental"?

If { {xp,win7,win8,rhel6}x{x86,x64} } x { Xen, KVM, VMWare } and Win7 XP 
compatibility mode are tested regularly, and only HyperV, L2 shadow, and 
paging / PoD don't work, I think we should be able to call this a "1.0" 
release for nested virt.  Then we can add in "now works with HyperV", 
"Now works with shadow", "Now works with paging" as those become mature.

I think if we have a wiki page describing what is expected to work, and 
some of the key things that don't work, then we should be able to add 
"Basic nested virt support for VMX no longer experimental (see the the 
wiki for details)" to the "4.4 feature" list.

  -George

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Tim Deegan]

B11;rgb:0000/0000/0000At 14:47 +0000 on 24 Jan (1390571231), George Dunlap wrote:
> On 01/17/2014 09:40 AM, Ian Campbell wrote:
> > On Fri, 2014-01-17 at 02:16 +0000, Zhang, Yang Z wrote:
> >> As Andrew said, nested still in experimental stage, because there are
> >> still lots of scenarios I am not covered in my testing. So it may not
> >> accurate to say it is good supported. But I hope people know that the
> >> nested is ready to use now. And encourage them to try it and report
> >> bug to us to push nested move forward.
> > Perhaps we could say it is "tech preview" rather than "experimental"?
> 
> If { {xp,win7,win8,rhel6}x{x86,x64} } x { Xen, KVM, VMWare } and Win7 XP 
> compatibility mode are tested regularly, and only HyperV, L2 shadow, and 
> paging / PoD don't work, I think we should be able to call this a "1.0" 
> release for nested virt.  Then we can add in "now works with HyperV", 
> "Now works with shadow", "Now works with paging" as those become mature.

That depends on what the failure modes are for the other cases --
esp. given that the L1 guest's choice of hypervisor, shadow vs HAP &c,
are not under the control of the L0 admin.  I thikn that has to be
clearly understood before we encourage people to turn this on.

Tim.

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Ian Campbell]

On Fri, 2014-01-24 at 15:56 +0100, Tim Deegan wrote:
> B11;rgb:0000/0000/0000At 14:47 +0000 on 24 Jan (1390571231), George Dunlap wrote:
> > On 01/17/2014 09:40 AM, Ian Campbell wrote:
> > > On Fri, 2014-01-17 at 02:16 +0000, Zhang, Yang Z wrote:
> > >> As Andrew said, nested still in experimental stage, because there are
> > >> still lots of scenarios I am not covered in my testing. So it may not
> > >> accurate to say it is good supported. But I hope people know that the
> > >> nested is ready to use now. And encourage them to try it and report
> > >> bug to us to push nested move forward.
> > > Perhaps we could say it is "tech preview" rather than "experimental"?
> > 
> > If { {xp,win7,win8,rhel6}x{x86,x64} } x { Xen, KVM, VMWare } and Win7 XP 
> > compatibility mode are tested regularly, and only HyperV, L2 shadow, and 
> > paging / PoD don't work, I think we should be able to call this a "1.0" 
> > release for nested virt.  Then we can add in "now works with HyperV", 
> > "Now works with shadow", "Now works with paging" as those become mature.
> 
> That depends on what the failure modes are for the other cases --
> esp. given that the L1 guest's choice of hypervisor, shadow vs HAP &c,
> are not under the control of the L0 admin.  I thikn that has to be
> clearly understood before we encourage people to turn this on.

Especially in the light of the previous two bugs here which let the
guest admin crash the host, in at least one of the two cases even if the
host admin had disabled nested virt for that guest (and I think it was
actually in both cases...)

Ian.

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[George Dunlap]

On 01/24/2014 03:02 PM, Ian Campbell wrote:
> On Fri, 2014-01-24 at 15:56 +0100, Tim Deegan wrote:
>> B11;rgb:0000/0000/0000At 14:47 +0000 on 24 Jan (1390571231), George Dunlap wrote:
>>> On 01/17/2014 09:40 AM, Ian Campbell wrote:
>>>> On Fri, 2014-01-17 at 02:16 +0000, Zhang, Yang Z wrote:
>>>>> As Andrew said, nested still in experimental stage, because there are
>>>>> still lots of scenarios I am not covered in my testing. So it may not
>>>>> accurate to say it is good supported. But I hope people know that the
>>>>> nested is ready to use now. And encourage them to try it and report
>>>>> bug to us to push nested move forward.
>>>> Perhaps we could say it is "tech preview" rather than "experimental"?
>>> If { {xp,win7,win8,rhel6}x{x86,x64} } x { Xen, KVM, VMWare } and Win7 XP
>>> compatibility mode are tested regularly, and only HyperV, L2 shadow, and
>>> paging / PoD don't work, I think we should be able to call this a "1.0"
>>> release for nested virt.  Then we can add in "now works with HyperV",
>>> "Now works with shadow", "Now works with paging" as those become mature.
>> That depends on what the failure modes are for the other cases --
>> esp. given that the L1 guest's choice of hypervisor, shadow vs HAP &c,
>> are not under the control of the L0 admin.  I thikn that has to be
>> clearly understood before we encourage people to turn this on.
> Especially in the light of the previous two bugs here which let the
> guest admin crash the host, in at least one of the two cases even if the
> host admin had disabled nested virt for that guest (and I think it was
> actually in both cases...)

Right -- well I think then we need to help try to define some criteria 
that VMX nested virt would need to meet for portions of it to stop being 
considered "experimental" or "tech preview".  Just a couple of angles:

* L1 / L2 guests tested.  What do people think of the mix of L1 / L2 
guests there?  They look like a pretty good combination to me.

* L2 workloads tested

Other than booting, what kinds of workloads are run in the L2 guests?  
Do the L2 guests ever get into heavy swapping scenarios, for instance?

* Minimum subset of functionality

I think it makes sense to explicitly say that we support only certain 
hypervisors, and to not support some advanced features in L2 guests.  
Saying only L1 HAP L2 HAP is reasonable, I think.  No HyperV, no L2 
shadow, no PoD are reasonable restrictions; it should be fine for us to 
say that the L1 admin enables that, and badness ensues, he has only 
himself to blame.

* Security

That said, I think we must assume that some of our users will have L0 
admin != L1 admin.  This means that L1 admin must not be able to do 
anything to crash L0.  In the PoD case above, for example, if L1 enables 
PoD or paging, it might cause locking issue in L0; that's not acceptable.

Anything else?

  -George

Re: Status of Nested Virt in 4.4 (Was: Re: Xen 4.4 development update)

[Zhang, Yang Z]

George Dunlap wrote on 2014-01-25:
> On 01/24/2014 03:02 PM, Ian Campbell wrote:
>> On Fri, 2014-01-24 at 15:56 +0100, Tim Deegan wrote:
>>> B11;rgb:0000/0000/0000At 14:47 +0000 on 24 Jan (1390571231), George
> Dunlap wrote:
>>>> On 01/17/2014 09:40 AM, Ian Campbell wrote:
>>>>> On Fri, 2014-01-17 at 02:16 +0000, Zhang, Yang Z wrote:
>>>>>> As Andrew said, nested still in experimental stage, because there are
>>>>>> still lots of scenarios I am not covered in my testing. So it may not
>>>>>> accurate to say it is good supported. But I hope people know that the
>>>>>> nested is ready to use now. And encourage them to try it and report
>>>>>> bug to us to push nested move forward.
>>>>> Perhaps we could say it is "tech preview" rather than "experimental"?
>>>> If { {xp,win7,win8,rhel6}x{x86,x64} } x { Xen, KVM, VMWare } and Win7 XP
>>>> compatibility mode are tested regularly, and only HyperV, L2 shadow, and
>>>> paging / PoD don't work, I think we should be able to call this a "1.0"
>>>> release for nested virt.  Then we can add in "now works with HyperV",
>>>> "Now works with shadow", "Now works with paging" as those become
> mature.
>>> That depends on what the failure modes are for the other cases --
>>> esp. given that the L1 guest's choice of hypervisor, shadow vs HAP &c,
>>> are not under the control of the L0 admin.  I thikn that has to be
>>> clearly understood before we encourage people to turn this on.
>> Especially in the light of the previous two bugs here which let the
>> guest admin crash the host, in at least one of the two cases even if the
>> host admin had disabled nested virt for that guest (and I think it was
>> actually in both cases...)
> 
> Right -- well I think then we need to help try to define some criteria
> that VMX nested virt would need to meet for portions of it to stop being
> considered "experimental" or "tech preview".  Just a couple of angles:
> 
> * L1 / L2 guests tested.  What do people think of the mix of L1 / L2
> guests there?  They look like a pretty good combination to me.
> 
> * L2 workloads tested
> 
> Other than booting, what kinds of workloads are run in the L2 guests?
> Do the L2 guests ever get into heavy swapping scenarios, for instance?

Currently, we didn't start the workload testing. I guess there will be more bugs coming if running workload inside guest. :)

> 
> * Minimum subset of functionality
> 
> I think it makes sense to explicitly say that we support only certain
> hypervisors, and to not support some advanced features in L2 guests.
>
> Saying only L1 HAP L2 HAP is reasonable, I think.  No HyperV, no L2
> shadow, no PoD are reasonable restrictions; it should be fine for us to
> say that the L1 admin enables that, and badness ensues, he has only
> himself to blame.

I think Hyper-V should be acceptable. 

>
> 
> * Security
> 
> That said, I think we must assume that some of our users will have L0
> admin != L1 admin.  This means that L1 admin must not be able to do
> anything to crash L0.  In the PoD case above, for example, if L1 enables
> PoD or paging, it might cause locking issue in L0; that's not acceptable.
> 
> Anything else?
> 

Should we consider the save/restore and migration for L1? I believe it also doesn't working currently.

>   -George


Best regards,
Yang