Xen 4.3 development update CODE FREEZE

Xen 4.3 development update CODE FREEZE

[George Dunlap]

This information will be mirrored on the Xen 4.3 Roadmap wiki page:
 http://wiki.xen.org/wiki/Xen_Roadmap/4.3

Now that we've begun freezing, and gotten a number of the major features
in, I think we need to lock down on the new features.

The key goals we're focusing on now, in order, are as follows:
 1. Have a bug-free 4.3 release
 2. Have an awesome 4.3 release
 3. Have a 4.3 release that happens on schedule (ready by June 15th)

At this point I think a feature is going to have to be pretty cool to
merit an exception under #2.

The most important thing in making a case is to answer the question,
"If there are bugs in this patch, will they be discovered before the
June 15h release?"  The second most important thing is to consider the
cost/benefit analysis of bugs that are found: what is the risk of
introducing a bug which will delay the release, vs the benefit it will
have in making the release better?

Please begin to test and report bugs against the release.

= Timeline =

We are planning on a 9-month release cycle.  Based on that, below are
our estimated dates:
* Feature freeze: 25 March 2013
* Code freezing point: 15 April 2013 <== We are here
* First RC: 6 May 2013
* Release: 17 June 2013

The RCs and release will of course depend on stability and bugs, and
will therefore be fairly unpredictable.  Each new feature will be
considered on a case-by-case basis.

Last updated: 22 April 2013

== Completed ==

* Serial console improvements
  -EHCI debug port

* Default to QEMU upstream (partial)
 - pci pass-thru (external)
 - enable dirtybit tracking during migration (external)
 - xl cd-{insert,eject} (external)

* CPUID-based idle (don't rely on ACPI info f/ dom0)

* Persistent grants for blk (external)
 - Linux
 - qemu

* Allow XSM to override IS_PRIV checks in the hypervisor

* Scalability: 16TiB of RAM

* xl QXL Spice support

* Install into /usr/local by default
  owner: Ian Campbell

* openvswitch toostack integration
  To label "tech-preview" unless we get good testing (>10 individuals)

* NUMA scheduler affinity

* ARM v7 server port (basic)
* ARM v8 server port (basic)

== Bugs ==

* xl, compat mode, and older kernels
  owner: Wei Liu
  Many older 32-bit PV kernels that can run on a 64-bit hypervisor with
  xend do not work when started with xl.  The following work-around seems to
  work:
    xl create -p lightning.cfg
    xenstore-write /local/domain/$(xl domid
lightning)/device/vbd/51713/protocol x86_32-abi
    xl unpause lightning
  This node is normally written by the guest kernel, but for older kernels
  seems not to be.  xend must have a work-around; port this work-around to xl.

* AMD NPT performance regression after c/s 24770:7f79475d3de7
  owner: ?
  Reference: http://marc.info/?l=xen-devel&m=135075376805215

* qemu-upstream: cd-insert and cd-eject not working
  http://marc.info/?l=xen-devel&m=135850249808960

* Revert Jan's debugging patch (commit bd9be94)
  owner: Jan Beulich

* Race condition in claim hypercall
  owner: Ian Jackson, Konrad Wilk

* Remove hardcoded mobprobe's in xencommons
  owner: Wei Liu
  status: ?

* Windows 2003 fails to install in Xen-unstable tip
  > Narrowed down to c/s 2fe82ac11fd078485388fe7c5e8bc3b6ac9185b0
  owner: Jan Beulich

== Not yet complete ==

* Multi-vector PCI MSI (support at least for Dom0)
  owner: jan@suse
  status: ?

* vTPM updates
  owner: Daniel DeGraaf @ NSA

* xl USB pass-through for HVM guests using Qemu USB emulation
  owner: George
  status: v6 patch series posted

* Rationalized backend scripts
  owner: roger@citrix

* Scripts for driver domains (depends on backend scripts)
  owner: roger@citrix

Re: Xen 4.3 development update CODE FREEZE

[George Dunlap]

On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> * Multi-vector PCI MSI (support at least for Dom0)
>   owner: jan@suse
>   status: ?

Jan, what's the status of this one?  Are you still pushing for this to
make it into 4.3, or will you wait until 4.4?

 -George

Re: Xen 4.3 development update CODE FREEZE

[George Dunlap]

On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> * vTPM updates
>   owner: Daniel DeGraaf @ NSA

Daniel, I saw that your patchs posted back on March 22 were accepted.
Are there more patches that need to be accepted for 4.3, or can I move
this to the "Completed" list?

 -George

Re: Xen 4.3 development update CODE FREEZE

[Daniel De Graaf]

On 04/22/2013 11:02 AM, George Dunlap wrote:
> On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
> <George.Dunlap@eu.citrix.com> wrote:
>> * vTPM updates
>>    owner: Daniel DeGraaf @ NSA
>
> Daniel, I saw that your patchs posted back on March 22 were accepted.
> Are there more patches that need to be accepted for 4.3, or can I move
> this to the "Completed" list?
>
>   -George
>

It's done except for the Linux driver being accepted - which I believe is
outside the scope of your list, so yes, mark it as completed.

-- 
Daniel De Graaf
National Security Agency

Re: Xen 4.3 development update CODE FREEZE

[George Dunlap]

On 22/04/13 16:04, Daniel De Graaf wrote:
> On 04/22/2013 11:02 AM, George Dunlap wrote:
>> On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
>> <George.Dunlap@eu.citrix.com> wrote:
>>> * vTPM updates
>>>     owner: Daniel DeGraaf @ NSA
>> Daniel, I saw that your patchs posted back on March 22 were accepted.
>> Are there more patches that need to be accepted for 4.3, or can I move
>> this to the "Completed" list?
>>
>>    -George
>>
> It's done except for the Linux driver being accepted - which I believe is
> outside the scope of your list, so yes, mark it as completed.

Great, thanks.

  -George

Re: Xen 4.3 development update CODE FREEZE

[Jan Beulich]

>>> On 22.04.13 at 17:01, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
> <George.Dunlap@eu.citrix.com> wrote:
>> * Multi-vector PCI MSI (support at least for Dom0)
>>   owner: jan@suse
>>   status: ?
> 
> Jan, what's the status of this one?  Are you still pushing for this to
> make it into 4.3, or will you wait until 4.4?

As also said on irc earlier this morning (I was gone already when
you asked yesterday), I'm still hoping to get this in, but it's not
really under my control - I'm depending on AMD to help out
debugging on their hardware.

Suravee actually pointed out that the problem with the earlier
posted prerequisite patches was apparently spurious, but he
found a problem with the final patch of the full series, which we'll
have to work out.

Jan

Re: Xen 4.3 development update CODE FREEZE

[George Dunlap]

On Tue, Apr 23, 2013 at 8:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 22.04.13 at 17:01, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>> On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
>> <George.Dunlap@eu.citrix.com> wrote:
>>> * Multi-vector PCI MSI (support at least for Dom0)
>>>   owner: jan@suse
>>>   status: ?
>>
>> Jan, what's the status of this one?  Are you still pushing for this to
>> make it into 4.3, or will you wait until 4.4?
>
> As also said on irc earlier this morning (I was gone already when
> you asked yesterday), I'm still hoping to get this in, but it's not
> really under my control - I'm depending on AMD to help out
> debugging on their hardware.
>
> Suravee actually pointed out that the problem with the earlier
> posted prerequisite patches was apparently spurious, but he
> found a problem with the final patch of the full series, which we'll
> have to work out.

Right -- then from a release perspective, should we take it off the table?

Remember the criteria:
1. Will we find all the bugs before the release?
2. What is the risk of finding bugs that slip the release?
3. Is it valuable enough that it's worth the risk of slipping the release?

Will this be enabled on typical hardware, or will it only be present
if the user specifically does something to enable it?  If the second,
#1 is probably less important; otherwise, it seems pretty risky.

Regarding 2 and 3, I admit I don't have a good handle on how valuable
the feature is, but absent a pretty high value, it seems like it might
be better to wait.

Thoughts?

 -George

Re: Xen 4.3 development update CODE FREEZE

[Jan Beulich]

>>> On 23.04.13 at 11:09, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
> On Tue, Apr 23, 2013 at 8:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 22.04.13 at 17:01, George Dunlap <George.Dunlap@eu.citrix.com> wrote:
>>> On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
>>> <George.Dunlap@eu.citrix.com> wrote:
>>>> * Multi-vector PCI MSI (support at least for Dom0)
>>>>   owner: jan@suse
>>>>   status: ?
>>>
>>> Jan, what's the status of this one?  Are you still pushing for this to
>>> make it into 4.3, or will you wait until 4.4?
>>
>> As also said on irc earlier this morning (I was gone already when
>> you asked yesterday), I'm still hoping to get this in, but it's not
>> really under my control - I'm depending on AMD to help out
>> debugging on their hardware.
>>
>> Suravee actually pointed out that the problem with the earlier
>> posted prerequisite patches was apparently spurious, but he
>> found a problem with the final patch of the full series, which we'll
>> have to work out.
> 
> Right -- then from a release perspective, should we take it off the table?
> 
> Remember the criteria:
> 1. Will we find all the bugs before the release?
> 2. What is the risk of finding bugs that slip the release?
> 3. Is it valuable enough that it's worth the risk of slipping the release?
> 
> Will this be enabled on typical hardware, or will it only be present
> if the user specifically does something to enable it?  If the second,
> #1 is probably less important; otherwise, it seems pretty risky.

On suitable hardware it would get used by default, provided the
kernel also has been enabled accordingly.

> Regarding 2 and 3, I admit I don't have a good handle on how valuable
> the feature is, but absent a pretty high value, it seems like it might
> be better to wait.

It's primarily a load spreading feature, allowing to distribute
interrupts more evenly across (v)CPUs.

And yes, since we're making so slow progress with this, I'm
inclined to agree, however much I regret that.

Jan

Re: Xen 4.3 development update CODE FREEZE

[George Dunlap]

On Mon, Apr 22, 2013 at 4:00 PM, George Dunlap
<George.Dunlap@eu.citrix.com> wrote:
> == Bugs ==

> * qemu-upstream: cd-insert and cd-eject not working
>   http://marc.info/?l=xen-devel&m=135850249808960

Where are we with the problems reported in this thread?

"xl cd-eject" works fine for me with qemu-upstream, as does network
after suspend/resume.  I also don't have any extraneous cdrom or
floppy devices (AFAICT at a quick smoke-test).

 -George