Xen 4.4 development update: Code freezing point reached

Xen 4.4 development update: Code freezing point reached

[George Dunlap]

This information will be mirrored on the Xen 4.4 Roadmap wiki page:
 http://wiki.xen.org/wiki/Xen_Roadmap/4.4

(And I actually updated the wiki this time.)

The code "freezing point" is today; which means that starting today
non-bug fixes need a freeze exception to be included.

Remember our goal for the release:
 1. A bug-free release
 2. An awesome release
 3. An on-time release

Accepting a new feature may make Xen more awesome; but it also
introduces a risk that it will introduce more bugs.  That bug may be
found before the release (threatening #3), or it may not be found
until after the release (threatening #1).  Each freeze exception
request will attempt to balance the benefits (how awesome the
exception is) vs the risks (will it cause the release to slip, or
worse, cause a bug which goes un-noticed into the final release).

The idea is that today we will be pretty permissive, but that we will
become progressively more conservative until the first RC, which is
scheduled for 3 weeks' time (6 December).  After that, we will only
accept bug fixes.

Bug fixes can be checked in without a freeze exception throughout the
code freeze, unless the maintianer thinks they are particularly high
risk.  In later RC's, we may even begin rejecting bug fixes if the
broken functionality is small and the risk to other functionality is
high.

Features which are currently marked "experimental" or do not at the
moment work at all cannot be broken really; so changes to code only
used by those features should be able to get a freeze exception
easily.  (Tianocore is something which would probably fall under
this.)

Features which change or add new interfaces which will need to be
supported in a backwards-compatible way (for instance, vNUMA) will
need freeze exceptions to make sure that the interface itself has
enough time to be considered stable.

These are guidelines and principles to give you an idea where we're
coming from; if you think there's a good reason why making an
exception for you will help us achieve goals 1-3 above better than not
doing so, feel free to make your case.

= Timeline =

Here is our current timeline based on a 6-month release:

* Feature freeze: 18 October 2013
* Code freezing point: 18 November 2013 <== WE ARE HERE
* First RC: 6 December 2013
* Release: 21 January 2014

Last updated: 18 November 2016

== Completed  ==

* Event channel scalability (FIFO event channels)

* Non-udev scripts for driver domains (non-Linux driver domains)

* Multi-vector PCI MSI (Hypervisor side)

* Improved Spice support on libxl
 - Added Spice vdagent support
 - Added Spice clipboard sharing support

* PHV domU (experimental only)

* Guest EFI booting (tianocore)

* kexec

* Testing: Xen on ARM

* Update to SeaBIOS 1.7.3.1

* pvgrub2 checked into grub upstream

== Resolved since last update ==

* credit scheduler doesn't update other fields when tslice updated from sysctl

== Open ==

* qemu-upstream not freeing pirq
 > http://www.gossamer-threads.com/lists/xen/devel/281498
 status: patches posted; latest patches need testing

* Race in PV shutdown between tool detection and shutdown watch
 > http://www.gossamer-threads.com/lists/xen/devel/282467
 > Nothing to do with ACPI
 status: Patches posted

* Supposed regression from a3513737 ("x86: allow guest to set/clear
 > MSI-X mask bit (try 2)"), as per
 > http://lists.xenproject.org/archives/html/xen-devel/2013-09/msg01589.html.

* qemu-traditional mis-parses host bus 8 as 0
 > http://bugs.xenproject.org/xen/bug/15

* xen_platform_pci=0 doesn't work with qemu-xen
 > http://bugs.xenproject.org/xen/bug/20

* xl does not support specifying virtual function for passthrough device
 > http://bugs.xenproject.org/xen/bug/22

* xl does not handle migrate interruption gracefully
  > If you start a localhost migrate, and press "Ctrl-C" in the middle,
  > you get two hung domains

* libxl / xl does not handle failure of remote qemu gracefully
  > Easiest way to reproduce:
  >  - set "vncunused=0" and do a local migrate
  >  - The "remote" qemu will fail because the vnc port is in use
  > The failure isn't the problem, but everything being stuck afterwards is

* HPET interrupt stack overflow (when using hpet_broadcast mode and MSI
capable HPETs)
  owner: andyh@citrix
  status: patches posted, undergoing review iteration.

== Backlog ==

=== Testing coverage ===

* new libxl w/ previous versions of xl
 @IanJ

* Host S3 suspend
 @bguthro, @dariof

* Default [example] XSM policy
 @Stefano to ask Daniel D

* Storage driver domains
 @roger

* HVM pci passthrough
 @anthony

* PV pci passthrough
 @konrad (or @george if he gets to it first)

* Network driver domains
 @George

* Nested virt?
 @intel (chased by George)

* Fix SRIOV test (chase intel)
 @ianj

* Fix bisector to e-mail blame-worthy parties
 @ianj

* Fix xl shutdown
  @ianj

* stub domains
  @athony

* performance benchmarks
  @dario

=== Meta-items (composed of other items) ===

* Meta: PVIO NUMA improvements
 - NUMA affinity for vcpus (4.4 possible)
 - PV guest NUMA interface (4.4 possible)
 - Sensible dom0 NUMA layout
 - Toolstack pinning backend thread / virq to appropraite d0 vcpu
 - NUMA-aware ballooning

* xend still in tree (x)
 - xl list -l on a dom0-only system
 - xl list -l doesn't contain tty console port
 - xl Alternate transport support for migration*
 - xl PVSCSI support
 - xl PVUSB support

=== Big ticket items ===

* PVH mode (w/ Linux)
  owner: mukesh@oracle, george@citrix
  status (Linux): Acked, waiting for ABI to be nailed down
  status (Xen): Initial version checked in, still nailing down interface

* Update to qemu 1.6
  owner: Anthonyper
  status: In staging, still working out a bug in the VMX code

* Live Migration Support
  owner: Jaeyong Yoo <jaeyong.yoo@samsung.com>
  status: v5 posted, looking good for code freeze

* ARM64 guest
  owner: IanC
  status: v3 posted, v4 in progress. looking good.

* SWIOTLB (kernel side thing)
  owner: Stefano
  status: Pull request sent.

* soft affinity for vcpus (was NUMA affinity for vcpus)
    owner: Dario
    status: v2 posted

* PV guest NUMA interface
    owner: Elena
    status: v3 posted

* libvirt/libxl integration (external)
 - owner: jfehlig@suse, dario@citrix
 - patches posted (should be released before 4.4)
  - migration
  - PCI pass-through
 - In progress
  - integration w/ libvirt's lock manager
  - improved concurrency

* xl USB pass-through for HVM guests using Qemu USB emulation
  prognosis: Good if extended
  owner: George
  status: v6 patch series posted

* libxl: Spice usbredirection support for upstream qemu
 owner: fabio@M2R
 status: I'll post new patch version shortly

* libxl: usb2 and usb3 controller support for upstream qemu
 owner: fabio@M2R
 status: patch v5 posted, tested and working, awaiting reviews

* libxl network buffering support for Remus
   @shriram
   status: patches posted
   prognosis: fair

* Disk: indirect descriptors
   owner: roger@citrix
   status: Linux side in 3.11, Xen-side patch posted

Re: Xen 4.4 development update: Code freezing point reached

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1068 bytes --]

On lun, 2013-11-18 at 18:19 +0000, George Dunlap wrote:
> [..]
> = Timeline =
> 
> Here is our current timeline based on a 6-month release:
> 
> * Feature freeze: 18 October 2013
> * Code freezing point: 18 November 2013 <== WE ARE HERE
> * First RC: 6 December 2013
> * Release: 21 January 2014
> 
> Last updated: 18 November 2016
> [..]
> === Meta-items (composed of other items) ===
> 
> * Meta: PVIO NUMA improvements
>  - NUMA affinity for vcpus (4.4 possible)
>
v3 of the series posted right now. I don' think is that intrusive and
likely to cause bugs _but_ it introduces new a couple of libxl API
calls. I therefore think that the decision of whether or not to make
this a 4.4 thing would mostly be whether or not we think such interface
is good enough.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.4 development update: Code freezing point reached

[Pasi Kärkkäinen]

On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
> 
> = Timeline =
> 
> Here is our current timeline based on a 6-month release:
> 
> * Feature freeze: 18 October 2013
> * Code freezing point: 18 November 2013 <== WE ARE HERE
> * First RC: 6 December 2013
> * Release: 21 January 2014
> 
> Last updated: 18 November 2016
>

2016 :) 


> 
> == Backlog ==
> 
> 
> === Big ticket items ===
> 

 * PCI hole resize support hvmloader/qemu-traditional/qemu-upstream with PCI/GPU passthrough
   http://lists.xen.org/archives/html/xen-devel/2013-05/msg02813.html
   Where Stefano writes:

   1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
   qemu-xen-unstable in terms of configuration and not to introduce any
   regressions. Do this for the Xen 4.3 release.

   2) for Xen 4.4 rework the two patches above and improve
   i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
   enough, it also needs to be able to resize the system memory region
   (xen.ram) to make room for the bigger pci_hole



-- Pasi

Re: Xen 4.4 development update: Code freezing point reached

[Don Slutz]

[-- Attachment #1.1: Type: text/plain, Size: 862 bytes --]

On 11/18/13 13:19, George Dunlap wrote:
> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>   http://wiki.xen.org/wiki/Xen_Roadmap/4.4
[snip]

I realize that I did not send anything to you before this.  However:

http://lists.xen.org/archives/html/xen-devel/2013-10/msg01117.html
http://lists.xen.org/archives/html/xen-devel/2013-11/msg02352.html
http://lists.xen.org/archives/html/xen-devel/2013-11/msg02569.html

Is about a new minor feature.  Clearly I would like it in 4.4.  This is 
close to:

    xl dump-core <Domain> <filename>
    crash <filename>

But does not require the disk space, nor the delay while writing the 
file.  For me it took ~2 minutes to write out a 3G core file.
     -Don Slutz

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


[-- Attachment #1.2: Type: text/html, Size: 2151 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.4 development update: Code freezing point reached

[Roger Pau Monné]

On 18/11/13 19:19, George Dunlap wrote:
> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>  http://wiki.xen.org/wiki/Xen_Roadmap/4.4
> 
> (And I actually updated the wiki this time.)
> 
> The code "freezing point" is today; which means that starting today
> non-bug fixes need a freeze exception to be included.
> 
> Remember our goal for the release:
>  1. A bug-free release
>  2. An awesome release
>  3. An on-time release
> 
> Accepting a new feature may make Xen more awesome; but it also
> introduces a risk that it will introduce more bugs.  That bug may be
> found before the release (threatening #3), or it may not be found
> until after the release (threatening #1).  Each freeze exception
> request will attempt to balance the benefits (how awesome the
> exception is) vs the risks (will it cause the release to slip, or
> worse, cause a bug which goes un-noticed into the final release).
> 
> The idea is that today we will be pretty permissive, but that we will
> become progressively more conservative until the first RC, which is
> scheduled for 3 weeks' time (6 December).  After that, we will only
> accept bug fixes.
> 
> Bug fixes can be checked in without a freeze exception throughout the
> code freeze, unless the maintianer thinks they are particularly high
> risk.  In later RC's, we may even begin rejecting bug fixes if the
> broken functionality is small and the risk to other functionality is
> high.
> 
> Features which are currently marked "experimental" or do not at the
> moment work at all cannot be broken really; so changes to code only
> used by those features should be able to get a freeze exception
> easily.  (Tianocore is something which would probably fall under
> this.)
> 
> Features which change or add new interfaces which will need to be
> supported in a backwards-compatible way (for instance, vNUMA) will
> need freeze exceptions to make sure that the interface itself has
> enough time to be considered stable.
> 
> These are guidelines and principles to give you an idea where we're
> coming from; if you think there's a good reason why making an
> exception for you will help us achieve goals 1-3 above better than not
> doing so, feel free to make your case.
> 
> = Timeline =
> 
> Here is our current timeline based on a 6-month release:
> 
> * Feature freeze: 18 October 2013
> * Code freezing point: 18 November 2013 <== WE ARE HERE
> * First RC: 6 December 2013
> * Release: 21 January 2014
> 
> Last updated: 18 November 2016
> 
> == Completed  ==
> 
> * Event channel scalability (FIFO event channels)
> 
> * Non-udev scripts for driver domains (non-Linux driver domains)
> 
> * Multi-vector PCI MSI (Hypervisor side)
> 
> * Improved Spice support on libxl
>  - Added Spice vdagent support
>  - Added Spice clipboard sharing support
> 
> * PHV domU (experimental only)
> 
> * Guest EFI booting (tianocore)
> 
> * kexec
> 
> * Testing: Xen on ARM
> 
> * Update to SeaBIOS 1.7.3.1
> 
> * pvgrub2 checked into grub upstream

* Driver domain userspace daemon.

> 
> == Resolved since last update ==
> 
> * credit scheduler doesn't update other fields when tslice updated from sysctl
> 
> == Open ==
> 
> * qemu-upstream not freeing pirq
>  > http://www.gossamer-threads.com/lists/xen/devel/281498
>  status: patches posted; latest patches need testing
> 
> * Race in PV shutdown between tool detection and shutdown watch
>  > http://www.gossamer-threads.com/lists/xen/devel/282467
>  > Nothing to do with ACPI
>  status: Patches posted
> 
> * Supposed regression from a3513737 ("x86: allow guest to set/clear
>  > MSI-X mask bit (try 2)"), as per
>  > http://lists.xenproject.org/archives/html/xen-devel/2013-09/msg01589.html.
> 
> * qemu-traditional mis-parses host bus 8 as 0
>  > http://bugs.xenproject.org/xen/bug/15
> 
> * xen_platform_pci=0 doesn't work with qemu-xen
>  > http://bugs.xenproject.org/xen/bug/20
> 
> * xl does not support specifying virtual function for passthrough device
>  > http://bugs.xenproject.org/xen/bug/22
> 
> * xl does not handle migrate interruption gracefully
>   > If you start a localhost migrate, and press "Ctrl-C" in the middle,
>   > you get two hung domains
> 
> * libxl / xl does not handle failure of remote qemu gracefully
>   > Easiest way to reproduce:
>   >  - set "vncunused=0" and do a local migrate
>   >  - The "remote" qemu will fail because the vnc port is in use
>   > The failure isn't the problem, but everything being stuck afterwards is
> 
> * HPET interrupt stack overflow (when using hpet_broadcast mode and MSI
> capable HPETs)
>   owner: andyh@citrix
>   status: patches posted, undergoing review iteration.
> 
> == Backlog ==
> 
> === Testing coverage ===
> 
> * new libxl w/ previous versions of xl
>  @IanJ
> 
> * Host S3 suspend
>  @bguthro, @dariof
> 
> * Default [example] XSM policy
>  @Stefano to ask Daniel D
> 
> * Storage driver domains
>  @roger
> 
> * HVM pci passthrough
>  @anthony
> 
> * PV pci passthrough
>  @konrad (or @george if he gets to it first)
> 
> * Network driver domains
>  @George
> 
> * Nested virt?
>  @intel (chased by George)
> 
> * Fix SRIOV test (chase intel)
>  @ianj
> 
> * Fix bisector to e-mail blame-worthy parties
>  @ianj
> 
> * Fix xl shutdown
>   @ianj
> 
> * stub domains
>   @athony
> 
> * performance benchmarks
>   @dario
> 
> === Meta-items (composed of other items) ===
> 
> * Meta: PVIO NUMA improvements
>  - NUMA affinity for vcpus (4.4 possible)
>  - PV guest NUMA interface (4.4 possible)
>  - Sensible dom0 NUMA layout
>  - Toolstack pinning backend thread / virq to appropraite d0 vcpu
>  - NUMA-aware ballooning
> 
> * xend still in tree (x)
>  - xl list -l on a dom0-only system
>  - xl list -l doesn't contain tty console port
>  - xl Alternate transport support for migration*
>  - xl PVSCSI support
>  - xl PVUSB support
> 
> === Big ticket items ===
> 
> * PVH mode (w/ Linux)
>   owner: mukesh@oracle, george@citrix
>   status (Linux): Acked, waiting for ABI to be nailed down
>   status (Xen): Initial version checked in, still nailing down interface
> 
> * Update to qemu 1.6
>   owner: Anthonyper
>   status: In staging, still working out a bug in the VMX code
> 
> * Live Migration Support
>   owner: Jaeyong Yoo <jaeyong.yoo@samsung.com>
>   status: v5 posted, looking good for code freeze
> 
> * ARM64 guest
>   owner: IanC
>   status: v3 posted, v4 in progress. looking good.
> 
> * SWIOTLB (kernel side thing)
>   owner: Stefano
>   status: Pull request sent.
> 
> * soft affinity for vcpus (was NUMA affinity for vcpus)
>     owner: Dario
>     status: v2 posted
> 
> * PV guest NUMA interface
>     owner: Elena
>     status: v3 posted
> 
> * libvirt/libxl integration (external)
>  - owner: jfehlig@suse, dario@citrix
>  - patches posted (should be released before 4.4)
>   - migration
>   - PCI pass-through
>  - In progress
>   - integration w/ libvirt's lock manager
>   - improved concurrency
> 
> * xl USB pass-through for HVM guests using Qemu USB emulation
>   prognosis: Good if extended
>   owner: George
>   status: v6 patch series posted
> 
> * libxl: Spice usbredirection support for upstream qemu
>  owner: fabio@M2R
>  status: I'll post new patch version shortly
> 
> * libxl: usb2 and usb3 controller support for upstream qemu
>  owner: fabio@M2R
>  status: patch v5 posted, tested and working, awaiting reviews
> 
> * libxl network buffering support for Remus
>    @shriram
>    status: patches posted
>    prognosis: fair
> 
> * Disk: indirect descriptors
>    owner: roger@citrix
>    status: Linux side in 3.11, Xen-side patch posted
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: Xen 4.4 development update: Code freezing point reached

[George Dunlap]

On 11/19/2013 10:50 AM, Roger Pau Monné wrote:
>> == Completed  ==
>>
>
> * Driver domain userspace daemon.

That's what I was trying to get at with this one:

 >>
 >> * Non-udev scripts for driver domains (non-Linux driver domains)
 >>

Sorry, I put it near the top because I thought it was among one of the 
more important user-visible features. :-)

  -George

Re: Xen 4.4 development update: Code freezing point reached

[Roger Pau Monné]

On 19/11/13 12:06, George Dunlap wrote:
> On 11/19/2013 10:50 AM, Roger Pau Monné wrote:
>>> == Completed  ==
>>>
>>
>> * Driver domain userspace daemon.
> 
> That's what I was trying to get at with this one:
> 
>>>
>>> * Non-udev scripts for driver domains (non-Linux driver domains)
>>>
> 
> Sorry, I put it near the top because I thought it was among one of the
> more important user-visible features. :-)

My bad, I didn't recognize it :)

Re: Xen 4.4 development update: Code freezing point reached

[Ian Campbell]

On Mon, 2013-11-18 at 18:19 +0000, George Dunlap wrote:
> === Big ticket items ===

> * Live Migration Support
NB:^ARM

>   owner: Jaeyong Yoo <jaeyong.yoo@samsung.com>
>   status: v5 posted, looking good for code freeze

I think I was terribly confused about the release timeline when I gave
this prognosis.

Things are certainly looking good but given that we are now past feature
freeze and into code freeze I think we might be better off targeting
4.5. The patch is reasonably deeply involved with the lowlevel memory
mgmt stuff and the risk is that we regress the basic functionality.

Sorry for the earlier bum steer.

> * ARM64 guest
>   owner: IanC
>   status: v3 posted, v4 in progress. looking good.

v6 posted just now, I hope this can go in soon.

> * Disk: indirect descriptors
>    owner: roger@citrix
>    status: Linux side in 3.11, Xen-side patch posted

NB this is just an interface documentation update -- there is no
functionality on the Xen side.

Ian.

Re: Xen 4.4 development update: Code freezing point reached

[Konrad Rzeszutek Wilk]

On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>  http://wiki.xen.org/wiki/Xen_Roadmap/4.4
> 
> (And I actually updated the wiki this time.)
> 
> The code "freezing point" is today; which means that starting today
> non-bug fixes need a freeze exception to be included.
> 
> Remember our goal for the release:
>  1. A bug-free release
>  2. An awesome release
>  3. An on-time release
> 
> Accepting a new feature may make Xen more awesome; but it also
> introduces a risk that it will introduce more bugs.  That bug may be
> found before the release (threatening #3), or it may not be found
> until after the release (threatening #1).  Each freeze exception
> request will attempt to balance the benefits (how awesome the
> exception is) vs the risks (will it cause the release to slip, or
> worse, cause a bug which goes un-noticed into the final release).
> 
> The idea is that today we will be pretty permissive, but that we will
> become progressively more conservative until the first RC, which is
> scheduled for 3 weeks' time (6 December).  After that, we will only
> accept bug fixes.
> 
> Bug fixes can be checked in without a freeze exception throughout the
> code freeze, unless the maintianer thinks they are particularly high
> risk.  In later RC's, we may even begin rejecting bug fixes if the
> broken functionality is small and the risk to other functionality is
> high.
> 
> Features which are currently marked "experimental" or do not at the
> moment work at all cannot be broken really; so changes to code only
> used by those features should be able to get a freeze exception
> easily.  (Tianocore is something which would probably fall under
> this.)
> 
> Features which change or add new interfaces which will need to be
> supported in a backwards-compatible way (for instance, vNUMA) will
> need freeze exceptions to make sure that the interface itself has
> enough time to be considered stable.
> 
> These are guidelines and principles to give you an idea where we're
> coming from; if you think there's a good reason why making an
> exception for you will help us achieve goals 1-3 above better than not
> doing so, feel free to make your case.

I am wondering in which category the tmem cleanup patches fall?

They aren't bug-fixes, they could be considered a feature. They were
posted before the deadline. I posted the GIT PULL (see
http://article.gmane.org/gmane.comp.emulators.xen.devel/178043)
to one of the folks who has write access to the repository (as
documented in http://www.xenproject.org/governance.html)?


> == Open ==
> 
> * qemu-upstream not freeing pirq
>  > http://www.gossamer-threads.com/lists/xen/devel/281498
>  status: patches posted; latest patches need testing

Duan, ping?

> 
> * Race in PV shutdown between tool detection and shutdown watch
>  > http://www.gossamer-threads.com/lists/xen/devel/282467
>  > Nothing to do with ACPI
>  status: Patches posted

I think I am going to slurp that one for v3.13-rc1

> * xend still in tree (x)
>  - xl list -l on a dom0-only system
>  - xl list -l doesn't contain tty console port
>  - xl Alternate transport support for migration*
>  - xl PVSCSI support
>  - xl PVUSB support

Add this one please:
  -xl needs to disallow PoD with PCI passthrough
   (see http://xen.1045712.n5.nabble.com/PATCH-VT-d-Dis-allow-PCI-device-assignment-if-PoD-is-enabled-td2547788.html)

I believe some of these were tacked by Wei - but he has
been doing other things. And I am busy right now tackling
bugs.

> * SWIOTLB (kernel side thing)
>   owner: Stefano
>   status: Pull request sent.

In v3.13.

> * Disk: indirect descriptors
>    owner: roger@citrix
>    status: Linux side in 3.11, Xen-side patch posted

I think you can drop this from your list.

Re: Xen 4.4 development update: Code freezing point reached

[Jan Beulich]

>>> On 19.11.13 at 15:40, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
>> The idea is that today we will be pretty permissive, but that we will
>> become progressively more conservative until the first RC, which is
>> scheduled for 3 weeks' time (6 December).  After that, we will only
>> accept bug fixes.
>> 
>> Bug fixes can be checked in without a freeze exception throughout the
>> code freeze, unless the maintianer thinks they are particularly high
>> risk.  In later RC's, we may even begin rejecting bug fixes if the
>> broken functionality is small and the risk to other functionality is
>> high.
>> 
>> Features which are currently marked "experimental" or do not at the
>> moment work at all cannot be broken really; so changes to code only
>> used by those features should be able to get a freeze exception
>> easily.  (Tianocore is something which would probably fall under
>> this.)
>> 
>> Features which change or add new interfaces which will need to be
>> supported in a backwards-compatible way (for instance, vNUMA) will
>> need freeze exceptions to make sure that the interface itself has
>> enough time to be considered stable.
>> 
>> These are guidelines and principles to give you an idea where we're
>> coming from; if you think there's a good reason why making an
>> exception for you will help us achieve goals 1-3 above better than not
>> doing so, feel free to make your case.
> 
> I am wondering in which category the tmem cleanup patches fall?
> 
> They aren't bug-fixes, they could be considered a feature. They were
> posted before the deadline. I posted the GIT PULL (see
> http://article.gmane.org/gmane.comp.emulators.xen.devel/178043)
> to one of the folks who has write access to the repository (as
> documented in http://www.xenproject.org/governance.html)? 

Considering the state of TMEM, I think these could go in at almost
any time. (Btw, I would have committed them already, but wasn't
up to trying out my first git pull from a foreign tree, due to my
expectation of it not going to work as I expect the first time
through, and there's other more important work that needs my
attention first. And of course you sent the pull request to Keir
only anyway...)

Jan

Re: Xen 4.4 development update: Code freezing point reached

[Konrad Rzeszutek Wilk]

On Tue, Nov 19, 2013 at 02:54:19PM +0000, Jan Beulich wrote:
> >>> On 19.11.13 at 15:40, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
> >> The idea is that today we will be pretty permissive, but that we will
> >> become progressively more conservative until the first RC, which is
> >> scheduled for 3 weeks' time (6 December).  After that, we will only
> >> accept bug fixes.
> >> 
> >> Bug fixes can be checked in without a freeze exception throughout the
> >> code freeze, unless the maintianer thinks they are particularly high
> >> risk.  In later RC's, we may even begin rejecting bug fixes if the
> >> broken functionality is small and the risk to other functionality is
> >> high.
> >> 
> >> Features which are currently marked "experimental" or do not at the
> >> moment work at all cannot be broken really; so changes to code only
> >> used by those features should be able to get a freeze exception
> >> easily.  (Tianocore is something which would probably fall under
> >> this.)
> >> 
> >> Features which change or add new interfaces which will need to be
> >> supported in a backwards-compatible way (for instance, vNUMA) will
> >> need freeze exceptions to make sure that the interface itself has
> >> enough time to be considered stable.
> >> 
> >> These are guidelines and principles to give you an idea where we're
> >> coming from; if you think there's a good reason why making an
> >> exception for you will help us achieve goals 1-3 above better than not
> >> doing so, feel free to make your case.
> > 
> > I am wondering in which category the tmem cleanup patches fall?
> > 
> > They aren't bug-fixes, they could be considered a feature. They were
> > posted before the deadline. I posted the GIT PULL (see
> > http://article.gmane.org/gmane.comp.emulators.xen.devel/178043)
> > to one of the folks who has write access to the repository (as
> > documented in http://www.xenproject.org/governance.html)? 
> 
> Considering the state of TMEM, I think these could go in at almost
> any time. (Btw, I would have committed them already, but wasn't
> up to trying out my first git pull from a foreign tree, due to my
> expectation of it not going to work as I expect the first time

:-)
> through, and there's other more important work that needs my
> attention first. And of course you sent the pull request to Keir
> only anyway...)

Duh! I will make sure to send it all committers in the hypervisor
next time.

Thanks!

Re: Xen 4.4 development update: Code freezing point reached

[George Dunlap]

On 11/18/2013 07:22 PM, Pasi Kärkkäinen wrote:
> On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
>>
>> = Timeline =
>>
>> Here is our current timeline based on a 6-month release:
>>
>> * Feature freeze: 18 October 2013
>> * Code freezing point: 18 November 2013 <== WE ARE HERE
>> * First RC: 6 December 2013
>> * Release: 21 January 2014
>>
>> Last updated: 18 November 2016
>>
>
> 2016 :)
>
>
>>
>> == Backlog ==
>>
>>
>> === Big ticket items ===
>>
>
>   * PCI hole resize support hvmloader/qemu-traditional/qemu-upstream with PCI/GPU passthrough
>     http://lists.xen.org/archives/html/xen-devel/2013-05/msg02813.html
>     Where Stefano writes:
>
>     1) modify upstream QEMU to start the PCI hole at 0xe0000000, to match
>     qemu-xen-unstable in terms of configuration and not to introduce any
>     regressions. Do this for the Xen 4.3 release.
>
>     2) for Xen 4.4 rework the two patches above and improve
>     i440fx_update_pci_mem_hole: resizing the pci_hole subregion is not
>     enough, it also needs to be able to resize the system memory region
>     (xen.ram) to make room for the bigger pci_hole

Right -- I'll put it on my list; but given that it needs a new interface 
into qemu, and work hasn't even begun on that yet, I rather doubt it's 
going to make it for 4.4, unfortunately. :-(

  -George

Re: Xen 4.4 development update: Code freezing point reached

[George Dunlap]

On 11/19/2013 12:59 AM, Don Slutz wrote:
> On 11/18/13 13:19, George Dunlap wrote:
>> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>>   http://wiki.xen.org/wiki/Xen_Roadmap/4.4
> [snip]
>
> I realize that I did not send anything to you before this.  However:
>
> http://lists.xen.org/archives/html/xen-devel/2013-10/msg01117.html
> http://lists.xen.org/archives/html/xen-devel/2013-11/msg02352.html
> http://lists.xen.org/archives/html/xen-devel/2013-11/msg02569.html
>
> Is about a new minor feature.  Clearly I would like it in 4.4.  This is
> close to:
>
>     xl dump-core <Domain> <filename>
>     crash <filename>
>
> But does not require the disk space, nor the delay while writing the
> file.  For me it took ~2 minutes to write out a 3G core file.
>      -Don Slutz

Got it, thanks.

  -George

Re: Xen 4.4 development update: Code freezing point reached

[George Dunlap]

On 11/19/2013 02:40 PM, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
>> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>>   http://wiki.xen.org/wiki/Xen_Roadmap/4.4
>>
>> (And I actually updated the wiki this time.)
>>
>> The code "freezing point" is today; which means that starting today
>> non-bug fixes need a freeze exception to be included.
>>
>> Remember our goal for the release:
>>   1. A bug-free release
>>   2. An awesome release
>>   3. An on-time release
>>
>> Accepting a new feature may make Xen more awesome; but it also
>> introduces a risk that it will introduce more bugs.  That bug may be
>> found before the release (threatening #3), or it may not be found
>> until after the release (threatening #1).  Each freeze exception
>> request will attempt to balance the benefits (how awesome the
>> exception is) vs the risks (will it cause the release to slip, or
>> worse, cause a bug which goes un-noticed into the final release).
>>
>> The idea is that today we will be pretty permissive, but that we will
>> become progressively more conservative until the first RC, which is
>> scheduled for 3 weeks' time (6 December).  After that, we will only
>> accept bug fixes.
>>
>> Bug fixes can be checked in without a freeze exception throughout the
>> code freeze, unless the maintianer thinks they are particularly high
>> risk.  In later RC's, we may even begin rejecting bug fixes if the
>> broken functionality is small and the risk to other functionality is
>> high.
>>
>> Features which are currently marked "experimental" or do not at the
>> moment work at all cannot be broken really; so changes to code only
>> used by those features should be able to get a freeze exception
>> easily.  (Tianocore is something which would probably fall under
>> this.)
>>
>> Features which change or add new interfaces which will need to be
>> supported in a backwards-compatible way (for instance, vNUMA) will
>> need freeze exceptions to make sure that the interface itself has
>> enough time to be considered stable.
>>
>> These are guidelines and principles to give you an idea where we're
>> coming from; if you think there's a good reason why making an
>> exception for you will help us achieve goals 1-3 above better than not
>> doing so, feel free to make your case.
>
> I am wondering in which category the tmem cleanup patches fall?
>
> They aren't bug-fixes, they could be considered a feature. They were
> posted before the deadline. I posted the GIT PULL (see
> http://article.gmane.org/gmane.comp.emulators.xen.devel/178043)
> to one of the folks who has write access to the repository (as
> documented in http://www.xenproject.org/governance.html)?

But it's still marked "experimental", right?  As long as it only touches 
tmem code, it should be OK until fairly late.

>
>
>> == Open ==
>>
>> * qemu-upstream not freeing pirq
>>   > http://www.gossamer-threads.com/lists/xen/devel/281498
>>   status: patches posted; latest patches need testing
>
> Duan, ping?
>
>>
>> * Race in PV shutdown between tool detection and shutdown watch
>>   > http://www.gossamer-threads.com/lists/xen/devel/282467
>>   > Nothing to do with ACPI
>>   status: Patches posted
>
> I think I am going to slurp that one for v3.13-rc1

Cool, let me know when it's in.

>
>> * xend still in tree (x)
>>   - xl list -l on a dom0-only system
>>   - xl list -l doesn't contain tty console port
>>   - xl Alternate transport support for migration*
>>   - xl PVSCSI support
>>   - xl PVUSB support
>
> Add this one please:
>    -xl needs to disallow PoD with PCI passthrough
>     (see http://xen.1045712.n5.nabble.com/PATCH-VT-d-Dis-allow-PCI-device-assignment-if-PoD-is-enabled-td2547788.html)

Done.

>
> I believe some of these were tacked by Wei - but he has
> been doing other things. And I am busy right now tackling
> bugs.
>
>> * SWIOTLB (kernel side thing)
>>    owner: Stefano
>>    status: Pull request sent.
>
> In v3.13.

Sweet!

>
>> * Disk: indirect descriptors
>>     owner: roger@citrix
>>     status: Linux side in 3.11, Xen-side patch posted
>
> I think you can drop this from your list.

I've moved it to the "completed" section.

Thanks!
  -George

Re: Xen 4.4 development update: Code freezing point reached

[Konrad Rzeszutek Wilk]

On Tue, Nov 19, 2013 at 05:52:39PM +0000, George Dunlap wrote:
> On 11/19/2013 02:40 PM, Konrad Rzeszutek Wilk wrote:
> >On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
> >>This information will be mirrored on the Xen 4.4 Roadmap wiki page:
> >>  http://wiki.xen.org/wiki/Xen_Roadmap/4.4
> >>
> >>(And I actually updated the wiki this time.)
> >>
> >>The code "freezing point" is today; which means that starting today
> >>non-bug fixes need a freeze exception to be included.
> >>
> >>Remember our goal for the release:
> >>  1. A bug-free release
> >>  2. An awesome release
> >>  3. An on-time release
> >>
> >>Accepting a new feature may make Xen more awesome; but it also
> >>introduces a risk that it will introduce more bugs.  That bug may be
> >>found before the release (threatening #3), or it may not be found
> >>until after the release (threatening #1).  Each freeze exception
> >>request will attempt to balance the benefits (how awesome the
> >>exception is) vs the risks (will it cause the release to slip, or
> >>worse, cause a bug which goes un-noticed into the final release).
> >>
> >>The idea is that today we will be pretty permissive, but that we will
> >>become progressively more conservative until the first RC, which is
> >>scheduled for 3 weeks' time (6 December).  After that, we will only
> >>accept bug fixes.
> >>
> >>Bug fixes can be checked in without a freeze exception throughout the
> >>code freeze, unless the maintianer thinks they are particularly high
> >>risk.  In later RC's, we may even begin rejecting bug fixes if the
> >>broken functionality is small and the risk to other functionality is
> >>high.
> >>
> >>Features which are currently marked "experimental" or do not at the
> >>moment work at all cannot be broken really; so changes to code only
> >>used by those features should be able to get a freeze exception
> >>easily.  (Tianocore is something which would probably fall under
> >>this.)
> >>
> >>Features which change or add new interfaces which will need to be
> >>supported in a backwards-compatible way (for instance, vNUMA) will
> >>need freeze exceptions to make sure that the interface itself has
> >>enough time to be considered stable.
> >>
> >>These are guidelines and principles to give you an idea where we're
> >>coming from; if you think there's a good reason why making an
> >>exception for you will help us achieve goals 1-3 above better than not
> >>doing so, feel free to make your case.
> >
> >I am wondering in which category the tmem cleanup patches fall?
> >
> >They aren't bug-fixes, they could be considered a feature. They were
> >posted before the deadline. I posted the GIT PULL (see
> >http://article.gmane.org/gmane.comp.emulators.xen.devel/178043)
> >to one of the folks who has write access to the repository (as
> >documented in http://www.xenproject.org/governance.html)?
> 
> But it's still marked "experimental", right?  As long as it only

No idea what it is marked as. You have to use 'tmem=1' to actually
enable it - so it is no by default enabled.
> touches tmem code, it should be OK until fairly late.

OK.
> 
> >
> >
> >>== Open ==
> >>
> >>* qemu-upstream not freeing pirq
> >>  > http://www.gossamer-threads.com/lists/xen/devel/281498
> >>  status: patches posted; latest patches need testing
> >
> >Duan, ping?
> >
> >>
> >>* Race in PV shutdown between tool detection and shutdown watch
> >>  > http://www.gossamer-threads.com/lists/xen/devel/282467
> >>  > Nothing to do with ACPI
> >>  status: Patches posted
> >
> >I think I am going to slurp that one for v3.13-rc1
> 
> Cool, let me know when it's in.

Will do.

Re: Xen 4.4 development update: Code freezing point reached

[Zhenzhong Duan]

On 2013-11-19 22:40, Konrad Rzeszutek Wilk wrote:
> On Mon, Nov 18, 2013 at 06:19:46PM +0000, George Dunlap wrote:
>> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>>   http://wiki.xen.org/wiki/Xen_Roadmap/4.4
>>
>> (And I actually updated the wiki this time.)
>>
>> The code "freezing point" is today; which means that starting today
>> non-bug fixes need a freeze exception to be included.
>>
>> Remember our goal for the release:
>>   1. A bug-free release
>>   2. An awesome release
>>   3. An on-time release
>>
>> Accepting a new feature may make Xen more awesome; but it also
>> introduces a risk that it will introduce more bugs.  That bug may be
>> found before the release (threatening #3), or it may not be found
>> until after the release (threatening #1).  Each freeze exception
>> request will attempt to balance the benefits (how awesome the
>> exception is) vs the risks (will it cause the release to slip, or
>> worse, cause a bug which goes un-noticed into the final release).
>>
>> The idea is that today we will be pretty permissive, but that we will
>> become progressively more conservative until the first RC, which is
>> scheduled for 3 weeks' time (6 December).  After that, we will only
>> accept bug fixes.
>>
>> Bug fixes can be checked in without a freeze exception throughout the
>> code freeze, unless the maintianer thinks they are particularly high
>> risk.  In later RC's, we may even begin rejecting bug fixes if the
>> broken functionality is small and the risk to other functionality is
>> high.
>>
>> Features which are currently marked "experimental" or do not at the
>> moment work at all cannot be broken really; so changes to code only
>> used by those features should be able to get a freeze exception
>> easily.  (Tianocore is something which would probably fall under
>> this.)
>>
>> Features which change or add new interfaces which will need to be
>> supported in a backwards-compatible way (for instance, vNUMA) will
>> need freeze exceptions to make sure that the interface itself has
>> enough time to be considered stable.
>>
>> These are guidelines and principles to give you an idea where we're
>> coming from; if you think there's a good reason why making an
>> exception for you will help us achieve goals 1-3 above better than not
>> doing so, feel free to make your case.
> I am wondering in which category the tmem cleanup patches fall?
>
> They aren't bug-fixes, they could be considered a feature. They were
> posted before the deadline. I posted the GIT PULL (see
> http://article.gmane.org/gmane.comp.emulators.xen.devel/178043)
> to one of the folks who has write access to the repository (as
> documented in http://www.xenproject.org/governance.html)?
>
>
>> == Open ==
>>
>> * qemu-upstream not freeing pirq
>>   > http://www.gossamer-threads.com/lists/xen/devel/281498
>>   status: patches posted; latest patches need testing
> Duan, ping?
I didn't have a test env to test this. This need to reimage the env.

Regards
zduan