Re: Xen 4.4 development update

[George Dunlap <george.dunlap@eu.citrix.com>]

On 08/08/13 20:30, Konrad Rzeszutek Wilk wrote:
> On Thu, Aug 08, 2013 at 05:09:35PM +0100, George Dunlap wrote:
>> This information will be mirrored on the Xen 4.4 Roadmap wiki page:
>>   http://wiki.xen.org/wiki/Xen_Roadmap/4.4
>>
>> Rather than try to predict precisely what will make it into what
>> release (which was something of a disaster last release), I'm just
>> going to borrow a term from the Agile world and call all uncompleted
>> features the "Backlog".  I'll still track who is doing what, and when
>> we get close, what state things seem to be in.
>>
>> As mentioned in another e-mail, we'll also be working on improving the
>> regression tester.  Feel free to join us.
>>
>> And as always, if you are working on a feature / bug that you want
>> tracked, please respond to this e-mail.
>>
>> = Timeline =
>>
>> As discussed elsewhere, I am proposing a 6-month release cycle.  Xen
>> 4.3 was released on 9 July.  That would give us a release on 9 January
>> 2014.  This is fairly close after the Christmas season, so I propose
>> to make the estimated release date later, on 21 January, giving a few
>> extra weeks for the holiday season:
>>
>> * Feature freeze: 18 October 2013
>> * Code freezing point: 8 November 2013
>> * First RC: 26 November 2013
>> * Release: 21 January 2014
>>
>> Feedback on the estimated dates  is welcome.
>>
>> Last updated: 8 August 2013
>>
>> == Completed ==
>>
>> [none]
>>
>> == Open ==
>>
>> * xend still in tree
>>
>> * qemu-upstream not freeing pirq
>>   > http://www.gossamer-threads.com/lists/xen/devel/281498
>>   status: patches posted
> They did fix the problem Oracle saw and I believe have been
> Tested-and-Acked.

Thanks -- have they also been applied?

>
>> * __update_vcpu_system_time if system_time comes out negative
>>
>> * xl pci-detach broken for pv guests?
>>    > http://bugs.xenproject.org/xen/bug/12
>>    > kernel doesn't know about moving into states 7 and 8
>>    status: External
> Didn't I post a patch for this on Linux - and it is in Linux kernel
> already.

Great -- thanks for the update.

>> * Race in PV shutdown between tool detection and shutdown watch
>>   > http://www.gossamer-threads.com/lists/xen/devel/282467
>>   > Nothing to do with ACPI
>>   status: Probably a bug in Linux xen drivers
>>
>> == Backlog ==
>>
>> === Testing coverage ===
>>
>> * Network driver domains
>>   @George
>>
>> * new libxl w/ previous versions of xl
>>   @IanJ
>>
>> * Host S3 suspend
>>   @bguthro
>>
>> * Default [example] XSM policy
>>   @Stefano to ask Daniel D
>>
>> * Xen on ARM
>>   # hardware
>>    @ianc
>>     emulator: @stefano to think about it
>>
>> * Storage driver domains
>>   @roger
>>
>> * HVM pci passthrough
>>   @anthony
>>
>> * Nested virt?
>>   @intel (chased by George)
>>
>> * Fix SRIOV test (chase intel)
>>   @ianj
>>
>> * Fix bisector to e-mail blame-worthy parties
>>   @ianj
>>
>> * Fix xl shutdown
>>    @ianj
>>
>> * stub domains
>>    @athony
>>
>> === Big ticket items ===
>>
>> * NUMA Memory migration
>>    owner: dario@citrix
>>    status: in progress
>>
>> * Event channel scalability
>>    owner: david@citrix
>>    status: RFC v5 submitted
>>    Increase limit on event channels (currently 1024 for 32-bit guests,
>>    4096 for 64-bit guests)
> aka FIFO thing right?

Yep.  I'll make this a bit more clear.

>
>> * PVH mode (w/ Linux)
>>    owner: mukesh@oracle
>>    status (Linux): 3rd draft patches posted.
> More like v8. And already Acked - just waiting for ABI to be in some way
> "nailed down".
>
>>    status (Xen): v10 posted
>>
>> * ARM stuff: ??
>>
>> * Meta: PVIO NUMA improvements
>>   - NUMA affinity for vcpus
>>      owner: Dario
>>   - PV guest NUMA interface
>>      owner: Elena
>>   - Sensible dom0 NUMA layout
>>   - Toolstack pinning backend thread / virq to appropraite d0 vcpu
>>
>> * qemu-upstream stubdom, Linux
>>     owner: anthony@citrix
>>     status: in progress
>>     qemu-upstream needs a more fully-featured libc than exists in
>>     mini-os.  Either work on a minimalist linux-based stubdom with
>>     glibc, or port one of the BSD libcs to minios.
>>
>> * qemu-upstream stubdom, BSD libc
>>    owner: ianj@citrix
>>
>> * Network performance improvements
>>    owner: wei@citrix
>>
>> * Disk performance improvements
> I think you mean blkback and blkfront?
>> * Xen EFI feature: Xen can boot from grub.efi
>>   owner: Daniel Kiper
>>   status: Just begun
>>   prognosis: Fair
>>
>> * libvirt/libxl integration (external)
>>   > need a status update
>>   - owner: jfehlig@suse, dario@citrix
>>
>> * Default to credit2
>>   - cpu pinning
>>   - NUMA affinity
>>   - cpu "reservation"
>>
>> * xenperf
>>    Owner: Boris Otovsky
>>
>> * blktap3
>>    owner: thanos@citrix
>>    status: on hold pending XenServer decision
>>
>> * Nested virtualization on Intel
>>
>> * Nested virtualization on AMD
>>
>> * Make storage migration possible
>>    owner: ?
>>    status: none
>>    There needs to be a way, either via command-line or via some hooks,
>>    that someone can build a "storage migration" feature on top of libxl
>>    or xl.
>>
>> * Full-VM snapshotting
>>    owner: ?
>>    status: none
>>    Have a way of coordinating the taking and restoring of VM memory and
>>    disk snapshots.  This would involve some investigation into the best
>>    way to accomplish this.
>>
>> * VM Cloning
>>    owner: ?
>>    status: none
>>    Again, a way of coordinating the memory and disk aspects.  Research
>>    into the best way to do this would probably go along with the
>>    snapshotting feature.
>>
>> * xl vm-{export,import}
>>    owner: ?
>>    status: none
>>    Allow xl to import and export VMs to other formats; particularly
>>    ovf, perhaps the XenServer format, or more.
>>
>> * Memory: Replace PoD with paging mechanism
>>    owner: george@citrix
>>    status: none
>>
>> * PV audio (audio for stubdom qemu)
>>    owner: stefano.panella@citrix
>>    status: ?
>>
>> * Wait queues for mm
>>   > Needed for more advanced paging schemes
>>    owner: ?
>>    status: Draft posted Feb 2012; more work to do.
>>
>> * V4V: Inter-domain communication
>>    owner (Xen): dominic.curran@citrix.com
>>    status (Xen): patches submitted
>>    owner (Linux driver):  stefano.panella@citrix
>>    status (Linux driver): in progress
>>
>> === clean-ups ===
>>
>> * Sort out better memory / ballooning / dom0 autoballooning thing
>>   > Don't forget NUMA angle
>>   - Inaccurate / incomplete info from HV
>>
>> * Implement Xen hypervisor dmesg log entry timestamps
>>
>> * Make network driver domains easier to set up / more useful
>>   - Default backend (submitted)
>>   - Remove unnecessary restriction wrt dom0 hotplug scripts (submitted)
>>   - Make it easy to make a device assignable (in discussion)
>>   - Automatically start/shutdown (xendomains?)
>>   - Pause booting of other domains until network driver domain is up
>>
>> * libxl: More fine-grained control over when to pass through a device
>>   > Some IOMMUs are secure; some are merely functional, some are not present.
>>   > Allow the adminitrator to set the default
>>
>> * xl does not handle migrate interruption gracefully
>>    > If you start a localhost migrate, and press "Ctrl-C" in the middle,
>>    > you get two hung domains
>>    status: Probably not for 4.3
>>
>> * libxl / xl does not handle failure of remote qemu gracefully
>>    > Easiest way to reproduce:
>>    >  - set "vncunused=0" and do a local migrate
>>    >  - The "remote" qemu will fail because the vnc port is in use
>>    > The failure isn't the problem, but everything being stuck afterwards is
>>    status: Probably not for 4.3
>>
>> * mac address changes on reboot if not specified in config file
>>    > Needs a robust way to "add" to the config
>>    status: Too much for 4.3
>>
>> * qxl
>>    > http://bugs.xenproject.org/xen/bug/11http://bugs.xenproject.org/xen/bug/11
>>    - Uninitialized struct element in qemu
>>    - Revert 5479961 to re-enable qxl in xl,libxl
>>    - Option in Xen top-level to enable qxl support in qemu tree
>>    - Fix sse2 MMIO issue
>>
>> * libxl config file
>>
>> * libxl: Don't use RAW format for "URL"-based qdisks (e.g., rbd:rbd/foo.img)
>>    - Figure out whether to use a generic URL or have a specific type for each one
>>    - Check existence of disk file for all RAW
>>
>> * Polish up xenbugtool
>>    owner: wei.liu2@citrix.com
>>
>> * acpi-related xenstore entries not propagated on migrate
>>   > http://www.gossamer-threads.com/lists/xen/devel/282466
>>   > Only used by hvmloader; only a clean-up, not a bug.
>>   status: Not for 4.3
>>
>> * Remove hardcoded mobprobe's in xencommons
>>    owner: Wei Liu
>>    status: still in discussion
>>
>> * xl USB pass-through for HVM guests using Qemu USB emulation
>>    owner: George
>>    status: v6 patch series posted
>>
>> * Rationalized backend scripts
>>    owner: roger@citrix
>>    status: patches posted
>>
>> * Scripts for driver domains (depends on backend scripts)
>>    owner: roger@citrix
>>    status:
>>
>> * Multi-vector PCI MSI (support at least for Dom0)
>>    owner: jan@suse
>>    status: Patches posted for Intel; AMD not yet done
> And patches for Linux upstream are missing.



>
>> * xl: passing more defaults in configuration in xl.conf
>>    owner: ?
>>    There are a number of options for which it might be useful to pass a
>>    default in xl.conf.  For example, if we could have a default
>>    "backend" parameter for vifs, then it would be easy to switch back
>>    and forth between a backend in a driver domain and a backend in dom0.
>>
>> * xl PVUSB pass-through for PV guests
>> * xl PVUSB pass-through for HVM guests
>>    owner: George
>>    status: ?
>>    xm/xend supports PVUSB pass-through to guests with PVUSB drivers
>> (both PV and HVM guests).
>>    - port the xm/xend functionality to xl.
>>    - this PVUSB feature does not require support or emulation from Qemu.
>>    - upstream the Linux frontend/backend drivers. Current
>> work-in-progress versions are in Konrad's git tree.
> Which is to say I hadn't actually touched them in a year.

Yeah, I was under the impression they weren't really going anywhere.  If 
you think this is unclear, feel free to suggest alternate wording.

>
>>    - James Harper's GPLPV drivers for Windows include PVUSB frontend drivers.
>>
>> * Guest EFI booting
>>   - status: tianocore in-tree, some build problems.
>>     Needs new owner.
>>
>> * Xen EFI feature: pvops dom0 able to make use of EFI run-time
>> services (external)
>>   owner: Daniel Kiper
>>   status: Just begun
>>
>> * Serial console improvements
>>    owner: ?
>>    status: Stalled (see below)
>>    -xHCI debug port (Needs hardware)
>>    -Firewire (needs hardware)
>>
> I would like also to put GPU passthrough working here.

What aspect do you mean?

  -George