[Hackathon Minutes] Xen 4.4 Planning

[Hackathon Minutes] Xen 4.4 Planning

[Lars Kurth]

This took me a while to post, but given that we are not starting 4.4 
just yet, this may be appropriate now. I may have misrepresented some 
stuff as it has been 4 weeks since I wrote these.
Cheers
Lars

= Purpose of Roadmap =
* Set a vision for interesting features
* Track items
* Help consumers of Xen with their planning

= Release Models that work well =
There was a brief discussion on two different release models
* Train leaves the station (Linux)
* Release when ready (Debian)

== Stefano's Proposal ==
We should aim to reduce the release cycle to 4 months (or maybe 6 months 
as an intermediate step) from the current 9 months. A 4 months relase 
cycle should help accelerate development and lead to fewer patches being 
queued up. The implications are that we would have to operate a 2-3 
weeks merge window.

To do this, we would need to resolve a number of issues
* It is likely that code reviews for such a short merge window would 
become a bottleneck. We are not sure whether this would be a major issue 
: the review bandwith would depend on the patches submitted (and their 
complexity)
* [I can't remember who raised this] The concern was raised that we 
would not have enough x86 Xen reviewers got a 2-3 weeks merge window
* [Konrad] Stated that in PVOPS for Linux contributions we don't have a 
review bottleneck, but we should make sure that the Xen and PVOPS/Linux 
merge window don't overlap (as this would require the same set of people 
to review patches in two projects)
* The rest of the time (approx 3 months) would be used for stabilizing 
the release
* If we had a more extensive testing framework, and thus better testing, 
we could tighten the RC period (making this happen is also being 
discussed by the Advisory Board)

Additional conerns raised:
* [Matt Wilson]: if we had shorter merge windows, there is a risk that 
we would end up with unnused code (uncompleted features) in mainline. 
Something which we'd rather not have
* [I can't remember who raised this] But we already have a buffer via 
staging : we could make more use of this

[Conclusion] We aLL agreed, that a release cycle of less than 9 months 
is desirable. Maybe we can go to 6 months for Xen 4.4 (before being more 
aggressive).

= 4.3 Release cycle : what worked well / didn't work well =
* The 4.3 release updates and criteria went well
* BUT :  50% of what was supposed to be in 4.3 didn't make it
** In some cases, we simply underestimated the effort that is needed. 
Concrete example : QEMU/stubdomain was a combination of under-estimating 
the size and over-estimating the development bandwidth that was available
** Some of the high-impact features (e.g. PVH) came in too late in the 
dev cycle. Mitigation : break contributions into smaller parts and 
submit earlier in the merge window. The same applies to changes to 
generic code.
* BUT : Some patches were lost (i.e. when there are spikes of activity 
it becomes hard for some maintainers/committers to keep on top of their 
queue).
** [Ian Campell] said that we should rely on submitter to resend the 
patch: the assumption is that if the patch is not important, the 
submitter will badger and resend.
** [Lars] raised the point that this can alienate contributors and get 
them to look at other projects instead.
** [Can't remember who said this] Maybe use patchwork 
(http://jk.ozlabs.org/projects/patchwork/) to track patches
** [Ian Campbell]: patchwork looks like a good idea, but may not work 
well in practice
[Note] We should probably have a discussion or some sort of trial. It 
may also be possible to use the http://bugs.xenproject.org prototype (or 
add a "deferred patch" attribute)
[Note] We typically start opening the dev branch at RC5/6 (not sure I 
quite got this)
[Note] We don't actually have a list of patches that got lost in the 4.3 
release cycle )-:

[ACTION]: George write up a proposal for the beginning of the 4.4 
release cycle

= 4.4 Content =
George volunteers to be the Release co-ordinator for Xen 4.4 (to apply 
what he learned)

* Big features that did not make it in 4.3

* PVH? What will make it into 4.4

* Missed patches (don't have a list)

* "User" features that look interesting

** Network effects

** We shouldn't have broken feature

** What about XenClient / VirtualComputer being able to help out in 
adding mopre support for PCI / VGA cards

** Other features: Sharing dom0 keyboard / mouse


* GPU passthrough
** We have an issue with graphics card support
** A lot of users care about GPU passthrough (but not many vendors)
** Maybe we could mentor somebody about GPU passthrough?

** Maybe we could get XenClient, VirtualComputer or Qubes to pick this 
up (or partly do so)?

Re: [Hackathon Minutes] Xen 4.4 Planning

[Jan Beulich]

>>> On 13.06.13 at 16:00, Lars Kurth <lars.kurth@xen.org> wrote:
> = Release Models that work well =
> There was a brief discussion on two different release models
> * Train leaves the station (Linux)
> * Release when ready (Debian)
> 
> == Stefano's Proposal ==
> We should aim to reduce the release cycle to 4 months (or maybe 6 months 
> as an intermediate step) from the current 9 months. A 4 months relase 
> cycle should help accelerate development and lead to fewer patches being 
> queued up. The implications are that we would have to operate a 2-3 
> weeks merge window.
> 
> To do this, we would need to resolve a number of issues
> * It is likely that code reviews for such a short merge window would 
> become a bottleneck. We are not sure whether this would be a major issue 
> : the review bandwith would depend on the patches submitted (and their 
> complexity)
> * [I can't remember who raised this] The concern was raised that we 
> would not have enough x86 Xen reviewers got a 2-3 weeks merge window
> * [Konrad] Stated that in PVOPS for Linux contributions we don't have a 
> review bottleneck, but we should make sure that the Xen and PVOPS/Linux 
> merge window don't overlap (as this would require the same set of people 
> to review patches in two projects)
> * The rest of the time (approx 3 months) would be used for stabilizing 
> the release
> * If we had a more extensive testing framework, and thus better testing, 
> we could tighten the RC period (making this happen is also being 
> discussed by the Advisory Board)
> 
> Additional conerns raised:
> * [Matt Wilson]: if we had shorter merge windows, there is a risk that 
> we would end up with unnused code (uncompleted features) in mainline. 
> Something which we'd rather not have
> * [I can't remember who raised this] But we already have a buffer via 
> staging : we could make more use of this
> 
> [Conclusion] We aLL agreed, that a release cycle of less than 9 months 
> is desirable. Maybe we can go to 6 months for Xen 4.4 (before being more 
> aggressive).

I sincerely hope that we won't switch to this (Linux) model. Having
just a couple of weeks to get new code added means that everyone
halfway deeply involved in the development will have to permanently
carry a huge bag of patches. I'm thinking this is painful enough
already for the freeze periods we currently have.

That implies, however, that with an approximate freeze period of 3
months, shortening the release cycle to or even below 6 months is
going to be an issue. From the summary above I didn't really get
what's wrong with the current 9 month cycle, which - perhaps for
the first time since I joined the project - it seems like we can actually
meet with 4.3.

The only option I would see is to branch before RC1, thus keeping
the development tree open irrespective of the ongoing release. Of
course that's having the drawback of possibly/likely pulling away
the attention of some of us from the being released branch (I would
be very surprised if I wouldn't end up among those "some").

Jan

Re: [Hackathon Minutes] Xen 4.4 Planning

[George Dunlap]

On 13/06/13 15:22, Jan Beulich wrote:
>>>> On 13.06.13 at 16:00, Lars Kurth <lars.kurth@xen.org> wrote:
>> = Release Models that work well =
>> There was a brief discussion on two different release models
>> * Train leaves the station (Linux)
>> * Release when ready (Debian)
>>
>> == Stefano's Proposal ==
>> We should aim to reduce the release cycle to 4 months (or maybe 6 months
>> as an intermediate step) from the current 9 months. A 4 months relase
>> cycle should help accelerate development and lead to fewer patches being
>> queued up. The implications are that we would have to operate a 2-3
>> weeks merge window.
>>
>> To do this, we would need to resolve a number of issues
>> * It is likely that code reviews for such a short merge window would
>> become a bottleneck. We are not sure whether this would be a major issue
>> : the review bandwith would depend on the patches submitted (and their
>> complexity)
>> * [I can't remember who raised this] The concern was raised that we
>> would not have enough x86 Xen reviewers got a 2-3 weeks merge window
>> * [Konrad] Stated that in PVOPS for Linux contributions we don't have a
>> review bottleneck, but we should make sure that the Xen and PVOPS/Linux
>> merge window don't overlap (as this would require the same set of people
>> to review patches in two projects)
>> * The rest of the time (approx 3 months) would be used for stabilizing
>> the release
>> * If we had a more extensive testing framework, and thus better testing,
>> we could tighten the RC period (making this happen is also being
>> discussed by the Advisory Board)
>>
>> Additional conerns raised:
>> * [Matt Wilson]: if we had shorter merge windows, there is a risk that
>> we would end up with unnused code (uncompleted features) in mainline.
>> Something which we'd rather not have
>> * [I can't remember who raised this] But we already have a buffer via
>> staging : we could make more use of this
>>
>> [Conclusion] We aLL agreed, that a release cycle of less than 9 months
>> is desirable. Maybe we can go to 6 months for Xen 4.4 (before being more
>> aggressive).
> I sincerely hope that we won't switch to this (Linux) model. Having
> just a couple of weeks to get new code added means that everyone
> halfway deeply involved in the development will have to permanently
> carry a huge bag of patches. I'm thinking this is painful enough
> already for the freeze periods we currently have.

This would be much less painful if you used git branches. :-)

Seriously, check out stackgit -- it will make keeping track of all those 
patches, and of rebasing them, much much easier.  (Not to mention make 
it easier to send them, share them, merge them, &c.)

> That implies, however, that with an approximate freeze period of 3
> months, shortening the release cycle to or even below 6 months is
> going to be an issue. From the summary above I didn't really get
> what's wrong with the current 9 month cycle, which - perhaps for
> the first time since I joined the project - it seems like we can actually
> meet with 4.3.

The problem with the current cycle I think is that if you miss getting a 
feature into this release, it's going to be a full year and a half until 
the next release.  This means that there is increased pressure to try to 
"shove things in" at the last minute, and to ask for feature freeze 
exceptions.  If I was sure that 4.4 would be another 9-month cycle, I 
would have pushed much harder to make USB hot-plug support a blocker, 
for example.  That in turn causes less review, longer code freezes, and 
so on.

If there's a 4-month cadence, then if you miss one release, it's only 
another 4 months until the next release; not a big deal to wait.

> The only option I would see is to branch before RC1, thus keeping
> the development tree open irrespective of the ongoing release. Of
> course that's having the drawback of possibly/likely pulling away
> the attention of some of us from the being released branch (I would
> be very surprised if I wouldn't end up among those "some").

Ian C in fact proposed that this (doing bug-fixing, development, and 
review at the same time) might not be a bad thing.  It would make the 
"freeze" longer, but if we had shorter release cycles, and if people 
could still get review time and maybe even items merged, then having a 
long freeze might not actually be all that terrible.

Branching before RC1 would essentially be similar to the Linux model of 
"maintainers having their own branches", except that we'd just have one 
branch.  Given that there are only a handful of developers compared to 
Linux, that might not be such a bad model to start with.

Then there wouldn't actually be a merge window per se -- we'd just fork 
off a branch every N months and start stabilizing it.

  -George

Re: [Hackathon Minutes] Xen 4.4 Planning

[Jan Beulich]

>>> On 13.06.13 at 17:11, George Dunlap <george.dunlap@eu.citrix.com> wrote:
> The problem with the current cycle I think is that if you miss getting a 
> feature into this release, it's going to be a full year and a half until 
> the next release.

9 months != 1.5 years

>  This means that there is increased pressure to try to 
> "shove things in" at the last minute, and to ask for feature freeze 
> exceptions.  If I was sure that 4.4 would be another 9-month cycle, I 
> would have pushed much harder to make USB hot-plug support a blocker, 
> for example.  That in turn causes less review, longer code freezes, and 
> so on.
> 
> If there's a 4-month cadence, then if you miss one release, it's only 
> another 4 months until the next release; not a big deal to wait.

As long as releases are halfway predictable, I don't see 4 months
this significantly much better than 9 - those who don't get their
stuff ready in 4 months aren't that unlikely to also not finish it in
8 or 9.

But yes, I can see a 3+1 month cycle as still reasonable, albeit I'd
personally prefer longer cycles. That's not the least because at
least till now the window of older branches being maintained is
coupled to the main release cycle, and would hence also get
shortened, or the amount of work to maintain these branches
would increase simply because their number would grow - neither
of which is really desirable imo.

Jan

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Campbell]

On Thu, 2013-06-13 at 16:30 +0100, Jan Beulich wrote:
> But yes, I can see a 3+1 month cycle as still reasonable, albeit I'd
> personally prefer longer cycles. That's not the least because at
> least till now the window of older branches being maintained is
> coupled to the main release cycle, and would hence also get
> shortened, or the amount of work to maintain these branches
> would increase simply because their number would grow - neither
> of which is really desirable imo.

I raised this at the hackathon but didn't spot that it wasn't in the
minutes.

I think if we increase the release cadence we also need to tweak the
stable release policy, perhaps by nominating a fraction of releases for
longer term instead of the current "last 2". i.e perhaps we would
support the previous stable release in the normal way and every second
or third for a longer period (I'd need to sit down with a calendar to
figure out which cadence I really thought was sensible).

That of course undermines "there's a next release soon" since 1/N
releases somewhat become "special"...

Ian

Re: [Hackathon Minutes] Xen 4.4 Planning

[Fabio Fantoni]

About release time, I think probably a good choice could be the 3 months 
of development + 1 of testing but with Debian style: a minimum of 1 
month for testing and, if there are critical/important bugs, one more 
week until it will be solved.

About features for 4.4, I think that in the first post list some very 
good features already discussed are missing:
- Improve hvm domUs integration adding configure parameters to allow 
usage of packaged qemu and seabios instead of compiling the internal one.
About seabios and qemu upstream patch for configure already discussed 
with Ian Campbell, I tried to make a quick patch but failed (I couldn't 
find the right configure parameters to do it and the exact part to 
change on libxl for qemu upstream binary path), I can't find the 
discussion about it at the moment.

- Add support for SSE and AVX for the hvm domU to improve performance 
and solve qxl problem (qxl requires SSE):
http://bugs.xenproject.org/xen/bug/11

- Improve Spice support in libxl:
* vdagent, working and tested, patch already posted: 
http://lists.xen.org/archives/html/xen-devel/2013-05/msg00484.html
* usbredirection (different from usb passthrough: from spice client to 
qemu instead of from dom0 to qemu), working and tested, I want improve 
the patch before posting it, it doesn't look good but I'll post it anyway.
* support auto select of port similar to vnc, I'll probably try to make 
the patch.
* support spice also on pv domU as an alternative to vnc, I don't have 
enough knowledge to try to do it now.


What do you think?
Thanks for any reply

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Campbell]

On Mon, 2013-06-17 at 10:27 +0200, Fabio Fantoni wrote:
> - Improve hvm domUs integration adding configure parameters to allow 
> usage of packaged qemu and seabios instead of compiling the internal one.
> About seabios and qemu upstream patch for configure already discussed 
> with Ian Campbell, I tried to make a quick patch but failed (I couldn't 
> find the right configure parameters to do it and the exact part to 
> change on libxl for qemu upstream binary path), I can't find the 
> discussion about it at the moment.

I have a patch like this on my laptop (which is currently turned off at
home) which I wrote on the train to the last hackathon. It needs
cleaning up etc but I hope to post it in the 4.4 dev cycle.

Ian.

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Campbell]

On Thu, 2013-06-13 at 15:00 +0100, Lars Kurth wrote:
> == Stefano's Proposal ==
> We should aim to reduce the release cycle to 4 months (or maybe 6 months 
> as an intermediate step) from the current 9 months. A 4 months relase 
> cycle should help accelerate development and lead to fewer patches being 
> queued up. The implications are that we would have to operate a 2-3 
> weeks merge window.
> 
> To do this, we would need to resolve a number of issues
> * It is likely that code reviews for such a short merge window would 
> become a bottleneck. We are not sure whether this would be a major issue 
> : the review bandwith would depend on the patches submitted (and their 
> complexity)

If we do end up going for this model then I think we need to also
inherit the principal from Linux that the merge window is for *merging*
and that the code in question should have already been posted and
reviewed *before* the window opens (i.e. in the other 3 months of the
cycle). 

This would be something of a cultural change because at the minute
people seem to assume that a freeze means that development and review
must stop. Personally I don't think that should be the case anyway,
although I appreciate the need to focus peoples attention on the release
as well.

This could end up with subsystem maintainers queueing patches into their
own tree for merge into mainline during the window (this is what Linux
does). This has its own downsides WRT the merging work during window and
there would need to be a change since we would also have to ask people
to test those trees, which they aren't used to doing.

> * [I can't remember who raised this] The concern was raised that we 
> would not have enough x86 Xen reviewers got a 2-3 weeks merge window
> * [Konrad] Stated that in PVOPS for Linux contributions we don't have a 
> review bottleneck,

Largely due to Linux having the properties I described above, IMHO.
Although I'm not a Linux maintainer of a large subsystem so what would I
know ;-)

>  but we should make sure that the Xen and PVOPS/Linux 
> merge window don't overlap (as this would require the same set of people 
> to review patches in two projects)
> * The rest of the time (approx 3 months) would be used for stabilizing 
> the release
> * If we had a more extensive testing framework, and thus better testing, 
> we could tighten the RC period (making this happen is also being 
> discussed by the Advisory Board)
> 
> Additional conerns raised:
> * [Matt Wilson]: if we had shorter merge windows, there is a risk that 
> we would end up with unnused code (uncompleted features) in mainline. 
> Something which we'd rather not have
> * [I can't remember who raised this] But we already have a buffer via 
> staging : we could make more use of this

staging is simply the pre-test branch, I don't think it can serve as a
buffer of the type you seem to be suggesting.

We could encourage people to make more use of personal git trees,
including maintainers. 

> 
> [Conclusion] We aLL agreed, that a release cycle of less than 9 months 
> is desirable. Maybe we can go to 6 months for Xen 4.4 (before being more 
> aggressive).
> 
> = 4.3 Release cycle : what worked well / didn't work well =
> * The 4.3 release updates and criteria went well
> * BUT :  50% of what was supposed to be in 4.3 didn't make it

I think this is simply a normal part of the "train leaves the station"
model. There will always be stuff which misses, but then there is always
the next train, and the one after that.

To some extent I'm not sure how useful it is with the train model to try
and enumerate in advance the things which are going to be in a
particular release .

A list of things which people are working on and their current *target*
release seems more useful, with the knowledge that the targets can and
will change, probably more often than not for bigger items.

> ** In some cases, we simply underestimated the effort that is needed. 
> Concrete example : QEMU/stubdomain was a combination of under-estimating 
> the size and over-estimating the development bandwidth that was available
> ** Some of the high-impact features (e.g. PVH) came in too late in the 
> dev cycle. Mitigation : break contributions into smaller parts and 
> submit earlier in the merge window.

I would say "submit well in advance of the merge window".

>  The same applies to changes to 
> generic code.
> * BUT : Some patches were lost (i.e. when there are spikes of activity 
> it becomes hard for some maintainers/committers to keep on top of their 
> queue).
> ** [Ian Campell] said that we should rely on submitter to resend the 
> patch: the assumption is that if the patch is not important, the 
> submitter will badger and resend.

You have an extra "not" in there.

I consider the "badger and resend" as more of a fallback/backstop rather
than an intentional part of the workflow.

For my own part I do keep an explicit queue of mails containing patches
and I don't think I tend to miss many due to them simply falling through
the cracks, even during busts of activity. That's not to say I always
manage to process the queue in a timely manner though.

That said a little bit of badgering has the secondary effect of
reminding me that I need to look at the patch in my queue as well as the
primary affect of me enqueuing something I may have missed.

> ** [Lars] raised the point that this can alienate contributors and get 
> them to look at other projects instead.
> ** [Can't remember who said this] Maybe use patchwork 
> (http://jk.ozlabs.org/projects/patchwork/) to track patches
> ** [Ian Campbell]: patchwork looks like a good idea, but may not work 
> well in practice
> [Note] We should probably have a discussion or some sort of trial. It 
> may also be possible to use the http://bugs.xenproject.org prototype (or 
> add a "deferred patch" attribute)
> [Note] We typically start opening the dev branch at RC5/6 (not sure I 
> quite got this)

That's right, we usually fork around the late RCs when we are reaching
the end game.

> [Note] We don't actually have a list of patches that got lost in the 4.3 
> release cycle )-:

If they were on a list they by definition wouldn't be lost ;-)

FWIW I do have a separate queue of mails with patches which were
deferred to 4.4 (but only for stuff that I would have committed myself
had the tree not been frozen).

Ian.

Re: [Hackathon Minutes] Xen 4.4 Planning

[George Dunlap]

On 13/06/13 15:31, Ian Campbell wrote:
> On Thu, 2013-06-13 at 15:00 +0100, Lars Kurth wrote:
>> == Stefano's Proposal ==
>> We should aim to reduce the release cycle to 4 months (or maybe 6 months
>> as an intermediate step) from the current 9 months. A 4 months relase
>> cycle should help accelerate development and lead to fewer patches being
>> queued up. The implications are that we would have to operate a 2-3
>> weeks merge window.
>>
>> To do this, we would need to resolve a number of issues
>> * It is likely that code reviews for such a short merge window would
>> become a bottleneck. We are not sure whether this would be a major issue
>> : the review bandwith would depend on the patches submitted (and their
>> complexity)
> If we do end up going for this model then I think we need to also
> inherit the principal from Linux that the merge window is for *merging*
> and that the code in question should have already been posted and
> reviewed *before* the window opens (i.e. in the other 3 months of the
> cycle).
>
> This would be something of a cultural change because at the minute
> people seem to assume that a freeze means that development and review
> must stop. Personally I don't think that should be the case anyway,
> although I appreciate the need to focus peoples attention on the release
> as well.

I think these are interdependent: Because we view having a code freeze 
(and having people with out-of-tree patches) as the exception, we try to 
keep the code freeze as short as possible; which means prioritizing 
working on the bugs.  If we mixed development, review, and bug fixing, 
then the code freeze would necessarily be longer, as fixing bugs would 
be delayed.  However, this wouldn't necessarily be a terrible thing if 
everyone was expecting to keep out-of-tree branches anyway.

> I think this is simply a normal part of the "train leaves the station"
> model. There will always be stuff which misses, but then there is always
> the next train, and the one after that.

Yes -- I think this time we ended up sort of trying to do both; "The 
train is leaving at X time, what can we fit on before that time?"

Having more frequent trains means less need for trying to match up 
features with releases, as the impact of slipping one release is much lower.

  -George

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Campbell]

On Thu, 2013-06-13 at 15:52 +0100, George Dunlap wrote:
> On 13/06/13 15:31, Ian Campbell wrote:
> > On Thu, 2013-06-13 at 15:00 +0100, Lars Kurth wrote:
> >> == Stefano's Proposal ==
> >> We should aim to reduce the release cycle to 4 months (or maybe 6 months
> >> as an intermediate step) from the current 9 months. A 4 months relase
> >> cycle should help accelerate development and lead to fewer patches being
> >> queued up. The implications are that we would have to operate a 2-3
> >> weeks merge window.
> >>
> >> To do this, we would need to resolve a number of issues
> >> * It is likely that code reviews for such a short merge window would
> >> become a bottleneck. We are not sure whether this would be a major issue
> >> : the review bandwith would depend on the patches submitted (and their
> >> complexity)
> > If we do end up going for this model then I think we need to also
> > inherit the principal from Linux that the merge window is for *merging*
> > and that the code in question should have already been posted and
> > reviewed *before* the window opens (i.e. in the other 3 months of the
> > cycle).
> >
> > This would be something of a cultural change because at the minute
> > people seem to assume that a freeze means that development and review
> > must stop. Personally I don't think that should be the case anyway,
> > although I appreciate the need to focus peoples attention on the release
> > as well.
> 
> I think these are interdependent: Because we view having a code freeze 
> (and having people with out-of-tree patches) as the exception, we try to 
> keep the code freeze as short as possible; which means prioritizing 
> working on the bugs.  If we mixed development, review, and bug fixing, 
> then the code freeze would necessarily be longer, as fixing bugs would 
> be delayed.  However, this wouldn't necessarily be a terrible thing if 
> everyone was expecting to keep out-of-tree branches anyway.

Agreed. And as you say in your other mail 2 weeks dev + 3 months
stabilisation isn't a good fit for us, something more like 3 months dev
and 1 month stabilisation is a much better fit and avoids most of the
issues I was thinking of.

> > I think this is simply a normal part of the "train leaves the station"
> > model. There will always be stuff which misses, but then there is always
> > the next train, and the one after that.
> 
> Yes -- I think this time we ended up sort of trying to do both; "The 
> train is leaving at X time, what can we fit on before that time?"
> 
> Having more frequent trains means less need for trying to match up 
> features with releases, as the impact of slipping one release is much lower.

Agreed.

Ian.

Re: [Hackathon Minutes] Xen 4.4 Planning

[Konrad Rzeszutek Wilk]

On Thu, Jun 13, 2013 at 03:31:27PM +0100, Ian Campbell wrote:
> On Thu, 2013-06-13 at 15:00 +0100, Lars Kurth wrote:
> > == Stefano's Proposal ==
> > We should aim to reduce the release cycle to 4 months (or maybe 6 months 
> > as an intermediate step) from the current 9 months. A 4 months relase 
> > cycle should help accelerate development and lead to fewer patches being 
> > queued up. The implications are that we would have to operate a 2-3 
> > weeks merge window.
> > 
> > To do this, we would need to resolve a number of issues
> > * It is likely that code reviews for such a short merge window would 
> > become a bottleneck. We are not sure whether this would be a major issue 
> > : the review bandwith would depend on the patches submitted (and their 
> > complexity)
> 
> If we do end up going for this model then I think we need to also
> inherit the principal from Linux that the merge window is for *merging*
> and that the code in question should have already been posted and
> reviewed *before* the window opens (i.e. in the other 3 months of the
> cycle). 

Which would imply that if we use this for Xen 4.4, we would have 
almost no patches ready :-)

That would be a painless release - just fixing the carryover bugs
from Xen 4.3.

> 
> This would be something of a cultural change because at the minute
> people seem to assume that a freeze means that development and review
> must stop. Personally I don't think that should be the case anyway,
> although I appreciate the need to focus peoples attention on the release
> as well.
> 
> This could end up with subsystem maintainers queueing patches into their
> own tree for merge into mainline during the window (this is what Linux
> does). This has its own downsides WRT the merging work during window and
> there would need to be a change since we would also have to ask people
> to test those trees, which they aren't used to doing.

That in the Linux ecosystem was solved with the linux-next which would
take all of the subsystems's tree.

> 
> > * [I can't remember who raised this] The concern was raised that we 
> > would not have enough x86 Xen reviewers got a 2-3 weeks merge window
> > * [Konrad] Stated that in PVOPS for Linux contributions we don't have a 
> > review bottleneck,
> 
> Largely due to Linux having the properties I described above, IMHO.
> Although I'm not a Linux maintainer of a large subsystem so what would I
> know ;-)

This is also b/c of the down-time - those three months can be used
to thrash out bugs and be pedantic about peoples patches. A nice Linux release
is when at rc1 there are no regressions that I know of and I can concentrate
on developing a feature or clear some of my backlog.

But that would imply that the maintainer (release maintainer?) of Xen would 
now be primarily responsible for chasing down folks to fix bugs.

And if one does not have bugs, then there are nice three months to develop
that killer feature.

But that does not solve the problem of various bugs that exists or
are detected by users and chasing those down. They unfortunatly take
more time to troubleshoot and figure out than I personally would like.

Perhaps the way to think about this is what we need to make sure
works 100% between releases. There are a lot of pieces in it.

I appreciate very much that there are folks who are willing to test it out
during that time, report and bear with us trying to figure what got
broken.

But it would also help to have a regression test. I am very excited
about the 'extensive testing framework', but it is not ready yet.

The osstest is pretty awesome too - is there other things we can add
in to the testing framework to be able to do more of the regression tests?
More OS-es?

Re: [Hackathon Minutes] Xen 4.4 Planning

[George Dunlap]

On 13/06/13 15:00, Lars Kurth wrote:
> This took me a while to post, but given that we are not starting 4.4 
> just yet, this may be appropriate now. I may have misrepresented some 
> stuff as it has been 4 weeks since I wrote these.
> Cheers
> Lars
>
> = Purpose of Roadmap =
> * Set a vision for interesting features
> * Track items
> * Help consumers of Xen with their planning
>
> = Release Models that work well =
> There was a brief discussion on two different release models
> * Train leaves the station (Linux)
> * Release when ready (Debian)
>
> == Stefano's Proposal ==
> We should aim to reduce the release cycle to 4 months (or maybe 6 
> months as an intermediate step) from the current 9 months. A 4 months 
> relase cycle should help accelerate development and lead to fewer 
> patches being queued up. The implications are that we would have to 
> operate a 2-3 weeks merge window.
>
> To do this, we would need to resolve a number of issues
> * It is likely that code reviews for such a short merge window would 
> become a bottleneck. We are not sure whether this would be a major 
> issue : the review bandwith would depend on the patches submitted (and 
> their complexity)
> * [I can't remember who raised this] The concern was raised that we 
> would not have enough x86 Xen reviewers got a 2-3 weeks merge window
> * [Konrad] Stated that in PVOPS for Linux contributions we don't have 
> a review bottleneck, but we should make sure that the Xen and 
> PVOPS/Linux merge window don't overlap (as this would require the same 
> set of people to review patches in two projects)
> * The rest of the time (approx 3 months) would be used for stabilizing 
> the release

Why would we need 3 months to do testing for 2 weeks of merging (or 4 
months of development, depending on how you look at it), when 6 weeks 
has been just about right for 9 months of "merging"?

I think it takes that long for Linux because it's such a gigantic 
project.  I would think 3 months development / 1 month stabilization 
would be OK for Xen.

I would rather avoid the "merge window" workflow until it becomes really 
necessary, as it adds all kinds of other issues (e.g., needing something 
like linux-next to detect and sort out merge conflicts ahead of the 
merge window).

  -George

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ben Guthro]

On Thu, Jun 13, 2013 at 10:00 AM, Lars Kurth <lars.kurth@xen.org> wrote:
> This took me a while to post, but given that we are not starting 4.4 just
> yet, this may be appropriate now. I may have misrepresented some stuff as it
> has been 4 weeks since I wrote these.
> Cheers
> Lars
>
> = Purpose of Roadmap =
> * Set a vision for interesting features
> * Track items
> * Help consumers of Xen with their planning
>
> = Release Models that work well =
> There was a brief discussion on two different release models
> * Train leaves the station (Linux)
> * Release when ready (Debian)
>
> == Stefano's Proposal ==
> We should aim to reduce the release cycle to 4 months (or maybe 6 months as
> an intermediate step) from the current 9 months. A 4 months relase cycle
> should help accelerate development and lead to fewer patches being queued
> up. The implications are that we would have to operate a 2-3 weeks merge
> window.
>
> To do this, we would need to resolve a number of issues
> * It is likely that code reviews for such a short merge window would become
> a bottleneck. We are not sure whether this would be a major issue : the
> review bandwith would depend on the patches submitted (and their complexity)
> * [I can't remember who raised this] The concern was raised that we would
> not have enough x86 Xen reviewers got a 2-3 weeks merge window
> * [Konrad] Stated that in PVOPS for Linux contributions we don't have a
> review bottleneck, but we should make sure that the Xen and PVOPS/Linux
> merge window don't overlap (as this would require the same set of people to
> review patches in two projects)
> * The rest of the time (approx 3 months) would be used for stabilizing the
> release
> * If we had a more extensive testing framework, and thus better testing, we
> could tighten the RC period (making this happen is also being discussed by
> the Advisory Board)
>
> Additional conerns raised:
> * [Matt Wilson]: if we had shorter merge windows, there is a risk that we
> would end up with unnused code (uncompleted features) in mainline. Something
> which we'd rather not have
> * [I can't remember who raised this] But we already have a buffer via
> staging : we could make more use of this
>
> [Conclusion] We aLL agreed, that a release cycle of less than 9 months is
> desirable. Maybe we can go to 6 months for Xen 4.4 (before being more
> aggressive).
>
> = 4.3 Release cycle : what worked well / didn't work well =
> * The 4.3 release updates and criteria went well
> * BUT :  50% of what was supposed to be in 4.3 didn't make it
> ** In some cases, we simply underestimated the effort that is needed.
> Concrete example : QEMU/stubdomain was a combination of under-estimating the
> size and over-estimating the development bandwidth that was available
> ** Some of the high-impact features (e.g. PVH) came in too late in the dev
> cycle. Mitigation : break contributions into smaller parts and submit
> earlier in the merge window. The same applies to changes to generic code.
> * BUT : Some patches were lost (i.e. when there are spikes of activity it
> becomes hard for some maintainers/committers to keep on top of their queue).
> ** [Ian Campell] said that we should rely on submitter to resend the patch:
> the assumption is that if the patch is not important, the submitter will
> badger and resend.
> ** [Lars] raised the point that this can alienate contributors and get them
> to look at other projects instead.
> ** [Can't remember who said this] Maybe use patchwork
> (http://jk.ozlabs.org/projects/patchwork/) to track patches
> ** [Ian Campbell]: patchwork looks like a good idea, but may not work well
> in practice
> [Note] We should probably have a discussion or some sort of trial. It may
> also be possible to use the http://bugs.xenproject.org prototype (or add a
> "deferred patch" attribute)
> [Note] We typically start opening the dev branch at RC5/6 (not sure I quite
> got this)
> [Note] We don't actually have a list of patches that got lost in the 4.3
> release cycle )-:
>
> [ACTION]: George write up a proposal for the beginning of the 4.4 release
> cycle
>
> = 4.4 Content =
> George volunteers to be the Release co-ordinator for Xen 4.4 (to apply what
> he learned)
>
> * Big features that did not make it in 4.3
>
> * PVH? What will make it into 4.4
>
> * Missed patches (don't have a list)
>
> * "User" features that look interesting
>
> ** Network effects
>
> ** We shouldn't have broken feature
>
> ** What about XenClient / VirtualComputer being able to help out in adding
> mopre support for PCI / VGA cards
>

Did you have any more info on this bullet, and the other one below Re:
XenClient / VirtualComputer?

FWIW, the XenClient Enterprise product (what was VirtualComputer
NxTop) doesn't currently do PCI / VGA passthrough, in an attempt to
get a larger HCL.
The XenClient XT product (developed primarily in Citrix Cambridge)
does rely on this functionality, however.

It is a subtle (but sizable) architectural difference, that made each
team come to some different conclusions when solving similar problems.
As such, this is an area that the two teams have remained separate,
despite sharing a XenClient moniker.

There may be some areas that we may be able to help out, but we'd need
to enumerate the issues, and work with management to get it properly
staffed - naturally balancing against the product delivery
requirements.


> ** Other features: Sharing dom0 keyboard / mouse
>
>
> * GPU passthrough
> ** We have an issue with graphics card support
> ** A lot of users care about GPU passthrough (but not many vendors)
> ** Maybe we could mentor somebody about GPU passthrough?
>
> ** Maybe we could get XenClient, VirtualComputer or Qubes to pick this up
> (or partly do so)?
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [Hackathon Minutes] Xen 4.4 Planning

[Pasi Kärkkäinen]

On Thu, Jun 13, 2013 at 01:09:46PM -0400, Ben Guthro wrote:
> >
> > ** What about XenClient / VirtualComputer being able to help out in adding
> > mopre support for PCI / VGA cards
> >
> 
> Did you have any more info on this bullet, and the other one below Re:
> XenClient / VirtualComputer?
> 
> FWIW, the XenClient Enterprise product (what was VirtualComputer
> NxTop) doesn't currently do PCI / VGA passthrough, in an attempt to
> get a larger HCL.
> The XenClient XT product (developed primarily in Citrix Cambridge)
> does rely on this functionality, however.
> 
> It is a subtle (but sizable) architectural difference, that made each
> team come to some different conclusions when solving similar problems.
> As such, this is an area that the two teams have remained separate,
> despite sharing a XenClient moniker.
> 
> There may be some areas that we may be able to help out, but we'd need
> to enumerate the issues, and work with management to get it properly
> staffed - naturally balancing against the product delivery
> requirements.
> 

I'm not sure what the "more support for PCI / VGA cards" above meant,
but Xen would at least need someone to be the "GPU passthrough" maintainer,
there are many unmerged patches laying around on the mailinglists,
for all the three major GPU vendors: Intel, Nvidia and AMD.


> 
> > ** Other features: Sharing dom0 keyboard / mouse
>

This should be a pretty small task actually, we did some planning and 
analysis of various options available at last year's XenSummit 
with Gianluca and Stefano.

> >
> >
> > * GPU passthrough
> > ** We have an issue with graphics card support
> > ** A lot of users care about GPU passthrough (but not many vendors)
> > ** Maybe we could mentor somebody about GPU passthrough?
> >
> > ** Maybe we could get XenClient, VirtualComputer or Qubes to pick this up
> > (or partly do so)?
> >
> >

GPU passthrough is an important feature, and currently it's not well 
supported out-of-the-box with Xen. It is being discussed very often 
on xen related mailinglists and irc channels, so it seems to be "hot" feature for users.

-- Pasi

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

--On 13 June 2013 15:00:46 +0100 Lars Kurth <lars.kurth@xen.org> wrote:

> We should aim to reduce the release cycle to 4 months (or maybe 6 months
> as an intermediate step) from the current 9 months. A 4 months relase
> cycle should help accelerate development and lead to fewer patches being
> queued up. The implications are that we would have to operate a 2-3 weeks
> merge window.

Disclaimer: I don't claim to be a xen developer, despite writing a few
patches. But we are a heavy libxl API user. This perspective may or may
not be useful from a 'consumer of your development' perspective. Delete
or ignore if not.

The thing we like the least about Xen is (was?), stuff breaking between
major releases. If you have to drop 90% of your new features in order
not to break stuff, please do just that. Xen 3.x -> Xen 4.1 (we mostly
skipped 4.0) was a huge change, requiring a lot of rewriting. Xen 4.1
to Xen 4.2 was far, far worse; had we not needed it to support
qemu-upstream, we'd have probably stuck with 4.1. We talk to 4 hypervisors,
and have never had this difficulty with any other hypervisor. You can
imagine our joy when we found we could compile against 4.3 (we haven't
tried running yet) without a single line code changed. Please, please,
keep it like this. If we can also run unchanged against 4.3, this will
be even better.

The thing we like the second least about Xen is how long it seems to take
to get what we count as serious bugs fixed, even in stable releases. We
like it even less if we have to find them and fix them. By 'serious'
I mean basic functionality not working, crashes dom0, etc. I don't know
if this says something bad about us, but we've not found any such bug
ever in kvm (in well over 5 years). On the positive side, we've found
xen-devel pretty friendly and receptive. The perception is, however, that
new development takes priority over stable releases. I recognise that I
have a bias here, so this may not be fair.

The result of this is two-fold. Firstly, we've never (yet) been able to
run a production version of xen which is a standard xen release. We've
always had to maintain our own patches even on 'stable' releases.
Frankly, this is a pain. Secondly, there is a perception that moving
versions of Xen is going to be a huge PITA; that perception does not
exist for other hypervisors. I'm really really hoping Xen 4.3 dispels
this, and signs are good so far (we've done a lot of testing at xl
level, not so much at libxl level).

What does this mean for the development process?

1. I think more testing would be useful, particularly against API driven
   stuff. I find the current test stuff a bit confusing. What I wan't
   to know is whether commit X works or not - if things fail randomly,
   they aren't useful tests, particularly if it's difficult to
   distinguish them from other failures. Spoken as someone who's broken
   things :-(

2. My concern about early branching of -next (at -rc1 for instance) is
   that developing new stuff is far more interesting than fixing bugs.
   We liked the way stuff we raised with 4.3 (e.g. tsc issues) got looked
   at, and would hope that continues.

3. I would quite like a slightly shorter development cycle IFF it
   doesn't impact on stability. EG I thought we were probably bending the
   rules to get live migrate on qemu-upstream backported into 4.2,
   and had 4.3 been available sooner, we wouldn't have pushed. At
   a guess, 6 months would be about right, 4 months would be too short.

4. Following xen has been 10 times easier since you've moved to git.
   I agree with George's statement that you aren't using it enough -
   particularly branches.

5. At risk of repetition, we don't really care, so long as you don't break
   stuff.

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Murray]

On 13/06/13 22:03, Alex Bligh wrote:
>
>
> --On 13 June 2013 15:00:46 +0100 Lars Kurth <lars.kurth@xen.org> wrote:
>
>> We should aim to reduce the release cycle to 4 months (or maybe 6 months
>> as an intermediate step) from the current 9 months. A 4 months relase
>> cycle should help accelerate development and lead to fewer patches being
>> queued up. The implications are that we would have to operate a 2-3 
>> weeks
>> merge window.
>
> Disclaimer: I don't claim to be a xen developer, despite writing a few
> patches. But we are a heavy libxl API user. This perspective may or may
> not be useful from a 'consumer of your development' perspective. Delete
> or ignore if not.
>
> The thing we like the least about Xen is (was?), stuff breaking between
> major releases. If you have to drop 90% of your new features in order
> not to break stuff, please do just that. Xen 3.x -> Xen 4.1 (we mostly
> skipped 4.0) was a huge change, requiring a lot of rewriting. Xen 4.1
> to Xen 4.2 was far, far worse; had we not needed it to support
> qemu-upstream, we'd have probably stuck with 4.1. We talk to 4 
> hypervisors,
> and have never had this difficulty with any other hypervisor. You can
> imagine our joy when we found we could compile against 4.3 (we haven't
> tried running yet) without a single line code changed. Please, please,
> keep it like this. If we can also run unchanged against 4.3, this will
> be even better.
>
When you talk about "stuff" breaking between major releases, are you 
talking about Xen code not functioning or your code failing because of 
changes in Xen? If the latter, are we talking designed changes in Xen's 
behaviour or non-designed ones (=bugs)?


> The thing we like the second least about Xen is how long it seems to take
> to get what we count as serious bugs fixed, even in stable releases. We
> like it even less if we have to find them and fix them. By 'serious'
> I mean basic functionality not working, crashes dom0, etc. I don't know
> if this says something bad about us, but we've not found any such bug
> ever in kvm (in well over 5 years). On the positive side, we've found
> xen-devel pretty friendly and receptive. The perception is, however, that
> new development takes priority over stable releases. I recognise that I
> have a bias here, so this may not be fair.

I am at a loss as to what is wrong with contributing a few bug fixes 
back if you're technically capable of finding and fixing... I am not 
feeling the community spirit here.

>
> The result of this is two-fold. Firstly, we've never (yet) been able to
> run a production version of xen which is a standard xen release. We've
> always had to maintain our own patches even on 'stable' releases.
> Frankly, this is a pain. Secondly, there is a perception that moving
> versions of Xen is going to be a huge PITA; that perception does not
> exist for other hypervisors. I'm really really hoping Xen 4.3 dispels
> this, and signs are good so far (we've done a lot of testing at xl
> level, not so much at libxl level).

I thought Xen was a "project" not a "product".

Also, a little bit of Googling tells me that Flexiant (that is the "we" 
in all of this, right?) provides cloud software to third-parties and 
does not provide cloud services itself. To make your product work 
reliably with Xen for your customers, are you distributing your own 
patches for Xen to these third-parties?



>
> What does this mean for the development process?
>
> 1. I think more testing would be useful, particularly against API driven
>   stuff. I find the current test stuff a bit confusing. What I wan't
>   to know is whether commit X works or not - if things fail randomly,
>   they aren't useful tests, particularly if it's difficult to
>   distinguish them from other failures. Spoken as someone who's broken
>   things :-(
>
> 2. My concern about early branching of -next (at -rc1 for instance) is
>   that developing new stuff is far more interesting than fixing bugs.
>   We liked the way stuff we raised with 4.3 (e.g. tsc issues) got looked
>   at, and would hope that continues.
>
> 3. I would quite like a slightly shorter development cycle IFF it
>   doesn't impact on stability. EG I thought we were probably bending the
>   rules to get live migrate on qemu-upstream backported into 4.2,
>   and had 4.3 been available sooner, we wouldn't have pushed. At
>   a guess, 6 months would be about right, 4 months would be too short.
>
> 4. Following xen has been 10 times easier since you've moved to git.
>   I agree with George's statement that you aren't using it enough -
>   particularly branches.
>
> 5. At risk of repetition, we don't really care, so long as you don't 
> break
>   stuff.
>

Again, what "stuff"? IMHO woolly language isn't wonderfully helpful in 
most cases, especially on a technical list.

Thanks for reading.

Disclosure: An interested individual who was delighted to recently have 
had a tincy-wincy patch accepted into unstable, fixing a bug that has 
annoyed me for donkey's.

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

Ian,

--On 14 June 2013 00:56:14 +0100 Ian Murray <murrayie@yahoo.co.uk> wrote:

> When you talk about "stuff" breaking between major releases, are you
> talking about Xen code not functioning or your code failing because of
> changes in Xen? If the latter, are we talking designed changes in Xen's
> behaviour or non-designed ones (=bugs)?

Both.

As an example of the first, the API changed very significantly between
3.x and 4.1, and 4.1 and 4.2. Some of the API changes were subtle (e.g.
what you had to do across a fork()).

As an example of the second, see the very long thread with O_DIRECT
in the subject line.

> I am at a loss as to what is wrong with contributing a few bug fixes back
> if you're technically capable of finding and fixing... I am not feeling
> the community spirit here.

There's nothing wrong with that, and we have done. However this thread
is about development cycles. Critical stuff breaks, we know that.
That should happen in the unstable version. Patches that fix critical
stuff should be made in the unstable version, not the stable version.

> Also, a little bit of Googling tells me that Flexiant (that is the "we"
> in all of this, right?) provides cloud software to third-parties and does
> not provide cloud services itself. To make your product work reliably
> with Xen for your customers, are you distributing your own patches for
> Xen to these third-parties?

Yes. We ship with a patched version of xen. All our patches have been sent
here and are on github. If the implication is that we're somehow keeping
these to ourselves, you have that totally wrong. We would like nothing
better than all our patches to be in mainline. For 4.2 we are currently
carrying one patch for fixing the tsc (see 'clock stalled on live
migration') which is a backport of a 4.3 patch here, and my minideb
patch which is a packaging change which (for quite understandable
reasons) people don't want in mainline. In the past, we've carried
tens of patches - see the live migrate on qemu-upstream-dm for 4.2
patch series I posted here, since accepted.

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Murray]

----- Original Message -----
> From: Alex Bligh <alex@alex.org.uk>
> To: Ian Murray <murrayie@yahoo.co.uk>; xen-devel@lists.xen.org
> Cc: Alex Bligh <alex@alex.org.uk>
> Sent: Friday, 14 June 2013, 8:01
> Subject: Re: [Xen-devel] [Hackathon Minutes] Xen 4.4 Planning
> 
> Ian,
> 
> --On 14 June 2013 00:56:14 +0100 Ian Murray <murrayie@yahoo.co.uk> wrote:
> 
>>  When you talk about "stuff" breaking between major releases, are 
> you
>>  talking about Xen code not functioning or your code failing because of
>>  changes in Xen? If the latter, are we talking designed changes in Xen's
>>  behaviour or non-designed ones (=bugs)?
> 
> Both.
> 
> As an example of the first, the API changed very significantly between
> 3.x and 4.1, and 4.1 and 4.2. Some of the API changes were subtle (e.g.
> what you had to do across a fork()).

But surely, this is your business. This is what you do. Sounds like you want API changes to be held back for the entire community so your company has to do less work.


> 
> As an example of the second, see the very long thread with O_DIRECT
> in the subject line.
> 
>>  I am at a loss as to what is wrong with contributing a few bug fixes back
>>  if you're technically capable of finding and fixing... I am not feeling
>>  the community spirit here.
> 
> There's nothing wrong with that, and we have done. However this thread
> is about development cycles. Critical stuff breaks, we know that.
> That should happen in the unstable version. Patches that fix critical
> stuff should be made in the unstable version, not the stable version.

Do *you* test unstable and point out the critical bugs BEFORE it goes stable? If you don't, then don't be surprised when bugs crop up... 


> 
>>  Also, a little bit of Googling tells me that Flexiant (that is the 
> "we"
>>  in all of this, right?) provides cloud software to third-parties and does
>>  not provide cloud services itself. To make your product work reliably
>>  with Xen for your customers, are you distributing your own patches for
>>  Xen to these third-parties?
> 
> Yes. We ship with a patched version of xen. All our patches have been sent
> here and are on github. If the implication is that we're somehow keeping
> these to ourselves, you have that totally wrong. We would like nothing
> better than all our patches to be in mainline. For 4.2 we are currently
> carrying one patch for fixing the tsc (see 'clock stalled on live
> migration') which is a backport of a 4.3 patch here, and my minideb
> patch which is a packaging change which (for quite understandable
> reasons) people don't want in mainline. In the past, we've carried
> tens of patches - see the live migrate on qemu-upstream-dm for 4.2
> patch series I posted here, since accepted.

I looked on your website and couldn't find any download section for patches or a pointer to github or anything. I am not a lawyer but I didn't get the impression that sticking patches on a mailing list was fulfilling ones obligation under the appropriate licence. So please can you link from your company website to your github were these patches are available.

One aspect from your previous email I'd like to pickup on, when you mentioned KVM not having these issues, are you compiling from the source or using a distribution version? If the latter, then I think that is an unfair comparison.


> 
> -- 
> Alex Bligh
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

Ian,

--On 14 June 2013 10:46:31 +0100 Ian Murray <murrayie@yahoo.co.uk> wrote:

>> As an example of the first, the API changed very significantly between
>> 3.x and 4.1, and 4.1 and 4.2. Some of the API changes were subtle (e.g.
>> what you had to do across a fork()).
>
> But surely, this is your business. This is what you do. Sounds like you
> want API changes to be held back for the entire community so your company
> has to do less work.

I think forward compatible APIs are a good thing. This is not
controversial. I believe there are few people in the Xen community
who think not having forward compatible APIs was anything but a mistake.

>> There's nothing wrong with that, and we have done. However this thread
>> is about development cycles. Critical stuff breaks, we know that.
>> That should happen in the unstable version. Patches that fix critical
>> stuff should be made in the unstable version, not the stable version.
>
> Do *you* test unstable and point out the critical bugs BEFORE it goes
> stable? If you don't, then don't be surprised when bugs crop up...

Yes we do. If you want a concrete example, go search for the migration
wallclock lockup thread this month.

> I looked on your website and couldn't find any download section for
> patches or a pointer to github or anything. I am not a lawyer but I
> didn't get the impression that sticking patches on a mailing list was
> fulfilling ones obligation under the appropriate licence. So please can
> you link from your company website to your github were these patches are
> available.

The fact it is not immediately obvious to you on a marketing web site
does not mean it isn't there. You'll find it in our repo too (though
not as a git repository which is actually what people want).

Saying that, I have no objection at all to making this more obvious,
and will do so. We're proud of what we do with open source.

> One aspect from your previous email I'd like to pickup on, when you
> mentioned KVM not having these issues, are you compiling from the source
> or using a distribution version? If the latter, then I think that is an
> unfair comparison.

We use a distribution version (we run Ubuntu throughout). I'm not
quite sure why it's an unfair comparison. If there was an Ubuntu version
of 4.2.x for an LTS release, we'd be using that.

Are you implying that the STABLE Xen version on xenbits should not be
expected to be as reliable as the same version packaged by a maintainer?
If so, why?

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Murray]

> 
> I think forward compatible APIs are a good thing. This is not
> controversial. I believe there are few people in the Xen community
> who think not having forward compatible APIs was anything but a mistake.
>

It certainly is a good thing, but if there is a need to change them then a major release is a good place to do it. Surely this is a governance issue, not a release cycle issue?

I have no idea whether the API changes you descrbe were a "mistake" or not as I don't use them.


 
>>>  There's nothing wrong with that, and we have done. However this 
> thread
>>>  is about development cycles. Critical stuff breaks, we know that.
>>>  That should happen in the unstable version. Patches that fix critical
>>>  stuff should be made in the unstable version, not the stable version.
>> 
>>  Do *you* test unstable and point out the critical bugs BEFORE it goes
>>  stable? If you don't, then don't be surprised when bugs crop up...
> 
> Yes we do. If you want a concrete example, go search for the migration
> wallclock lockup thread this month.
> 
>>  I looked on your website and couldn't find any download section for
>>  patches or a pointer to github or anything. I am not a lawyer but I
>>  didn't get the impression that sticking patches on a mailing list was
>>  fulfilling ones obligation under the appropriate licence. So please can
>>  you link from your company website to your github were these patches are
>>  available.
> 
> The fact it is not immediately obvious to you on a marketing web site
> does not mean it isn't there. You'll find it in our repo too (though
> not as a git repository which is actually what people want).
> 
> Saying that, I have no objection at all to making this more obvious,
> and will do so. We're proud of what we do with open source.
> 

Well, I spent more time than "immediate" to try to find them and if it's not obvious where they are or they aren't linked to, then they might as well not be there tbh.

so where is this publically available repo?


>>  One aspect from your previous email I'd like to pickup on, when you
>>  mentioned KVM not having these issues, are you compiling from the source
>>  or using a distribution version? If the latter, then I think that is an
>>  unfair comparison.
> 
> We use a distribution version (we run Ubuntu throughout). I'm not
> quite sure why it's an unfair comparison. If there was an Ubuntu version
> of 4.2.x for an LTS release, we'd be using that.
> 
> Are you implying that the STABLE Xen version on xenbits should not be
> expected to be as reliable as the same version packaged by a maintainer?
> If so, why?

I seem to remember that a (perhaps old) Xen mantra was that it is a project and not a product. Ubuntu is a product, Xen is a project. RedHat 5 is a product and I seem to remember Xen being stable for my limited purposes. Your experience may have differed.


> 
> -- 
> Alex Bligh
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

Ian,

--On 14 June 2013 13:32:36 +0100 Ian Murray <murrayie@yahoo.co.uk> wrote:

> I have no idea whether the API changes you descrbe were a "mistake" or
> not as I don't use them.

I'm not saying the API changes were a mistake. I'm saying that it was
a mistake that before 4.2 there was not a forward compatible API.
Clearly there had to be some pain to fix that.

> Well, I spent more time than "immediate" to try to find them and if it's
> not obvious where they are or they aren't linked to, then they might as
> well not be there tbh.
>
> so where is this publically available repo?

repo.flexiant.com/ ..., which is installed by default for anyone using
our binaries.

You'll obviously have found:
 http://docs.flexiant.com/display/DOCS/Open+Source+Software

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Murray]

>> 
>>  so where is this publically available repo?
> 
> repo.flexiant.com/ ..., which is installed by default for anyone using
> our binaries.
> 
> You'll obviously have found:
> http://docs.flexiant.com/display/DOCS/Open+Source+Software

Nope. Still can't find a link from the main site.

Besides, I am not sure if providing source via a repo who's address isn't publically published is really complying with your resposibilities, is it? Again, not a lawyer, just IMHO.


> 
> -- 
> Alex Bligh
> 

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Campbell]

On Fri, 2013-06-14 at 14:34 +0100, Ian Murray wrote:
> 
> >> 
> >>  so where is this publically available repo?
> > 
> > repo.flexiant.com/ ..., which is installed by default for anyone using
> > our binaries.
> > 
> > You'll obviously have found:
> > http://docs.flexiant.com/display/DOCS/Open+Source+Software
> 
> Nope. Still can't find a link from the main site.
> 
> Besides, I am not sure if providing source via a repo who's address
> isn't publically published is really complying with your
> resposibilities, is it?

Assuming you are referring to the responsibilities of the GPL you should
probably read the license (or at least do some basic investigation of
the terms) to see what it actually requires before making these sorts of
accusations.

There are options for applying the GPL which don't require anything like
what you seem to expect. There's certainly no requirement for a link on
ones website when one uses GPL software.

The text of the GPL is freely available. I strongly suggest you go read
it, it's not all that long nor is it particularly full of legalese.

Accusing someone of a license infringement is rather a serious matter
and you really should make sure you know what you are talking about
before doing so.

I think you are totally out of order hounding Alex like this.

>  Again, not a lawyer, just IMHO.

Lawyer or otherwise I think you would be well advised to keep your
opinions to yourself until you've actually read the license in question.

Ian.

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Murray]

----- Original Message -----
> From: Ian Campbell <Ian.Campbell@citrix.com>
> To: Ian Murray <murrayie@yahoo.co.uk>
> Cc: Alex Bligh <alex@alex.org.uk>; "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
> Sent: Friday, 14 June 2013, 14:55
> Subject: Re: [Xen-devel] [Hackathon Minutes] Xen 4.4 Planning
> 
> On Fri, 2013-06-14 at 14:34 +0100, Ian Murray wrote:
>> 
>>  >> 
>>  >>  so where is this publically available repo?
>>  > 
>>  > repo.flexiant.com/ ..., which is installed by default for anyone using
>>  > our binaries.
>>  > 
>>  > You'll obviously have found:
>>  > http://docs.flexiant.com/display/DOCS/Open+Source+Software
>> 
>>  Nope. Still can't find a link from the main site.
>> 
>>  Besides, I am not sure if providing source via a repo who's address
>>  isn't publically published is really complying with your
>>  resposibilities, is it?
> 
> Assuming you are referring to the responsibilities of the GPL you should
> probably read the license (or at least do some basic investigation of
> the terms) to see what it actually requires before making these sorts of
> accusations.
> 
> There are options for applying the GPL which don't require anything like
> what you seem to expect. There's certainly no requirement for a link on
> ones website when one uses GPL software.
> 
> The text of the GPL is freely available. I strongly suggest you go read
> it, it's not all that long nor is it particularly full of legalese.
> 
> Accusing someone of a license infringement is rather a serious matter
> and you really should make sure you know what you are talking about
> before doing so.
> 
> I think you are totally out of order hounding Alex like this.
> 
>>   Again, not a lawyer, just IMHO.
> 
> Lawyer or otherwise I think you would be well advised to keep your
> opinions to yourself until you've actually read the license in question.
> 
> Ian.
> 

My understanding is that source should be made available in a reasonable format to any 3rd party if you distribute modified code that is licenced to you via the GPL. Do I have that wrong?

I was curious to see his patches to see what he was referring to and I seem to have had to go through hoops a bit.

Not accusing anybody of anything... I merely asked the question. No meaning to hounding. Just trying to get an understanding. I am sure Alex has had to deal with worse than me in his time! :)

Re: [Hackathon Minutes] Xen 4.4 Planning

[Gordan Bobic]

 On Fri, 14 Jun 2013 15:44:50 +0100 (BST), Ian Murray 
 <murrayie@yahoo.co.uk> wrote:
> ----- Original Message -----
>> From: Ian Campbell <Ian.Campbell@citrix.com>
>> To: Ian Murray <murrayie@yahoo.co.uk>
>> Cc: Alex Bligh <alex@alex.org.uk>; "xen-devel@lists.xen.org" 
>> <xen-devel@lists.xen.org>
>> Sent: Friday, 14 June 2013, 14:55
>> Subject: Re: [Xen-devel] [Hackathon Minutes] Xen 4.4 Planning
>>
>> On Fri, 2013-06-14 at 14:34 +0100, Ian Murray wrote:
>>>
>>>  >>
>>>  >>  so where is this publically available repo?
>>>  >
>>>  > repo.flexiant.com/ ..., which is installed by default for anyone 
>>> using
>>>  > our binaries.
>>>  >
>>>  > You'll obviously have found:
>>>  > http://docs.flexiant.com/display/DOCS/Open+Source+Software
>>>
>>>  Nope. Still can't find a link from the main site.
>>>
>>>  Besides, I am not sure if providing source via a repo who's 
>>> address
>>>  isn't publically published is really complying with your
>>>  resposibilities, is it?
>>
>> Assuming you are referring to the responsibilities of the GPL you 
>> should
>> probably read the license (or at least do some basic investigation 
>> of
>> the terms) to see what it actually requires before making these 
>> sorts of
>> accusations.
>>
>> There are options for applying the GPL which don't require anything 
>> like
>> what you seem to expect. There's certainly no requirement for a link 
>> on
>> ones website when one uses GPL software.
>>
>> The text of the GPL is freely available. I strongly suggest you go 
>> read
>> it, it's not all that long nor is it particularly full of legalese.
>>
>> Accusing someone of a license infringement is rather a serious 
>> matter
>> and you really should make sure you know what you are talking about
>> before doing so.
>>
>> I think you are totally out of order hounding Alex like this.
>>
>>>   Again, not a lawyer, just IMHO.
>>
>> Lawyer or otherwise I think you would be well advised to keep your
>> opinions to yourself until you've actually read the license in 
>> question.
>>
>> Ian.
>>
>
> My understanding is that source should be made available in a
> reasonable format to any 3rd party if you distribute modified code
> that is licenced to you via the GPL. Do I have that wrong?

 IANAL, but:

 Reasonable format can mean anything. For example the kernel source
 for the Toshiba AC100 (ARM based Android laptop) was provided by
 Toshiba upon request on a CD by snail mail when those of us playing
 with the device requested it.

 If nobody is requesting the source, it arguably doesn't matter.
 But if somebody is asking for it, it must be provided. That's
 about the size of it as far as I can work out.

 Gordan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Hackathon Minutes] Xen 4.4 Planning

[George Dunlap]

On Fri, Jun 14, 2013 at 3:55 PM, Gordan Bobic <gordan@bobich.net> wrote:
> Reasonable format can mean anything. For example the kernel source
> for the Toshiba AC100 (ARM based Android laptop) was provided by
> Toshiba upon request on a CD by snail mail when those of us playing
> with the device requested it.

In fact, when the GPL first came out, the internet was only available
to a select few; it might have been considered an *unreasonble*
format.  Typically how you got the source code was writing a
snail-mail letter, and being shipped a stack of floppies.  And the
requester had to pay for the floppies, the postage, and a reasonable
handling fee.

 -George

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Campbell]

On Fri, 2013-06-14 at 15:44 +0100, Ian Murray wrote:
> My understanding is that source should be made available in a
> reasonable format to any 3rd party if you distribute modified code
> that is licenced to you via the GPL. Do I have that wrong?

That is (roughly, although not exactly) one of the options for complying
with the GPL, but it is not the only one. So, yes, you have it wrong.

Again, the text of the license is freely available, so you could easily
have checked what it says for yourself.

Ian.

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

--On 14 June 2013 14:55:07 +0100 Ian Campbell <Ian.Campbell@citrix.com> 
wrote:

> I think you are totally out of order hounding Alex like this.

Despite which, and despite there being no legal obligation to do so,
we actually like publishing the open source stuff we do, and the
fact it's not fantastically easy to find is well taken. I'm trying
to find out who (hopefully someone internal) registered
github.com/flexiant - if not I'll use something else - and we will
be putting all our open source stuff up there, and have a link from
the open source page in the manual and the open source page on
the website. This plus at least one of those links should be up
in the next few days.

Note this is because we want to publish stuff, not because we
think there is a compliance issue. Ian M: had you said "please
could you publish this more obviously" you would have got the
same response.

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Ian Murray]

On 14/06/13 16:43, Alex Bligh wrote:
>
>
> --On 14 June 2013 14:55:07 +0100 Ian Campbell 
> <Ian.Campbell@citrix.com> wrote:
>
>> I think you are totally out of order hounding Alex like this.
>
> Despite which, and despite there being no legal obligation to do so,
> we actually like publishing the open source stuff we do, and the
> fact it's not fantastically easy to find is well taken. I'm trying
> to find out who (hopefully someone internal) registered
> github.com/flexiant - if not I'll use something else - and we will
> be putting all our open source stuff up there, and have a link from
> the open source page in the manual and the open source page on
> the website. This plus at least one of those links should be up
> in the next few days.
>
> Note this is because we want to publish stuff, not because we
> think there is a compliance issue. Ian M: had you said "please
> could you publish this more obviously" you would have got the
> same response.
>

Thanks very much for that. As a casual browser, it will be a lot easier 
than the repo. Absolutely no rush.

After reflecting on the conversation earlier, I realise it was 
ill-conceived to take it in the direction I did, so please accept my 
apologies for any distress that may have caused. Please also accept my 
apologies if you felt "hounded" (granted, not your word). This wasn't my 
intention.

Thanks again and have a good weekend.

Ian (M).

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

--On 14 June 2013 16:43:41 +0100 Alex Bligh <alex@alex.org.uk> wrote:

> Despite which, and despite there being no legal obligation to do so,
> we actually like publishing the open source stuff we do, and the
> fact it's not fantastically easy to find is well taken. I'm trying
> to find out who (hopefully someone internal) registered
> github.com/flexiant - if not I'll use something else - and we will
> be putting all our open source stuff up there, and have a link from
> the open source page in the manual and the open source page on
> the website. This plus at least one of those links should be up
> in the next few days.

Just to close the loop on this, https://github.com/flexiant now
is a single point for all our (that's work's - i.e. Flexiant's)
open source components, in git format (rather than tarballs),
which is linked from our main open source page on our web site,
and will be (as of mid next week) linked from our product manual.

Probably more relevant to xen-devel, if I'm sticking patches up
for review etc. they will still be in https://github.com/abligh
which carries more branches, e.g. including code that doesn't
actually get released.

Constructive criticism welcomed, but I would suggest it is off
topic for xen-devel, so please send off list.

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

--On 14 June 2013 14:34:00 +0100 Ian Murray <murrayie@yahoo.co.uk> wrote:

> Besides, I am not sure if providing source via a repo who's address isn't
> publically published is really complying with your resposibilities, is
> it? Again, not a lawyer, just IMHO.

I suggest you read the requirements of the licence.

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Konrad Rzeszutek Wilk]

On Fri, Jun 14, 2013 at 10:46:31AM +0100, Ian Murray wrote:
> 
> 
> 
> 
> ----- Original Message -----
> > From: Alex Bligh <alex@alex.org.uk>
> > To: Ian Murray <murrayie@yahoo.co.uk>; xen-devel@lists.xen.org
> > Cc: Alex Bligh <alex@alex.org.uk>
> > Sent: Friday, 14 June 2013, 8:01
> > Subject: Re: [Xen-devel] [Hackathon Minutes] Xen 4.4 Planning
> > 
> > Ian,
> > 
> > --On 14 June 2013 00:56:14 +0100 Ian Murray <murrayie@yahoo.co.uk> wrote:
> > 
> >>  When you talk about "stuff" breaking between major releases, are 
> > you
> >>  talking about Xen code not functioning or your code failing because of
> >>  changes in Xen? If the latter, are we talking designed changes in Xen's
> >>  behaviour or non-designed ones (=bugs)?
> > 
> > Both.
> > 
> > As an example of the first, the API changed very significantly between
> > 3.x and 4.1, and 4.1 and 4.2. Some of the API changes were subtle (e.g.
> > what you had to do across a fork()).
> 
> But surely, this is your business. This is what you do. Sounds like you want API changes to be held back for the entire community so your company has to do less work.

That is not true. I know what Alex is referring. I used to work
for VirtualIron and we had our own C based 'agent' that would interface
with libxc. Meaning we did not have 'xend', nor any 'xl'. Launching of
guests, etc was all done via our 'agent'.

What we had issues was that when compiling against a new hypervisor
API it all compiled, but launching guests was broken. What we found was
that some structures had changed and their corresponding XEN_VERSION_SOMETHING
changed as well. But it was not clear _which_ of the structures changed
so it took a while to figure out that the cpumap had been changed
(I think that is what it was). There was also some HVM parameter field
that had to be added otherwise the guest would not boot (can't remember
the details).

The XEN_VERSION_someting in the header doesn't really say - what
had changed. It is incremented to guard against this (so the call to
the hypervisor will error out with -ENOSYS if you don't use the
right version)- but worst it does not provide very much a backwards
compatible way of having a runtime work against Xen 4.1 vs Xen 4.3
(or Xen 3.4).

And now looking back at Xen 4.3 I have to admin that I did this as well
- so the xcinfo has an extra unsigned long in it expanding it.

I am not really sure how to solve that. The 'libxl' should provide a nice
isolation layer for this kind of stuff and guard the consumers
of the API with the #NEW_SOMETHING_FEATURE_ADDED defines.

Re: [Hackathon Minutes] Xen 4.4 Planning

[Jan Beulich]

>>> On 13.06.13 at 23:03, Alex Bligh <alex@alex.org.uk> wrote:
> The thing we like the second least about Xen is how long it seems to take
> to get what we count as serious bugs fixed, even in stable releases.

I think applicable bug fixes get applied to the stable branches in
quite timely a manner, at least on the hypervisor side. Hence I
can only assume that you're unhappy with the rate of stable
releases. Yet I don't think we're going to get anywhere near of
the almost weekly stable releases that get done for Linux. As
you also didn't say what expectations you really have, it's hard
to take out anything useful from that complaint.

> We like it even less if we have to find them and fix them.

But isn't that how open source projects work - everyone contributes
and fixes bugs. If you don't want to help fixing bugs, I'm afraid
there's also no good reason for you to complain they don't get fixed.

> By 'serious' I mean basic functionality not working, crashes dom0, etc.

When did we last break Dom0, and not fix it in a timely manner?

> The result of this is two-fold. Firstly, we've never (yet) been able to
> run a production version of xen which is a standard xen release. We've
> always had to maintain our own patches even on 'stable' releases.
> Frankly, this is a pain.

But if you don't contribute back your patches, how do you expect
them to get accepted/merged? Or are you saying that it's _far_
more than occasional that patches from you get entirely lost?

Jan

Re: [Hackathon Minutes] Xen 4.4 Planning

[George Dunlap]

On 14/06/13 09:15, Jan Beulich wrote:
> But isn't that how open source projects work - everyone contributes
> and fixes bugs. If you don't want to help fixing bugs, I'm afraid
> there's also no good reason for you to complain they don't get fixed.

Well, Jan, that's how the market works -- people use the project that 
has the best code and the least bugs; if people find more stability with 
KVM, then people are likely to switch to it even if they don't think the 
performance or architecture is as good as they want.  It's no good 
complaining that your users aren't helping to fix bugs if there are 
other projects with fewer bugs to fix. :-)

Alex and his team has been very helpful in helping test and track down 
some issues in 4.3, and we really appreciate the effort they've put in.

  -George

Re: [Hackathon Minutes] Xen 4.4 Planning

[Lars Kurth]

[-- Attachment #1.1: Type: text/plain, Size: 3327 bytes --]

Hi all,

I will go through the thread and try and answer question (where they have
not been answered and discussed), but will need the help of the people who
actually attended and drove the discussion at the Hackathon. I merely took
notes of what was said, and at times I missed things.

And no decision of any kind has been made : for this we do have a decision
making process. But this thread seems to show that we do have a number of
problems that need fixing. Maybe we need to crystallize this a little more.

>From Jan:
> From the summary above I didn't really get what's wrong with the current
9 month cycle, which ...
I don't think this was indeed covered. I think the rationale was really
covered in the thread. I also think I minuted the 2-3 weeks + 3 months
wrong (which seems to have been clarified).

>From Ben Guthro:
> Did you have any more info on this bullet, and the other one below Re: XenClient
/ VirtualComputer?
The only thing I remember was that this discussion was originally started
by George and that IanC, Konrad, Stefano and Matt mostly covered it. We do
seem to have a problem with PCI passthrough - Pasi's points. Maybe they
recall more detail.

I also wanted to add to the point that Alex has made on serious bugs in xen
vs. kvm : this gets raised regularly by Xen users when I attend
conferences. And I also heard a few times now that we appear to focus more
on sexy features rather than the basics. This is relatively new though: the
first time I noticed was towards the end of last year. Which does not mean
that this is new: it may just mean that top issues/concerns that were
frequently raised before (mainly trust in the future of the project) have
disappeared.

Lars


On Fri, Jun 14, 2013 at 9:15 AM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 13.06.13 at 23:03, Alex Bligh <alex@alex.org.uk> wrote:
> > The thing we like the second least about Xen is how long it seems to take
> > to get what we count as serious bugs fixed, even in stable releases.
>
> I think applicable bug fixes get applied to the stable branches in
> quite timely a manner, at least on the hypervisor side. Hence I
> can only assume that you're unhappy with the rate of stable
> releases. Yet I don't think we're going to get anywhere near of
> the almost weekly stable releases that get done for Linux. As
> you also didn't say what expectations you really have, it's hard
> to take out anything useful from that complaint.
>
> > We like it even less if we have to find them and fix them.
>
> But isn't that how open source projects work - everyone contributes
> and fixes bugs. If you don't want to help fixing bugs, I'm afraid
> there's also no good reason for you to complain they don't get fixed.
>
> > By 'serious' I mean basic functionality not working, crashes dom0, etc.
>
> When did we last break Dom0, and not fix it in a timely manner?
>
> > The result of this is two-fold. Firstly, we've never (yet) been able to
> > run a production version of xen which is a standard xen release. We've
> > always had to maintain our own patches even on 'stable' releases.
> > Frankly, this is a pain.
>
> But if you don't contribute back your patches, how do you expect
> them to get accepted/merged? Or are you saying that it's _far_
> more than occasional that patches from you get entirely lost?
>
> Jan
>
>

[-- Attachment #1.2: Type: text/html, Size: 4806 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Hackathon Minutes] Xen 4.4 Planning

[Jan Beulich]

>>> On 14.06.13 at 11:59, Lars Kurth <lars.kurth.xen@gmail.com> wrote:
> I also wanted to add to the point that Alex has made on serious bugs in xen
> vs. kvm : this gets raised regularly by Xen users when I attend
> conferences. And I also heard a few times now that we appear to focus more
> on sexy features rather than the basics. This is relatively new though: the
> first time I noticed was towards the end of last year. Which does not mean
> that this is new: it may just mean that top issues/concerns that were
> frequently raised before (mainly trust in the future of the project) have
> disappeared.

I'd really like to see examples of this - there ought to be quite a
few according to what you write, yet I don't seem to recall any
that got plainly ignored. Of course there are always bugs which
take longer than others to figure out and fix.

Jan

Re: [Hackathon Minutes] Xen 4.4 Planning

[George Dunlap]

On 14/06/13 11:45, Jan Beulich wrote:
>>>> On 14.06.13 at 11:59, Lars Kurth <lars.kurth.xen@gmail.com> wrote:
>> I also wanted to add to the point that Alex has made on serious bugs in xen
>> vs. kvm : this gets raised regularly by Xen users when I attend
>> conferences. And I also heard a few times now that we appear to focus more
>> on sexy features rather than the basics. This is relatively new though: the
>> first time I noticed was towards the end of last year. Which does not mean
>> that this is new: it may just mean that top issues/concerns that were
>> frequently raised before (mainly trust in the future of the project) have
>> disappeared.
> I'd really like to see examples of this - there ought to be quite a
> few according to what you write, yet I don't seem to recall any
> that got plainly ignored. Of course there are always bugs which
> take longer than others to figure out and fix.

Well one example was that from the beginning of time until the 4.3 
release, you could only specify a single USB device in the config file 
for an HVM domain.

Another one is the qemu one we're discussing right now -- it was 
reported back in February I think, but it's only recently actually 
getting the attention that it needs to be sorted out.  The root cause 
was known by March, at which point there would have been plenty of time 
for a "proper fix".

And there's random things like cd-eject or cd-insert not working in 
certain circumstances (namely, when blktap was available).  It's 
something basic, but something we don't really use very much, and never 
got flagged up.  So it was broken in the 4.2 release.

I don't really know if this is the kind of thing that users are talking 
about, but if I used a release of a (supposedly) mature product, and 
something as basic as "insert a CD" crashed, I wouldn't be very impressed.

  -George

Re: [Hackathon Minutes] Xen 4.4 Planning

[Gordan Bobic]

 On Fri, 14 Jun 2013 11:45:52 +0100, "Jan Beulich" <JBeulich@suse.com> 
 wrote:
>>>> On 14.06.13 at 11:59, Lars Kurth <lars.kurth.xen@gmail.com> wrote:
>> I also wanted to add to the point that Alex has made on serious bugs 
>> in xen
>> vs. kvm : this gets raised regularly by Xen users when I attend
>> conferences. And I also heard a few times now that we appear to 
>> focus more
>> on sexy features rather than the basics. This is relatively new 
>> though: the
>> first time I noticed was towards the end of last year. Which does 
>> not mean
>> that this is new: it may just mean that top issues/concerns that 
>> were
>> frequently raised before (mainly trust in the future of the project) 
>> have
>> disappeared.
>
> I'd really like to see examples of this - there ought to be quite a
> few according to what you write, yet I don't seem to recall any
> that got plainly ignored. Of course there are always bugs which
> take longer than others to figure out and fix.

 Not a bug per se, but arguably a relatively important feature; does
 the lack of feature to pass multiple USB devices to domU
 count as an example of this?

 Gordan

Re: [Hackathon Minutes] Xen 4.4 Planning

[Sander Eikelenboom]

Friday, June 14, 2013, 11:59:35 AM, you wrote:

> Hi all,

> I will go through the thread and try and answer question (where they have
> not been answered and discussed), but will need the help of the people who
> actually attended and drove the discussion at the Hackathon. I merely took
> notes of what was said, and at times I missed things.

> And no decision of any kind has been made : for this we do have a decision
> making process. But this thread seems to show that we do have a number of
> problems that need fixing. Maybe we need to crystallize this a little more.

>>From Jan:
>> From the summary above I didn't really get what's wrong with the current
> 9 month cycle, which ...
> I don't think this was indeed covered. I think the rationale was really
> covered in the thread. I also think I minuted the 2-3 weeks + 3 months
> wrong (which seems to have been clarified).

>>From Ben Guthro:
>> Did you have any more info on this bullet, and the other one below Re: XenClient
> / VirtualComputer?
> The only thing I remember was that this discussion was originally started
> by George and that IanC, Konrad, Stefano and Matt mostly covered it. We do
> seem to have a problem with PCI passthrough - Pasi's points. Maybe they
> recall more detail.

> I also wanted to add to the point that Alex has made on serious bugs in xen
> vs. kvm : this gets raised regularly by Xen users when I attend
> conferences. And I also heard a few times now that we appear to focus more
> on sexy features rather than the basics. This is relatively new though: the
> first time I noticed was towards the end of last year. Which does not mean
> that this is new: it may just mean that top issues/concerns that were
> frequently raised before (mainly trust in the future of the project) have
> disappeared.

One of the things mentioned below is the "break dom0", i think KVM has some advantage here.
They could learn from the "mistakes" from Xen and make a new implementation from the ground up.

One of the causes could be that the Xen project has had to make a giant leap on multiple fronts from the "(forced) choice / mistake (in hind sight perhaps)"
to fork the Kernel and Qemu. Not to forget the tool-stack change that are also underway ...
Especially since that leap had to be made on multiple fronts at once and has it's interdepencies everywhere.

In the mean time there is still the wish and effort to not go and lag behind feature-wise as well (arm port).

Xen still seems to lag somewhat behind, although it doesn't need a seperate "special" Xen kernel anymore
since the PV parts have been upstreamed. But the giant leap, limited resource and wish to keep PV and functionality on par at one side,
and the views and interest of the Linux community and maintainers on the other side seems to have:

1) Disgruntle some Linux maintainers
2) Made code and interfaces not always clean and well documented
3) Not always uses the current kernel interfaces in the ways it was expected (by the authors/maintainers)

Which causes:
1) It makes patches from authors not familiar with Xen break things, this often happens in the merge window.
2) Maintenance burden, it requires quite some effort to keep up with the impact of (non primarily xen-targeted) changes in Linux and Qemu (which can't be invested in nice new features)
3) A lot of legacy stuff perhaps hinders the way forward ...

I must say Konrad does a tremendous job of keeping things together here and is *very* responsive !
I know Konrad is trying to clean this all up, refactor and document, but i think his time is split between quite some tasks and kernel parts.

One of the things I question is the need to still support machines without hardware virtualization ?
- Yes, it's one of the main features compared to f.e. KVM
- No, 32bit support seems to be on it's way out, although there are (intel) 64bit processors without hardware virtualiztion, is that worth the effort ?

But there doesn't seem to be a "architectural-roadmap", which not only tells "WHAT features", but is more orientated on (desirably) "HOW to implement / refactor / reimplement" ?
(so the grant-scheme of things so to say, and perhaps there can be things learned from KVM now ?)



Could a shorter development cycle, together with more focus, get things going faster ?

Perhaps the key is in more focus, focus can be seen in two ways:
1) focus of the complete release cycle (as a "theme"), f.e. do a tick-tock, tick=feature release cycle, tock=cleanup, code refactoring.
2) focus at front of which (limited number) of main features or cleanups the cycle focusses on, and make sure there is enough man-power in reserve to get these done.
   Any other feature can be included, but won't be commited unless in such a state that is well tested and will "guarantee" to not make the release date shift.
   For the chosen mean features, it could be a debian style release when ready and happy. The cleanup/refactoring release cycle could be shorter, but will probably need more
   synchronization with the 2 main dependencies (linux/qemu) projects.

So one release cycle would focus on new Xen-features and implements them in Xen and related projects (inside->out),
the second release cycle focuses on cleaning up legacy and refactoring, cleaning up code and tries to relieve the maintenance burden especially with the related projects (outside->in).

Perhaps this could suppress incomplete code and incomplete features between releases somewhat.

Unfortunately, i'm not that good of a low-level programmer, that my efforts would contribute much code wise,
but i try to contribute by testing new kernels early during the development cycle.
(in the hope it's easier to pinpoint the changes causing the bug and maximizing the time to fix things up)

Just my few cents ...

P.S.: I think the Xen community is quite responsive and nice to work with, and Xen as a whole works great for me :-)



--
Sander


> Lars


> On Fri, Jun 14, 2013 at 9:15 AM, Jan Beulich <JBeulich@suse.com> wrote:

>> >>> On 13.06.13 at 23:03, Alex Bligh <alex@alex.org.uk> wrote:
>> > The thing we like the second least about Xen is how long it seems to take
>> > to get what we count as serious bugs fixed, even in stable releases.
>>
>> I think applicable bug fixes get applied to the stable branches in
>> quite timely a manner, at least on the hypervisor side. Hence I
>> can only assume that you're unhappy with the rate of stable
>> releases. Yet I don't think we're going to get anywhere near of
>> the almost weekly stable releases that get done for Linux. As
>> you also didn't say what expectations you really have, it's hard
>> to take out anything useful from that complaint.
>>
>> > We like it even less if we have to find them and fix them.
>>
>> But isn't that how open source projects work - everyone contributes
>> and fixes bugs. If you don't want to help fixing bugs, I'm afraid
>> there's also no good reason for you to complain they don't get fixed.
>>
>> > By 'serious' I mean basic functionality not working, crashes dom0, etc.
>>
>> When did we last break Dom0, and not fix it in a timely manner?
>>
>> > The result of this is two-fold. Firstly, we've never (yet) been able to
>> > run a production version of xen which is a standard xen release. We've
>> > always had to maintain our own patches even on 'stable' releases.
>> > Frankly, this is a pain.
>>
>> But if you don't contribute back your patches, how do you expect
>> them to get accepted/merged? Or are you saying that it's _far_
>> more than occasional that patches from you get entirely lost?
>>
>> Jan
>>
>>

Re: [Hackathon Minutes] Xen 4.4 Planning

[George Dunlap]

On 13/06/13 22:03, Alex Bligh wrote:
>
>
> --On 13 June 2013 15:00:46 +0100 Lars Kurth <lars.kurth@xen.org> wrote:
>
>> We should aim to reduce the release cycle to 4 months (or maybe 6 months
>> as an intermediate step) from the current 9 months. A 4 months relase
>> cycle should help accelerate development and lead to fewer patches being
>> queued up. The implications are that we would have to operate a 2-3 
>> weeks
>> merge window.
>
> Disclaimer: I don't claim to be a xen developer, despite writing a few
> patches. But we are a heavy libxl API user. This perspective may or may
> not be useful from a 'consumer of your development' perspective. Delete
> or ignore if not.
>
> The thing we like the least about Xen is (was?), stuff breaking between
> major releases. If you have to drop 90% of your new features in order
> not to break stuff, please do just that. Xen 3.x -> Xen 4.1 (we mostly
> skipped 4.0) was a huge change, requiring a lot of rewriting. Xen 4.1
> to Xen 4.2 was far, far worse; had we not needed it to support
> qemu-upstream, we'd have probably stuck with 4.1. We talk to 4 
> hypervisors,
> and have never had this difficulty with any other hypervisor. You can
> imagine our joy when we found we could compile against 4.3 (we haven't
> tried running yet) without a single line code changed. Please, please,
> keep it like this. If we can also run unchanged against 4.3, this will
> be even better.

I think we've been aware of this for some time, and have only in the 
last few years really put an effort into sorting out some of these 
issues.  Having libxl as a library, with an API compatibility promise, 
was one step in that direction.  Another step in that was having a 
better testing infrastructure, which has helped a great deal.

If it's any comfort, the problems you had upgrading 3.x->4.1 and 
4.1->4.2 have been shared by the Citrix XenServer team, so there's a lot 
of desire to make the process better.

The testing of the upstream Xen is still really lacking, however -- this 
is a known problem, and is actually being discussed at the moment by the 
Linux Foundation project members.  At the moment, the real hard-core 
testing of Xen happens by the downstream users -- XenServer, OracleVM, 
SuSE, plus cloud vendors (including you) -- but this is a rather 
wasteful duplication of effort.  Having a more thorough testing 
infrastructure and set of test cases for upstream Xen would mean the 
standard Xen releases were of much higher quality; this would benefit 
everyone.

In any case, thanks for sharing your thoughts -- it's useful to have 
your perspective.  We'll have to keep these in mind going forward.

  -George

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

Jan,

--On 14 June 2013 09:15:35 +0100 Jan Beulich <JBeulich@suse.com> wrote:

>>>> On 13.06.13 at 23:03, Alex Bligh <alex@alex.org.uk> wrote:
>> The thing we like the second least about Xen is how long it seems to take
>> to get what we count as serious bugs fixed, even in stable releases.
>
> I think applicable bug fixes get applied to the stable branches in
> quite timely a manner, at least on the hypervisor side. Hence I
> can only assume that you're unhappy with the rate of stable
> releases. Yet I don't think we're going to get anywhere near of
> the almost weekly stable releases that get done for Linux. As
> you also didn't say what expectations you really have, it's hard
> to take out anything useful from that complaint.

I obviously miswrote some of this, as it wasn't intended as a complaint,
rather as observation. As with all open source code, the answer to 'it
doesn't work' is (a) try to find out why, and (b) send code. Which
we have on occasions tried to do.

I'm not unhappy with the rate of stable releases, I am/was unhappy about
the quality of stable releases (particularly 4.2). All our testing to
date on 4.3 indicates it's a lot higher quality release, at least
for what we want it for. It's possible that our expectations were
incorrect, because we were using 4.2 to support qemu-upstream dm,
and this was (IIRC) marked as a 'preview' feature; however we were
surprised things like live migrate were missing, and we had several
'killer' bugs. We've not found that on 4.3. Saying that, we've tested
4.3 quite a lot on its own, but not with our agent code, because we've
been trying to get 4.2 working properly first.

>> We like it even less if we have to find them and fix them.
>
> But isn't that how open source projects work - everyone contributes
> and fixes bugs. If you don't want to help fixing bugs, I'm afraid
> there's also no good reason for you to complain they don't get fixed.

I should have explicitly added the words 'in stable releases'. We're
perfectly happy to test, find bugs, etc. and indeed contribute fixes
as we have done. However, we shouldn't (in my opinion) be finding these
in stable releases.

>> By 'serious' I mean basic functionality not working, crashes dom0, etc.
>
> When did we last break Dom0, and not fix it in a timely manner?

Well, the 'fatal crash on Xen4.2 HVM' thread that started on 14 Dec 2012
had the last fix committed on 5 Apr 2013, and I think came out in 4.2.2
on 23 April. Between those points, as far as I'm concerned anything
running with network backed VMs was likely to crash dom0. That's about
half a 9 month release cycle.

>> The result of this is two-fold. Firstly, we've never (yet) been able to
>> run a production version of xen which is a standard xen release. We've
>> always had to maintain our own patches even on 'stable' releases.
>> Frankly, this is a pain.
>
> But if you don't contribute back your patches, how do you expect
> them to get accepted/merged?

We contributed back ALL of our patches. Every one. And we work pretty
hard to get them merged too. The only one that hasn't been merged or
superseded by a better patch is the minideb patch, which I fully
understand why it wasn't merged, and is just packaging.

I'm not complaining you drop my patches on the floor. In fact
I'm not really complaining at all. What I'm pointing out is that
if a 'stable' is released with a nasty bug in, and it takes a while
to find the right solution, it also takes a while to get a point
release out that fixes it. The solution here is not to change the
rate of acceptance of patches into a stable tree, but to ensure
the bugs are caught before they make their way into a stable
release.

> Or are you saying that it's _far_
> more than occasional that patches from you get entirely lost?

Nope.

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Jan Beulich]

>>> On 14.06.13 at 13:46, Alex Bligh <alex@alex.org.uk> wrote:
> I'm not unhappy with the rate of stable releases, I am/was unhappy about
> the quality of stable releases (particularly 4.2).

Oh, okay, so we seem to mean slightly different things with "stable
releases" - you seem to mean x.y.0 (in the current numbering
scheme), whereas to me this means x.y.z, with z != 0. Indeed,
there had been many bad bugs in 4.2.0, presumably at least partly
because of its overlong development cycle. But I think we got most
of them sorted out by now, with 4.2.2 being out.

> All our testing to
> date on 4.3 indicates it's a lot higher quality release, at least
> for what we want it for.

Let's hope for this to be true - a lower rate of fixes going into
what I call stable releases would be very much to my taste.

>>> We like it even less if we have to find them and fix them.
>>
>> But isn't that how open source projects work - everyone contributes
>> and fixes bugs. If you don't want to help fixing bugs, I'm afraid
>> there's also no good reason for you to complain they don't get fixed.
> 
> I should have explicitly added the words 'in stable releases'. We're
> perfectly happy to test, find bugs, etc. and indeed contribute fixes
> as we have done. However, we shouldn't (in my opinion) be finding these
> in stable releases.

Which implies that the features you were after weren't tested
sufficiently before the major release went out. Which may (but
doesn't have to) be an indicator that you didn't get to test them
early enough.

>>> By 'serious' I mean basic functionality not working, crashes dom0, etc.
>>
>> When did we last break Dom0, and not fix it in a timely manner?
> 
> Well, the 'fatal crash on Xen4.2 HVM' thread that started on 14 Dec 2012
> had the last fix committed on 5 Apr 2013, and I think came out in 4.2.2
> on 23 April. Between those points, as far as I'm concerned anything
> running with network backed VMs was likely to crash dom0. That's about
> half a 9 month release cycle.

I participated in this discussion only very early on, and hence
don't really know which fixes for this got committed where and
when. Which may mean that none of them were actually to the
hypervisor.

Jan

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

Jan,

--On 14 June 2013 13:26:59 +0100 Jan Beulich <JBeulich@suse.com> wrote:

>>>> On 14.06.13 at 13:46, Alex Bligh <alex@alex.org.uk> wrote:
>> I'm not unhappy with the rate of stable releases, I am/was unhappy about
>> the quality of stable releases (particularly 4.2).
>
> Oh, okay, so we seem to mean slightly different things with "stable
> releases" - you seem to mean x.y.0 (in the current numbering
> scheme), whereas to me this means x.y.z, with z != 0. Indeed,
> there had been many bad bugs in 4.2.0, presumably at least partly
> because of its overlong development cycle. But I think we got most
> of them sorted out by now, with 4.2.2 being out.

Yes. I meant anything other than -unstable. And as you say, it may be
(let's hope!) that this is an echo from the past (i.e. 4.2) and 4.3
will be great.

> Which implies that the features you were after weren't tested
> sufficiently before the major release went out. Which may (but
> doesn't have to) be an indicator that you didn't get to test them
> early enough.

That's partly the case, yes, and a fair point. We assumed the move to
4.2 was far easier than it turn out to be (because of API changes)
and concentrated on getting our own stuff to work with 4.2, rather
than trying 4.2 with xl etc. in anger. In our defence, all our
torture tests test our own software controlling xen.

However, I still feel the internal testing (automated or whatever)
should have picked some of these up.

>> Well, the 'fatal crash on Xen4.2 HVM' thread that started on 14 Dec 2012
>> had the last fix committed on 5 Apr 2013, and I think came out in 4.2.2
>> on 23 April. Between those points, as far as I'm concerned anything
>> running with network backed VMs was likely to crash dom0. That's about
>> half a 9 month release cycle.
>
> I participated in this discussion only very early on, and hence
> don't really know which fixes for this got committed where and
> when. Which may mean that none of them were actually to the
> hypervisor.

I can't remember now either. And the root of this is a kernel bug,
which we are working around. However, this was a known problem: Ian
Campbell (apologies if this was Ian J) discovered it in 2007 I think!
(links at the top of the original thread if people are interested
in the gory details).

-- 
Alex Bligh

Re: [Hackathon Minutes] Xen 4.4 Planning

[Alex Bligh]

--On 14 June 2013 13:25:38 -0400 Konrad Rzeszutek Wilk 
<konrad.wilk@oracle.com> wrote:

> What we had issues was that when compiling against a new hypervisor
> API it all compiled, but launching guests was broken. What we found was
> that some structures had changed and their corresponding
> XEN_VERSION_SOMETHING changed as well. But it was not clear _which_ of
> the structures changed so it took a while to figure out that the cpumap
> had been changed (I think that is what it was). There was also some HVM
> parameter field that had to be added otherwise the guest would not boot
> (can't remember the details).

That's precisely it.

Actually the one that REALLY bit us was far more subtle. We have a
multithreaded, forking agent. In Xen 4.2 (but not 4.1) you have to
call some post-fork function (whose name escapes me) in order to
tell xen about a fork. Signal handlers have to be in a particular
state (in particular SIGCHLD being set to ignore causes subtle
problems). And to get various xl calls working (mainly server
creation), it turns out you need to do a fork (assuming your
client is multithreaded and other threads call other xen things
even with different contexts); again this wasn't necessary in 4.1.

We have this working reliably now, but it was a voyage of
discovery particularly as there are no multithreaded client examples,
and whilst this is (sort of) explained in the header files, it's a
bit opaque. Seeing xl launch stuff, and our agent fail when it was
passing byte for byte the same values was frustrating. If I felt I
really understood how this worked (rather than discovered from trial
and error and a lot of strace), I'd have sent a documentation patch.

So this wasn't a change in the API in the sense of structs changing
or parameters changing (we caught that early at compile time), but
in how the thing gets used in practice.

I fully understand why the API was changed 4.2 -> 4.3, and really
appreciate the fact it's now stable (or at least advertised as
such). Initial indications (i.e. it built first time without
warnings) are good.

-- 
Alex Bligh