RFC: LTS and stable release scheme

RFC: LTS and stable release scheme

[Wei Liu]

Hi all

A majority of developers express interests in trying a shorter release
cycle -- to change from 9 months to 6 months [0]. There are, however,
repercussions on how we manage stable and possible LTS releases.

I start this thread hoping it's clearer that downstream consumers like
distributions and individual packagers can voice their opinions. I've
CC'ed some people I can think of who might be interested in this topic.
Feel free to CC more people.

We don't have LTS scheme at the moment. Let me start with current
scheme for stable releases.

 - Release from xen-unstable every 9 months.
 - Maintain last 3 releases as stable releases.

So in effect, every stable release is maintained for at least 27
months.

When we switch to 6 months release cycle, if we stick with the same
scheme (last 3 releases as stable releases), every stable release is
only maintained for 18 months. It's too short for downstream
consumers.

Ian proposed a scheme [1] in reply to [0]. In short, that scheme
proposes we pick every Nth release as LTS. He also proposed N=4 in the
case of 6 months release cycle.

(FWIW I think N=4 is a sensible suggestion.)

I can think of some open questions.

1. How long should each LTS release be maintained?

Here are my two cents, there are two fundamental criteria for deciding
how long a LTS is supported: a) it can't be shorter than what we
already have (27 months); b) at any given time there must be at least
one LTS release available to downstream.

With these in mind and N=4, I would say supporting LTS for at least
2.5 years (5 release cycles) should be the minimal requirement. As for
the upper limit, I don't know. Linux has LTS supports ranging from 2.5
years to 5.5 years [2]. Another data point is Debian LTS is supported
for 5 years [3].

Steven would like to see LTS be supported for 4 year full support + 1
year security fix only [4]. I'm not sure how feasible this is with
current set of maintainers (in fact, only Jan and Ian at the
moment). But in the end there is nothing that prevents other people
from stepping up and taking over the tree. We saw that with Xen 3.4
already.

2. What to do with the non-LTS releases?

I think they should still be considered stable releases for some
time. I'm just not sure for how long they should receive updates. One
way of looking at them is to use the same concept as Linux -- they
receive updates until next stable kernel is out. We can tweak this, of
course.

3. How to make maintenance effort scale?

Depending on how long each LTS and stable release is supported, at any
given time the number of trees may vary. It might be necessary to have
more stable tree maintainers to share the burden.

The current model for requesting backports is designed for single
maintainer for multiple trees. There are two ways to request
backports, one is to reply to Jan's preparation poll email, the other
is to embed backport request in patches. My personal experience is
that backport request via second way falls through the crack
sometimes, and as a *developer* I don't normally keep track of whether
the patches I requested ends up in stable trees.

Luckily I do see technical and procedural solution this is issue -- we
can setup stable@ alias to keep track of requests [5]. With that all
backport requests embedded in patches won't get lost. Downstream
consumers can also benefit from this because they then easily know
which patches are backport candidates.

We may also want to create aliases for LTS maintenance teams if there
are such teams.

Feel free to add more items and comments are welcome!

Wei.

[0]: <20151002174356.GA3577@zion.uk.xensource.com>
[1]: <1444045497.11707.195.camel@citrix.com>
[2]: https://www.kernel.org/category/releases.html
[3]: https://wiki.debian.org/LTS
[4]: <5612794E.8090700@crc.id.au>
[5]: <20151005125233.GD29124@zion.uk.xensource.com>

Re: RFC: LTS and stable release scheme

[George Dunlap]

On 06/10/15 12:07, Wei Liu wrote:
> Hi all
> 
> A majority of developers express interests in trying a shorter release
> cycle -- to change from 9 months to 6 months [0]. There are, however,
> repercussions on how we manage stable and possible LTS releases.
> 
> I start this thread hoping it's clearer that downstream consumers like
> distributions and individual packagers can voice their opinions. I've
> CC'ed some people I can think of who might be interested in this topic.
> Feel free to CC more people.
> 
> We don't have LTS scheme at the moment. Let me start with current
> scheme for stable releases.
> 
>  - Release from xen-unstable every 9 months.
>  - Maintain last 3 releases as stable releases.
> 
> So in effect, every stable release is maintained for at least 27
> months.
> 
> When we switch to 6 months release cycle, if we stick with the same
> scheme (last 3 releases as stable releases), every stable release is
> only maintained for 18 months. It's too short for downstream
> consumers.
> 
> Ian proposed a scheme [1] in reply to [0]. In short, that scheme
> proposes we pick every Nth release as LTS. He also proposed N=4 in the
> case of 6 months release cycle.
> 
> (FWIW I think N=4 is a sensible suggestion.)
> 
> I can think of some open questions.
> 
> 1. How long should each LTS release be maintained?
> 
> Here are my two cents, there are two fundamental criteria for deciding
> how long a LTS is supported: a) it can't be shorter than what we
> already have (27 months); b) at any given time there must be at least
> one LTS release available to downstream.
> 
> With these in mind and N=4, I would say supporting LTS for at least
> 2.5 years (5 release cycles) should be the minimal requirement. As for
> the upper limit, I don't know. Linux has LTS supports ranging from 2.5
> years to 5.5 years [2]. Another data point is Debian LTS is supported
> for 5 years [3].
> 
> Steven would like to see LTS be supported for 4 year full support + 1
> year security fix only [4]. I'm not sure how feasible this is with
> current set of maintainers (in fact, only Jan and Ian at the
> moment). But in the end there is nothing that prevents other people
> from stepping up and taking over the tree. We saw that with Xen 3.4
> already.

Could we just keep the same general scheme of "maintain 3 releases"?

If the most recent release was *not* an LTS, we'd have one release + 2
LTSes.  If the most recent release *was* an LTS, we'd have 3 LTSes --
the last one supported until a non-LTS release came out.  If LTSes
happen every 4 releases, then we'd get an LTS release every 2 years,
giving is 4.5 years support (54 months).

 -George

Re: RFC: LTS and stable release scheme

[Ian Campbell]

On Tue, 2015-10-06 at 12:07 +0100, Wei Liu wrote:
> 2. What to do with the non-LTS releases?
> 
> I think they should still be considered stable releases for some
> time. I'm just not sure for how long they should receive updates. One
> way of looking at them is to use the same concept as Linux -- they
> receive updates until next stable kernel is out. We can tweak this, of
> course.

FWIW I think reducing the time non-LTS stable releases are supported would
be a reasonable way to offset the extra cost of maintaining the LTS
releases for longer overall.

> Luckily I do see technical and procedural solution this is issue -- we
> can setup stable@ alias to keep track of requests [5]. With that all
> backport requests embedded in patches won't get lost. Downstream
> consumers can also benefit from this because they then easily know
> which patches are backport candidates.

It sounds here like you are proposing something more than a simple mail
alias dropping into the relevant maintainer(s) INBOX, i.e. something which
actually tracks the requests and publishes the queue and status etc for the
benefit of both users and the maintainers?

Ian.

Re: RFC: LTS and stable release scheme

[Wei Liu]

On Tue, Oct 06, 2015 at 12:07:58PM +0100, Wei Liu wrote:
> Hi all
> 
> A majority of developers express interests in trying a shorter release
> cycle -- to change from 9 months to 6 months [0]. There are, however,
> repercussions on how we manage stable and possible LTS releases.
> 
> I start this thread hoping it's clearer that downstream consumers like
> distributions and individual packagers can voice their opinions. I've
> CC'ed some people I can think of who might be interested in this topic.
> Feel free to CC more people.
> 
> We don't have LTS scheme at the moment. Let me start with current
> scheme for stable releases.
> 
>  - Release from xen-unstable every 9 months.
>  - Maintain last 3 releases as stable releases.
> 

Important correction, according to 

http://wiki.xenproject.org/wiki/Xen_Project_Maintenance_Releases

A stable release is supported fully for 18 months + 18 months security
updates.

This changes some numbers the following calculations. But I think the
premises for following discussions are still the same.

Wei.

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 06.10.15 at 13:07, <wei.liu2@citrix.com> wrote:
> A majority of developers express interests in trying a shorter release
> cycle -- to change from 9 months to 6 months [0]. There are, however,
> repercussions on how we manage stable and possible LTS releases.

I find it kind of odd to try to answer question 2 (cadence) without
first having answered 1 (whether to have stable releases at all). In
particular, the major downside of stable releases - them being
"better" than others - hasn't even been mentioned in you mail. And
afaict the reason for there being various LTS maintainers in Linux
is mainly because almost anyone can propose to LTS-maintain a
certain branch, and hence whenever an organization want a
certain release to be long term maintained all they have to ask
themselves is "Do we want to invest the resources for making it
so?" Along with this question goes - as seen internally here - the
question whether to instead align which kernel version to pick for
a product with which kernel version is expected to become an LTS
one.

I'm not sure if it's still that way nowadays, but in the years after
stable and long term releases got introduced in Linux even long
term branches weren't all equal: The general stable tree maintainer
actively argued against the use of certain branches (or certain
releases on a branch after it changed ownership), due to it being
of unknown (in the best case) quality.

Bottom line: I think the current model, with all releases being
equal and there being the opportunity to hand over branches to
"external" maintainers after the XenProject support ended
(exercised exactly once to date), is quite a bit better than any
of the LTS options I've seen proposed so far.

Jan

Re: RFC: LTS and stable release scheme

[Wei Liu]

On Tue, Oct 06, 2015 at 07:38:30AM -0600, Jan Beulich wrote:
> >>> On 06.10.15 at 13:07, <wei.liu2@citrix.com> wrote:
> > A majority of developers express interests in trying a shorter release
> > cycle -- to change from 9 months to 6 months [0]. There are, however,
> > repercussions on how we manage stable and possible LTS releases.
> 
> I find it kind of odd to try to answer question 2 (cadence) without
> first having answered 1 (whether to have stable releases at all). In

Right, all my questions were based on one assumption that "we may need
LTS".

> particular, the major downside of stable releases - them being
> "better" than others - hasn't even been mentioned in you mail. And

Is that a particular downside? I don't see it that way. Different
releases are, in nature, different. I can see why you want to treat
every release equally, but with limited man power we sometimes have to
make a choice.

> afaict the reason for there being various LTS maintainers in Linux
> is mainly because almost anyone can propose to LTS-maintain a
> certain branch, and hence whenever an organization want a
> certain release to be long term maintained all they have to ask
> themselves is "Do we want to invest the resources for making it
> so?" Along with this question goes - as seen internally here - the
> question whether to instead align which kernel version to pick for
> a product with which kernel version is expected to become an LTS
> one.
> 

The proposed scheme doesn't preclude such arrangement.

> I'm not sure if it's still that way nowadays, but in the years after
> stable and long term releases got introduced in Linux even long
> term branches weren't all equal: The general stable tree maintainer
> actively argued against the use of certain branches (or certain
> releases on a branch after it changed ownership), due to it being
> of unknown (in the best case) quality.
> 
> Bottom line: I think the current model, with all releases being
> equal and there being the opportunity to hand over branches to
> "external" maintainers after the XenProject support ended
> (exercised exactly once to date), is quite a bit better than any
> of the LTS options I've seen proposed so far.
> 

Yes. I see your point.  But at least one user has asked for longer term
full support. Up till now the "hand over" hasn't happened that often
(only once). And keep in mind that downstream consumers are not only
commercial vendors that have engineers working old trees for a product,
they might also be a distributions that simply wishes to have full Xen
support longer than their own release cycle.

With unlimited manpower we can use the current model to equally treat
each release LTS. In reality this is not the case, so I see the need for
using LTS model to effectively direct limited resources.

Or do you have different view on how various downstream can expect from
Xen to make the current model good enough for them?

What do you propose when we regarding stable branches when we switch to
6 months cycle?

Wei.


> Jan

Re: RFC: LTS and stable release scheme

[George Dunlap]

On Tue, Oct 6, 2015 at 2:38 PM, Jan Beulich <JBeulich@suse.com> wrote:
> I'm not sure if it's still that way nowadays, but in the years after
> stable and long term releases got introduced in Linux even long
> term branches weren't all equal: The general stable tree maintainer
> actively argued against the use of certain branches (or certain
> releases on a branch after it changed ownership), due to it being
> of unknown (in the best case) quality.

Perhaps I'm not familiar enough with what went on with Linux, but I
can't make much sense out of what you're describing here.  Are you
saying that the stable tree maintainer would say, "Officially 3.X.Y is
a stable release, but really you shouldn't use it, because it's a bit
dodgy.  Use 3.Z.Q or 3.M.N instead."

> Bottom line: I think the current model, with all releases being
> equal and there being the opportunity to hand over branches to
> "external" maintainers after the XenProject support ended
> (exercised exactly once to date), is quite a bit better than any
> of the LTS options I've seen proposed so far.

There's no reason that people can't offer to take up specific versions
once they fall off our own LTS.

 -George

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 06.10.15 at 16:12, <George.Dunlap@eu.citrix.com> wrote:
> On Tue, Oct 6, 2015 at 2:38 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> I'm not sure if it's still that way nowadays, but in the years after
>> stable and long term releases got introduced in Linux even long
>> term branches weren't all equal: The general stable tree maintainer
>> actively argued against the use of certain branches (or certain
>> releases on a branch after it changed ownership), due to it being
>> of unknown (in the best case) quality.
> 
> Perhaps I'm not familiar enough with what went on with Linux, but I
> can't make much sense out of what you're describing here.  Are you
> saying that the stable tree maintainer would say, "Officially 3.X.Y is
> a stable release, but really you shouldn't use it, because it's a bit
> dodgy.  Use 3.Z.Q or 3.M.N instead."

Yes. Or "Use 3.X.Y only up to a certain Y."

>> Bottom line: I think the current model, with all releases being
>> equal and there being the opportunity to hand over branches to
>> "external" maintainers after the XenProject support ended
>> (exercised exactly once to date), is quite a bit better than any
>> of the LTS options I've seen proposed so far.
> 
> There's no reason that people can't offer to take up specific versions
> once they fall off our own LTS.

Of course not, and I don't think I ruled this out. It's just that such
a branch wouldn't be considered XenProject.org-maintained anymore.

Jan

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 06.10.15 at 16:09, <wei.liu2@citrix.com> wrote:
> What do you propose when we regarding stable branches when we switch to
> 6 months cycle?

See the chicken-and-egg problem: I can't answer this, because the
issues with the stable trees are the main reason I don't support the
change to the release cycle (yet).

Jan

Re: RFC: LTS and stable release scheme

[Wei Liu]

On Tue, Oct 06, 2015 at 08:52:23AM -0600, Jan Beulich wrote:
> >>> On 06.10.15 at 16:09, <wei.liu2@citrix.com> wrote:
> > What do you propose when we regarding stable branches when we switch to
> > 6 months cycle?
> 
> See the chicken-and-egg problem: I can't answer this, because the
> issues with the stable trees are the main reason I don't support the
> change to the release cycle (yet).
> 

I have mistakenly taken your final reply to the other thread as your
agreement to the new release cycle.  Maybe we should continue our
discussion there. Let's draw an conclusion if we really want 6 months
before pursuing this LTS topic further.

Wei.

> Jan

Re: RFC: LTS and stable release scheme

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 897 bytes --]

On Tue, 2015-10-06 at 12:07 +0100, Wei Liu wrote:
> Hi all
> 
> A majority of developers express interests in trying a shorter
> release
> cycle -- to change from 9 months to 6 months [0]. There are, however,
> repercussions on how we manage stable and possible LTS releases.
> 
> I start this thread hoping it's clearer that downstream consumers
> like
> distributions and individual packagers can voice their opinions. I've
> CC'ed some people I can think of who might be interested in this
> topic.
> Feel free to CC more people.
> 
Cc-ing Michael, the Fedora package maintainer, and Marek, from QubesOS.

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: LTS and stable release scheme

[Wei Liu]

On Tue, Oct 06, 2015 at 02:10:05PM +0100, Ian Campbell wrote:
> On Tue, 2015-10-06 at 12:07 +0100, Wei Liu wrote:
> > 2. What to do with the non-LTS releases?
> > 
> > I think they should still be considered stable releases for some
> > time. I'm just not sure for how long they should receive updates. One
> > way of looking at them is to use the same concept as Linux -- they
> > receive updates until next stable kernel is out. We can tweak this, of
> > course.
> 
> FWIW I think reducing the time non-LTS stable releases are supported would
> be a reasonable way to offset the extra cost of maintaining the LTS
> releases for longer overall.
> 
> > Luckily I do see technical and procedural solution this is issue -- we
> > can setup stable@ alias to keep track of requests [5]. With that all
> > backport requests embedded in patches won't get lost. Downstream
> > consumers can also benefit from this because they then easily know
> > which patches are backport candidates.
> 
> It sounds here like you are proposing something more than a simple mail
> alias dropping into the relevant maintainer(s) INBOX, i.e. something which
> actually tracks the requests and publishes the queue and status etc for the
> benefit of both users and the maintainers?

I don't have definitive answer in mind.

It could be an alias for all maintainers. Making sure a patch contains
CC stable@ is good enough for searching through git log for candidates.

It could be a public mailing list that stable tree maintainers and
interested downstream users subscribe to.

The system you mentioned is a bit overly complicated as a starter. Let's
start with something simple.

Wei.

> 
> Ian.

Re: RFC: LTS and stable release scheme

[Ian Campbell]

On Tue, 2015-10-06 at 17:25 +0100, Wei Liu wrote:
> It could be an alias for all maintainers. Making sure a patch contains
> CC stable@ is good enough for searching through git log for candidates.

Ah yes, I suppose the Linux folks use git log --grep=stable@vger a lot,
which does a bunch of the work for certain cases.

But beyond that grep-fodder I'm not convinced that the actual alias itself
as an email address is going to be any help at all.

Ian.

Re: RFC: LTS and stable release scheme

[George Dunlap]

On Tue, Oct 6, 2015 at 3:52 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 06.10.15 at 16:09, <wei.liu2@citrix.com> wrote:
>> What do you propose when we regarding stable branches when we switch to
>> 6 months cycle?
>
> See the chicken-and-egg problem: I can't answer this, because the
> issues with the stable trees are the main reason I don't support the
> change to the release cycle (yet).

OK, so we have to decide on a *set* of factors, rather than just one.

Let's try to be analytic.

So we have so far identified 4 "parameters" that we can tweak:

1. Release frequency.  At the moment, this is "9 months".
2. Length of time bug fixes are backported.  At the moment this is "18 months".
3. Length of time security fixes are backported.  At the moment this
is "36 months" (i.e., original 18 months of bug fixes + 18 further
months of security-only fixes)
4. Subset of releases given treatment for #1 and #2.  At the moment
this is "all".
5. The release frequency for maintenance release.  At the moment this
is every 3 months.

Let's call this "R9 S3 {18,36}"  (Release every 9 months, maintenance
release updates every 3 months, all releases get bug fixes for 18
months and security updates for 36 months".

Metric-able criteria that are important to people:
1. Amount of time once a feature has been accepted until the time it
shows up in a release.  This is directly related to release frequency.
2. Amount of time a bug fix has been checked into tree before it shows
up in a maintenance release.  This is directly related to the
maintenance release frequency.
3. Amount of time a release receives bug fixes and/or security
updates.  These are related directly.
4. Total amount of work to maintain stable releases.  The *number* of
such releases is directly related to all of the above, but must be
calculated.

"All releases the same" is a criteria that's been brought up, but I
think there are probably practical implications we haven't teased out
yet.

(Please let me know if there are other criteria we should be considering.)

The steady state for "R9 S3 {18,36}" is to have 2 releases receiving
bug fixes, and 2 releases receiving security fixes at any given time;
and this happens every 3 months; total of 8 bug-fix and 8
security-only ports per year.

There are many people in favor of changing #1 to "6 months".  This
gives us "R6 S3 {18, 36}".

The steady-state for "R6 S3 {18,36}" is to have 3 releases receiving
bug fixes and 3 releases receiving security-only fixes at any given
time.  This gives is a total of 12 bug-fix and 12 security-only ports
per year.

It has so far been assumed in this discussion that this would be an
undue burden, and therefore not acceptable.  But it's worth asking
whether that is actually the case.  It has been argued, for instance,
that the difficulty in backporting a patch scales with the distance in
commits that the patch has to be ported over.  Porting a patch to a
releases 3, 9, and 15 months ago isn't 1.5x as much work as porting it
to one 3 and 12 months ago.

But I would guess that the porting *is* more work; and the building,
testing, and releasing certainly is 1.5x more work.

We could also explore other options, like having more automation, or
spreading the work of porting patches among more people.

Another option could be to decrease the frequency of maintenance
releases.  "R6 S4 {18,36}" gives us 9 bug-fix and 9 security-only
ports per year -- not much more than what we have now.  But of course
this increases the amount of time between a bug fix and the time it
shows up in a maintenance release, which is less desirable.

Another thing which has been suggested is to treat releases
differently; for instance, every 4 releases (24 months), release an
"LTS" release that has longer support requirements.  If we chose to
have LTS releases, we may then also choose to have normal releases get
a shorter support window.  It's also been suggested that security
fixes are really what downstreams need from an LTS release; once
you've been using something for more than 2 years, you've probably
already found all the bugs which bother you.

So one option would be "R6 S3 {12,12} L24 S3 {24, 48}" (Regular
releases every 6 months, maintenance releases every 3 months, fixes
backported for 12 months; LTS release every 24 months, bug-fixes
backported for 24 months, security fixes backported for 48 months).
Steady-state this gives us a max 3 releases receiving bug-fix
backports and 1 release receiving security backports at any given
time.  This gives us 12 bug-fix and 4 security-only ports per year.

Any thoughts on this?  Anything else to add or discuss?

Personally I like the LTS model.  I think arguably if we use the
numbers I chose above, it shouldn't be any more actual work than it is
now (as there are still only 4 releases every 3-month maintenance
window), and I don't understand the concern with "treating releases
differently".

 -George

Re: RFC: LTS and stable release scheme

[George Dunlap]

On Tue, Oct 6, 2015 at 2:38 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 06.10.15 at 13:07, <wei.liu2@citrix.com> wrote:
>> A majority of developers express interests in trying a shorter release
>> cycle -- to change from 9 months to 6 months [0]. There are, however,
>> repercussions on how we manage stable and possible LTS releases.
>
> I find it kind of odd to try to answer question 2 (cadence) without
> first having answered 1 (whether to have stable releases at all). In
> particular, the major downside of stable releases - them being
> "better" than others - hasn't even been mentioned in you mail. And
> afaict the reason for there being various LTS maintainers in Linux
> is mainly because almost anyone can propose to LTS-maintain a
> certain branch, and hence whenever an organization want a
> certain release to be long term maintained all they have to ask
> themselves is "Do we want to invest the resources for making it
> so?" Along with this question goes - as seen internally here - the
> question whether to instead align which kernel version to pick for
> a product with which kernel version is expected to become an LTS
> one.
>
> I'm not sure if it's still that way nowadays, but in the years after
> stable and long term releases got introduced in Linux even long
> term branches weren't all equal: The general stable tree maintainer
> actively argued against the use of certain branches (or certain
> releases on a branch after it changed ownership), due to it being
> of unknown (in the best case) quality.
>
> Bottom line: I think the current model, with all releases being
> equal and there being the opportunity to hand over branches to
> "external" maintainers after the XenProject support ended
> (exercised exactly once to date), is quite a bit better than any
> of the LTS options I've seen proposed so far.

So going back over this mail, I don't think I could restate your
argument other than "releases are not equal".

The first half of the first paragraph is how Linux ended up with
multiple LTS maintainers.  So far we've been suggesting the xenproject
team doing it, and choosing based on a regular cadence, so that's not
relevant.

The second half is about choosing a kernel version based on expected
LTSes.  I don't quite see what the issue here is either; "is expected
to become" may point to the idea that it's not clear ahead of time
which release will become an LTS.  If that's the case, then it is
addressed by having a regular cadence announced ahead of time.

The second paragraph is about even LTS releases being unequal.  That
won't be an issue with our proposal, because the xenproject team is
doing the maintenance, so quality is under our control.  (Or to put it
another way -- it's not any different from recommending people to use
4.4 instead of 4.2, which are also stable releases.)

The final paragraph talks about handing over branches to "external"
maintainers, but that's also still an available option to us -- if
someone wants to take a non-LTS release and continue to support it, or
take an LTS release and support it beyond our normal window, they can
still do that.

So taking out everything that's been addressed or doesn't apply, I get:

"The major downside of stable releases -- them being 'better' than
others -- hasn't been mentioned. ...Bottom line: I think the current
model, with all releases being equal... is quite a bit better than any
of the LTS  options I've seen proposed so far."

Which unfortunately doesn't help me understand where you're coming
from.  Could you try to expand on that a bit?

 -George

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 07.10.15 at 19:45, <George.Dunlap@eu.citrix.com> wrote:
> OK, so we have to decide on a *set* of factors, rather than just one.
> 
> Let's try to be analytic.
> 
> So we have so far identified 4 "parameters" that we can tweak:
> 
> 1. Release frequency.  At the moment, this is "9 months".
> 2. Length of time bug fixes are backported.  At the moment this is "18 
> months".
> 3. Length of time security fixes are backported.  At the moment this
> is "36 months" (i.e., original 18 months of bug fixes + 18 further
> months of security-only fixes)
> 4. Subset of releases given treatment for #1 and #2.  At the moment
> this is "all".
> 5. The release frequency for maintenance release.  At the moment this
> is every 3 months.

We switched this to 4 months not too long ago. This changes some
of the numbers further down, but not in a way that would significantly
affect the meat of this analysis.

> The steady state for "R9 S3 {18,36}" is to have 2 releases receiving
> bug fixes, and 2 releases receiving security fixes at any given time;
> and this happens every 3 months; total of 8 bug-fix and 8
> security-only ports per year.

Not sure what the latter 8 stands for - we don't currently do any
more releases from branches in security maintenance mode, all we
do is provide respective backports in XSAs and apply those to the
trees once the issues get published.

> It has so far been assumed in this discussion that this would be an
> undue burden, and therefore not acceptable.

I don't think anyone said "undue". All that was said was that the
amount of work to be put into this increases.

>  But it's worth asking
> whether that is actually the case.  It has been argued, for instance,
> that the difficulty in backporting a patch scales with the distance in
> commits that the patch has to be ported over.  Porting a patch to a
> releases 3, 9, and 15 months ago isn't 1.5x as much work as porting it
> to one 3 and 12 months ago.
> 
> But I would guess that the porting *is* more work; and the building,
> testing, and releasing certainly is 1.5x more work.
> 
> We could also explore other options, like having more automation, or
> spreading the work of porting patches among more people.

Now that goes slightly astray: People like me having to maintain
older releases for much longer time need to do certain backports
anyway (the older the release, the higher the chance that a
difficult backport may result in a decision being made to ignore
that fix as not important enough, due to the risk of the backport
going wrong being higher than that of leaving the bug in place).
I.e. at least a good part of the _backporting_ overhead can
probably be discounted. What I consider more of an issue is the
tedious (because purely mechanical) task of committing the
patches to the respective stable trees, which (in my way of
doing it) implies test builds for every commit. This is time I
consider worthwhile spent as long as it doesn't grow too much,
but of course this time could be used for less boring and more
productive things.

Perhaps there's room for further automation here, yet as with
any automation the question is how quickly getting this in place
will amortize itself.

> Another thing which has been suggested is to treat releases
> differently; for instance, every 4 releases (24 months), release an
> "LTS" release that has longer support requirements.  If we chose to
> have LTS releases, we may then also choose to have normal releases get
> a shorter support window.  It's also been suggested that security
> fixes are really what downstreams need from an LTS release; once
> you've been using something for more than 2 years, you've probably
> already found all the bugs which bother you.

But wouldn't that mean that the current model of 3 years of security
support is (almost) what people want from an LTS branch?

> So one option would be "R6 S3 {12,12} L24 S3 {24, 48}" (Regular
> releases every 6 months, maintenance releases every 3 months, fixes
> backported for 12 months; LTS release every 24 months, bug-fixes
> backported for 24 months, security fixes backported for 48 months).
> Steady-state this gives us a max 3 releases receiving bug-fix
> backports and 1 release receiving security backports at any given
> time.  This gives us 12 bug-fix and 4 security-only ports per year.
> 
> Any thoughts on this?  Anything else to add or discuss?

So using the 4 month stable release cadence I get currently 6
stable releases a year (two maintained branches, ignoring short
periods of overlap), compared to 6 ordinary stable releases (two
ordinary branches) plus 6 LTS releases (two LTS branches) a year,
i.e. doubling the amount.

> Personally I like the LTS model.  I think arguably if we use the
> numbers I chose above, it shouldn't be any more actual work than it is
> now (as there are still only 4 releases every 3-month maintenance
> window), and I don't understand the concern with "treating releases
> differently".

If what becomes an LTS release is known up front, the pressure to
get things into such a release may (and, looking at what I got to
notice on Linux, will) be quite a bit higher. After all the expectation
and intention is for distros to particularly use those releases when
caring for long term maintenance. And with that why would
contributors bother much about getting things into non-LTS
releases when those won't get used by many / for a reasonable
time period anyway?

If what becomes an LTS release is to be determined only at the
end of a branch's ordinary life cycle, the above problem can be
avoided, but downstream consumers having picked the "wrong"
release will be penalized. This is what we've been getting
complaints on by openSUSE users relative to the kernel versions
(and which keeps coming up the discussion of up-rev-ing the
kernel version post release, which as you surely can imagine has
all kinds of up and down sides). Plus depending on who gets to
maintain the branch from that point on (if not handed to always
the same person), quality and hence value to the downstreams
may heavily vary.

Jan

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 07.10.15 at 19:56, <George.Dunlap@eu.citrix.com> wrote:
> On Tue, Oct 6, 2015 at 2:38 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 06.10.15 at 13:07, <wei.liu2@citrix.com> wrote:
>>> A majority of developers express interests in trying a shorter release
>>> cycle -- to change from 9 months to 6 months [0]. There are, however,
>>> repercussions on how we manage stable and possible LTS releases.
>>
>> I find it kind of odd to try to answer question 2 (cadence) without
>> first having answered 1 (whether to have stable releases at all). In
>> particular, the major downside of stable releases - them being
>> "better" than others - hasn't even been mentioned in you mail. And
>> afaict the reason for there being various LTS maintainers in Linux
>> is mainly because almost anyone can propose to LTS-maintain a
>> certain branch, and hence whenever an organization want a
>> certain release to be long term maintained all they have to ask
>> themselves is "Do we want to invest the resources for making it
>> so?" Along with this question goes - as seen internally here - the
>> question whether to instead align which kernel version to pick for
>> a product with which kernel version is expected to become an LTS
>> one.
>>
>> I'm not sure if it's still that way nowadays, but in the years after
>> stable and long term releases got introduced in Linux even long
>> term branches weren't all equal: The general stable tree maintainer
>> actively argued against the use of certain branches (or certain
>> releases on a branch after it changed ownership), due to it being
>> of unknown (in the best case) quality.
>>
>> Bottom line: I think the current model, with all releases being
>> equal and there being the opportunity to hand over branches to
>> "external" maintainers after the XenProject support ended
>> (exercised exactly once to date), is quite a bit better than any
>> of the LTS options I've seen proposed so far.
> 
> So going back over this mail, I don't think I could restate your
> argument other than "releases are not equal".

And indeed I didn't try to make any other statement, but just tried
to explain where un-equality can come from and what undesirable
things can result from it.

> The first half of the first paragraph is how Linux ended up with
> multiple LTS maintainers.  So far we've been suggesting the xenproject
> team doing it, and choosing based on a regular cadence, so that's not
> relevant.

It's been suggest, not decided, so I think considering the alternative(s)
is relevant.

> The second half is about choosing a kernel version based on expected
> LTSes.  I don't quite see what the issue here is either; "is expected
> to become" may point to the idea that it's not clear ahead of time
> which release will become an LTS.  If that's the case, then it is
> addressed by having a regular cadence announced ahead of time.
> 
> The second paragraph is about even LTS releases being unequal.  That
> won't be an issue with our proposal, because the xenproject team is
> doing the maintenance, so quality is under our control.  (Or to put it
> another way -- it's not any different from recommending people to use
> 4.4 instead of 4.2, which are also stable releases.)
> 
> The final paragraph talks about handing over branches to "external"
> maintainers, but that's also still an available option to us -- if
> someone wants to take a non-LTS release and continue to support it, or
> take an LTS release and support it beyond our normal window, they can
> still do that.
> 
> So taking out everything that's been addressed or doesn't apply, I get:
> 
> "The major downside of stable releases -- them being 'better' than
> others -- hasn't been mentioned. ...Bottom line: I think the current
> model, with all releases being equal... is quite a bit better than any
> of the LTS  options I've seen proposed so far."
> 
> Which unfortunately doesn't help me understand where you're coming
> from.  Could you try to expand on that a bit?

I think the main misunderstanding is - as said above already - that you
take it that what has been suggested is the one and only option, while
my previous reply really was meant to put side by side the problems
of that model with the problems of other models. With the outcome
that in my opinion the current model has the smallest set of problems
when considering the equality aspect. I didn't mean to negate that the
LTS model also has (consumer side) benefits over the current one, but
I'm undecided whether those outweigh the problems.

Jan

Re: RFC: LTS and stable release scheme

[George Dunlap]

On 08/10/15 09:05, Jan Beulich wrote:
>>>> On 07.10.15 at 19:45, <George.Dunlap@eu.citrix.com> wrote:
>> OK, so we have to decide on a *set* of factors, rather than just one.
>>
>> Let's try to be analytic.
>>
>> So we have so far identified 4 "parameters" that we can tweak:
>>
>> 1. Release frequency.  At the moment, this is "9 months".
>> 2. Length of time bug fixes are backported.  At the moment this is "18 
>> months".
>> 3. Length of time security fixes are backported.  At the moment this
>> is "36 months" (i.e., original 18 months of bug fixes + 18 further
>> months of security-only fixes)
>> 4. Subset of releases given treatment for #1 and #2.  At the moment
>> this is "all".
>> 5. The release frequency for maintenance release.  At the moment this
>> is every 3 months.
> 
> We switched this to 4 months not too long ago. This changes some
> of the numbers further down, but not in a way that would significantly
> affect the meat of this analysis.

Oh right -- it's still 3 months on the wiki page Wei pointed to earlier
in this thread [1]. That should probably be updated.

[1] http://wiki.xenproject.org/wiki/Xen_Project_Maintenance_Releases

> 
>> The steady state for "R9 S3 {18,36}" is to have 2 releases receiving
>> bug fixes, and 2 releases receiving security fixes at any given time;
>> and this happens every 3 months; total of 8 bug-fix and 8
>> security-only ports per year.
> 
> Not sure what the latter 8 stands for - we don't currently do any
> more releases from branches in security maintenance mode, all we
> do is provide respective backports in XSAs and apply those to the
> trees once the issues get published.

Maintenance release every 3 months -> 4 maintenance releases per year, *
2 outstanding bugfix relases = 8 total "bugfix" maintenance releases per
year.

>> It has so far been assumed in this discussion that this would be an
>> undue burden, and therefore not acceptable.
> 
> I don't think anyone said "undue". All that was said was that the
> amount of work to be put into this increases.

I think when you objected to the 6-month release cycle on the grounds
that it implies LTS releases, many people (Wei in particular)
interpreted that as saying that changing only that one parameter would
cause undue increase in load.  That is the assumption behind Wei's
commend earlier in this thread:

"With unlimited manpower we can use the current model to equally treat
each release LTS. In reality this is not the case, so I see the need for
using LTS model to effectively direct limited resources."

>>  But it's worth asking
>> whether that is actually the case.  It has been argued, for instance,
>> that the difficulty in backporting a patch scales with the distance in
>> commits that the patch has to be ported over.  Porting a patch to a
>> releases 3, 9, and 15 months ago isn't 1.5x as much work as porting it
>> to one 3 and 12 months ago.
>>
>> But I would guess that the porting *is* more work; and the building,
>> testing, and releasing certainly is 1.5x more work.
>>
>> We could also explore other options, like having more automation, or
>> spreading the work of porting patches among more people.
> 
> Now that goes slightly astray: People like me having to maintain
> older releases for much longer time need to do certain backports
> anyway (the older the release, the higher the chance that a
> difficult backport may result in a decision being made to ignore
> that fix as not important enough, due to the risk of the backport
> going wrong being higher than that of leaving the bug in place).
> I.e. at least a good part of the _backporting_ overhead can
> probably be discounted. What I consider more of an issue is the
> tedious (because purely mechanical) task of committing the
> patches to the respective stable trees, which (in my way of
> doing it) implies test builds for every commit. This is time I
> consider worthwhile spent as long as it doesn't grow too much,
> but of course this time could be used for less boring and more
> productive things.
> 
> Perhaps there's room for further automation here, yet as with
> any automation the question is how quickly getting this in place
> will amortize itself.

Aren't stable branches already tested with OSStest?  If they aren't they
certainly could be made to be tested pretty trivially, I think.
Wouldn't an osstest (which includes both build and functional tests)
plus an RC be sufficient?

>> Another thing which has been suggested is to treat releases
>> differently; for instance, every 4 releases (24 months), release an
>> "LTS" release that has longer support requirements.  If we chose to
>> have LTS releases, we may then also choose to have normal releases get
>> a shorter support window.  It's also been suggested that security
>> fixes are really what downstreams need from an LTS release; once
>> you've been using something for more than 2 years, you've probably
>> already found all the bugs which bother you.
> 
> But wouldn't that mean that the current model of 3 years of security
> support is (almost) what people want from an LTS branch?

Is there a specific proposal you are advocating here?

>> So one option would be "R6 S3 {12,12} L24 S3 {24, 48}" (Regular
>> releases every 6 months, maintenance releases every 3 months, fixes
>> backported for 12 months; LTS release every 24 months, bug-fixes
>> backported for 24 months, security fixes backported for 48 months).
>> Steady-state this gives us a max 3 releases receiving bug-fix
>> backports and 1 release receiving security backports at any given
>> time.  This gives us 12 bug-fix and 4 security-only ports per year.
>>
>> Any thoughts on this?  Anything else to add or discuss?
> 
> So using the 4 month stable release cadence I get currently 6
> stable releases a year (two maintained branches, ignoring short
> periods of overlap), compared to 6 ordinary stable releases (two
> ordinary branches) plus 6 LTS releases (two LTS branches) a year,
> i.e. doubling the amount.
> 
>> Personally I like the LTS model.  I think arguably if we use the
>> numbers I chose above, it shouldn't be any more actual work than it is
>> now (as there are still only 4 releases every 3-month maintenance
>> window), and I don't understand the concern with "treating releases
>> differently".
> 
> If what becomes an LTS release is known up front, the pressure to
> get things into such a release may (and, looking at what I got to
> notice on Linux, will) be quite a bit higher. After all the expectation
> and intention is for distros to particularly use those releases when
> caring for long term maintenance. And with that why would
> contributors bother much about getting things into non-LTS
> releases when those won't get used by many / for a reasonable
> time period anyway?
> 
> If what becomes an LTS release is to be determined only at the
> end of a branch's ordinary life cycle, the above problem can be
> avoided, but downstream consumers having picked the "wrong"
> release will be penalized. This is what we've been getting
> complaints on by openSUSE users relative to the kernel versions
> (and which keeps coming up the discussion of up-rev-ing the
> kernel version post release, which as you surely can imagine has
> all kinds of up and down sides). Plus depending on who gets to
> maintain the branch from that point on (if not handed to always
> the same person), quality and hence value to the downstreams
> may heavily vary.

OK, so the main argument in favor of 6-month release cycles is the
reduced time between check-in and a feature being in a release.  It has
been proposed that this shorter cycle will in part reduce the pressure
to get features in for any particular release.

Here is my summary of your argument above (let me know if it's accurate):

---
We have two options WRT LTS releases: Either we make LTS releases known
ahead of time, or we choose them afterwards.

There is no value to developers to getting features into non-LTS
releases; and therefore, if the LTS releases are known ahead of time,
then the LTS release will have exactly the same pressure as a normal
release to get features in.  (And of course, if we move from a 9-month
regular release to a 24-month LTS release, the pressure will be even
greater than it is now.)

[I'm not sure if you're arguing the following or not.] In fact, in the
time after an LTS release, many developers may completely stop
developing until the next LTS release comes around.

We could mitigate this developer pressure by not announcing LTS releases
ahead of time, so that looking forward, developers don't know which
release will be an LTS release.  But then we have other undesirable side
effects, namely, distros may end up settling on a version which turns
out not to be an LTS, and are then in a bind as to whether to change
versions post-release (disruptive and risky) or do all the maintenance
themselves (a lot of extra work) or just not have maintenance updates
(poor quality).
---

First of all, I completely agree that not announcing LTSes ahead of time
has unacceptable effects on distributions and other downstreams.  I
think LTSes, if we choose them, should happen on a regular, predictable
basis.

However, I don't agree that getting features into non-LTS releases is of
no value to developers.  Many users will use non-LTS releases (probably
Fedora, and a lot of our direct users, and probably anyone using
debian-testing most of the time).  Furthermore, downstreams that want
particular features are much more likely to cherry-pick them from stable
releases than from unreleased xen-unstable.

Of course there will be more desire to get a feature into an LTS than a
regular release, but I still think that this pressure will be less now,
and less frequent, than what we have now.

Ultimately we're all just making guesses here.  There is certainly a
risk that you will be proven correct (or partially correct).  I think
it's worth giving it a try.

On the other hand, if we can shorten to a 6-month release cycle while
*not* going to an LTS model (by making the maintenance releases more
automated and/or sharing the maintenance burden), I would be satisfied
with that as well.

I think at some point we may just have to throw out a couple of options
and do a 4-point survey as we've done before (i.e., "argue againts / not
happy but won't argue / happy but won't argue / argue for") and see
where we stand.

 -George

Re: RFC: LTS and stable release scheme

[Ian Jackson]

Jan Beulich writes ("Re: [Xen-devel] RFC: LTS and stable release scheme"):
>   What I consider more of an issue is the tedious (because purely
> mechanical) task of committing the patches to the respective stable
> trees, which (in my way of doing it) implies test builds for every
> commit. This is time I consider worthwhile spent as long as it
> doesn't grow too much, but of course this time could be used for
> less boring and more productive things.

I do test builds of my stable backports but only to the `oldest' tree.
I also find that the admin effort of backporting etc. is tedious.

> Perhaps there's room for further automation here, yet as with
> any automation the question is how quickly getting this in place
> will amortize itself.

Even with the current situation I think much more automation would be
good.  (But then I'm someone who really (a) likes automating things
(b) likes sitting back and watching the automation do its thing and
even (c) likes debugging the automation when it goes wrong.)

I think that maybe as a starting point, Jan and I could agree that
instead of build-testing our backports locally, we will throw them at
osstest and see what sticks.


As for release administrivia and tarball prep: gwd's work on the
release tarball target means that we are now close to the point where
I might think about turning some of the release checklist into a
script - at least, one that can be used for tagging and tarballing a
release on a stable branch.  (Full releases involve a lot more faff
and are different each time...)


Ian.

Re: RFC: LTS and stable release scheme

[Wei Liu]

On Thu, Oct 08, 2015 at 02:05:51AM -0600, Jan Beulich wrote:
> >>> On 07.10.15 at 19:45, <George.Dunlap@eu.citrix.com> wrote:
[...]
> 
> >  But it's worth asking
> > whether that is actually the case.  It has been argued, for instance,
> > that the difficulty in backporting a patch scales with the distance in
> > commits that the patch has to be ported over.  Porting a patch to a
> > releases 3, 9, and 15 months ago isn't 1.5x as much work as porting it
> > to one 3 and 12 months ago.
> > 
> > But I would guess that the porting *is* more work; and the building,
> > testing, and releasing certainly is 1.5x more work.
> > 
> > We could also explore other options, like having more automation, or
> > spreading the work of porting patches among more people.
> 
> Now that goes slightly astray: People like me having to maintain
> older releases for much longer time need to do certain backports
> anyway (the older the release, the higher the chance that a
> difficult backport may result in a decision being made to ignore
> that fix as not important enough, due to the risk of the backport
> going wrong being higher than that of leaving the bug in place).
> I.e. at least a good part of the _backporting_ overhead can
> probably be discounted. What I consider more of an issue is the
> tedious (because purely mechanical) task of committing the
> patches to the respective stable trees, which (in my way of
> doing it) implies test builds for every commit. This is time I
> consider worthwhile spent as long as it doesn't grow too much,
> but of course this time could be used for less boring and more
> productive things.
> 
> Perhaps there's room for further automation here, yet as with
> any automation the question is how quickly getting this in place
> will amortize itself.
> 

Building every commit can be easily implemented. I don't think it takes
more than a day to write a script.  What do you mean by "in place"? Are
there any other considerations?

> > Another thing which has been suggested is to treat releases
> > differently; for instance, every 4 releases (24 months), release an
> > "LTS" release that has longer support requirements.  If we chose to
> > have LTS releases, we may then also choose to have normal releases get
> > a shorter support window.  It's also been suggested that security
> > fixes are really what downstreams need from an LTS release; once
> > you've been using something for more than 2 years, you've probably
> > already found all the bugs which bother you.
> 
> But wouldn't that mean that the current model of 3 years of security
> support is (almost) what people want from an LTS branch?
> 

Perhaps. But we're still lacking data points here. The only downstream
that voiced opinion recently was Steven.

But for now, let's say 3 years of security support is good enough. Do
you have proposal that would fit 6 months release cycle? A majority of
developers seem to think 9 months is too long and 6 months is worth
trying.

> > So one option would be "R6 S3 {12,12} L24 S3 {24, 48}" (Regular
> > releases every 6 months, maintenance releases every 3 months, fixes
> > backported for 12 months; LTS release every 24 months, bug-fixes
> > backported for 24 months, security fixes backported for 48 months).
> > Steady-state this gives us a max 3 releases receiving bug-fix
> > backports and 1 release receiving security backports at any given
> > time.  This gives us 12 bug-fix and 4 security-only ports per year.
> > 
> > Any thoughts on this?  Anything else to add or discuss?
> 
> So using the 4 month stable release cadence I get currently 6
> stable releases a year (two maintained branches, ignoring short
> periods of overlap), compared to 6 ordinary stable releases (two
> ordinary branches) plus 6 LTS releases (two LTS branches) a year,
> i.e. doubling the amount.
> 

So this seems to be your major concern. My interpretation is that you
kept stressing workload because you think it is a major issue. 

I fail to get the idea why this would be a problem. Maybe you're seeing
every backport as your sole responsibility?  From Xen project's point of
view, this amount can be absorbed by a team of stable release
maintainers. Then this leads to your concern of varying quality of
different branches? But again, I fail to grasp that idea. All stable
trees maintained by Xen project are equally good.

Further more, there is argument such scheme wouldn't increase workload.
Ian mentioned the amount of backport during a set period is the same.
With automation mechanism in place (as simple as a script) there
wouldn't be too much work.

> > Personally I like the LTS model.  I think arguably if we use the
> > numbers I chose above, it shouldn't be any more actual work than it is
> > now (as there are still only 4 releases every 3-month maintenance
> > window), and I don't understand the concern with "treating releases
> > differently".
> 
> If what becomes an LTS release is known up front, the pressure to
> get things into such a release may (and, looking at what I got to
> notice on Linux, will) be quite a bit higher. After all the expectation
> and intention is for distros to particularly use those releases when
> caring for long term maintenance. And with that why would
> contributors bother much about getting things into non-LTS
> releases when those won't get used by many / for a reasonable
> time period anyway?
> 

I disagree. Many developers care about getting their patches upstream
but are not interested in which releases their features are going to be.

> If what becomes an LTS release is to be determined only at the
> end of a branch's ordinary life cycle, the above problem can be
> avoided, but downstream consumers having picked the "wrong"
> release will be penalized. This is what we've been getting
> complaints on by openSUSE users relative to the kernel versions
> (and which keeps coming up the discussion of up-rev-ing the
> kernel version post release, which as you surely can imagine has
> all kinds of up and down sides).

As you mentioned above, there are always upside and downside with *any*
change. We won't know until we try.

But what we do know at hand is that current scheme is not perfect, hence
the whole discussion.

> Plus depending on who gets to
> maintain the branch from that point on (if not handed to always
> the same person), quality and hence value to the downstreams
> may heavily vary.
> 

This is really irrelevant to this discussion. If we hand that to
external people, that means it's out of our control and by definition we
don't care that much -- no matter what scheme we use.

Wei.

> Jan

Re: RFC: LTS and stable release scheme

[George Dunlap]

On 08/10/15 09:05, Jan Beulich wrote:
>> It has so far been assumed in this discussion that this would be an
>> undue burden, and therefore not acceptable.
> 
> I don't think anyone said "undue". All that was said was that the
> amount of work to be put into this increases.

I think it's worth saying that the amount of effort it takes to make
maintenance releases -- in particular, the amount of tedious busy work
for you -- is a first-order consideration in this, particularly as you
don't see any value in the change.  It is certainly not reasonable for
everyone who is not directly affected to vote to give you more work, and
it would be perfectly reasonable for you to say that if we went to a
6-month release cycle, others would have to share (or take) the burden
of doing that busy-work.

 -George

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 08.10.15 at 12:59, <Ian.Jackson@eu.citrix.com> wrote:
> Jan Beulich writes ("Re: [Xen-devel] RFC: LTS and stable release scheme"):
>> Perhaps there's room for further automation here, yet as with
>> any automation the question is how quickly getting this in place
>> will amortize itself.
> 
> Even with the current situation I think much more automation would be
> good.  (But then I'm someone who really (a) likes automating things
> (b) likes sitting back and watching the automation do its thing and
> even (c) likes debugging the automation when it goes wrong.)
> 
> I think that maybe as a starting point, Jan and I could agree that
> instead of build-testing our backports locally, we will throw them at
> osstest and see what sticks.

Well, yes, we could. Otoh the overhead of fixing something that
didn't build but got committed already means more mechanical
work (revert, or create a fixup patch) compared to fixing it before
pushing to the respective staging tree.

What I would see as possibly useful would be a queue like thing
where backports could be added, and automation would take
care of committing and pushing as much of it as it can validate
to build (more severe problems are pretty rare in stable trees,
and hence relying on the normal osstest there like we do now
would seem reasonable). Yet again this would mean one may
have to turn attention to the respective tree more often (since
right now this is needed just once for each batch of backports,
unless something really odd happens).

Jan

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 08.10.15 at 12:39, <george.dunlap@citrix.com> wrote:
> On 08/10/15 09:05, Jan Beulich wrote:
>>>>> On 07.10.15 at 19:45, <George.Dunlap@eu.citrix.com> wrote:
>>> The steady state for "R9 S3 {18,36}" is to have 2 releases receiving
>>> bug fixes, and 2 releases receiving security fixes at any given time;
>>> and this happens every 3 months; total of 8 bug-fix and 8
>>> security-only ports per year.
>> 
>> Not sure what the latter 8 stands for - we don't currently do any
>> more releases from branches in security maintenance mode, all we
>> do is provide respective backports in XSAs and apply those to the
>> trees once the issues get published.
> 
> Maintenance release every 3 months -> 4 maintenance releases per year, *
> 2 outstanding bugfix relases = 8 total "bugfix" maintenance releases per
> year.

That explains the first 8, but I tried to point out that afaics there
is no second 8 (which however you had in your description as
"8 security-only ports per year").

>> Perhaps there's room for further automation here, yet as with
>> any automation the question is how quickly getting this in place
>> will amortize itself.
> 
> Aren't stable branches already tested with OSStest?  If they aren't they
> certainly could be made to be tested pretty trivially, I think.
> Wouldn't an osstest (which includes both build and functional tests)
> plus an RC be sufficient?

Yes, they are getting tested. Yet - as just said in a reply to Ian -
while it's a possible model to simply throw things in and wait for them
to break, I not only feel uneasy about doing so (we don't do that on
unstable after all), but I also dislike the idea of having to turn back
to the branch a day or so later (when the failure report arrives)
instead of (normally) being done once I hit enter on the "git push".

>>> Another thing which has been suggested is to treat releases
>>> differently; for instance, every 4 releases (24 months), release an
>>> "LTS" release that has longer support requirements.  If we chose to
>>> have LTS releases, we may then also choose to have normal releases get
>>> a shorter support window.  It's also been suggested that security
>>> fixes are really what downstreams need from an LTS release; once
>>> you've been using something for more than 2 years, you've probably
>>> already found all the bugs which bother you.
>> 
>> But wouldn't that mean that the current model of 3 years of security
>> support is (almost) what people want from an LTS branch?
> 
> Is there a specific proposal you are advocating here?

I'm rather asking whether, if security support is the main concern,
the current model really is that far from what is being wanted. E.g.
consider the option of just extending security support within the
current model.

> Here is my summary of your argument above (let me know if it's accurate):
> 
> ---
> We have two options WRT LTS releases: Either we make LTS releases known
> ahead of time, or we choose them afterwards.
> 
> There is no value to developers to getting features into non-LTS
> releases; and therefore, if the LTS releases are known ahead of time,
> then the LTS release will have exactly the same pressure as a normal
> release to get features in.  (And of course, if we move from a 9-month
> regular release to a 24-month LTS release, the pressure will be even
> greater than it is now.)

Well, not really "no value", but less.

> [I'm not sure if you're arguing the following or not.] In fact, in the
> time after an LTS release, many developers may completely stop
> developing until the next LTS release comes around.

And perhaps again not "completely", but slow down (perhaps
significantly).

> First of all, I completely agree that not announcing LTSes ahead of time
> has unacceptable effects on distributions and other downstreams.  I
> think LTSes, if we choose them, should happen on a regular, predictable
> basis.
> 
> However, I don't agree that getting features into non-LTS releases is of
> no value to developers.  Many users will use non-LTS releases (probably
> Fedora, and a lot of our direct users, and probably anyone using
> debian-testing most of the time).  Furthermore, downstreams that want
> particular features are much more likely to cherry-pick them from stable
> releases than from unreleased xen-unstable.

True.

> I think at some point we may just have to throw out a couple of options
> and do a 4-point survey as we've done before (i.e., "argue againts / not
> happy but won't argue / happy but won't argue / argue for") and see
> where we stand.

Agreed.

Jan

Re: RFC: LTS and stable release scheme

[Juergen Gross]

On 10/08/2015 01:34 PM, Jan Beulich wrote:
>>>> On 08.10.15 at 12:59, <Ian.Jackson@eu.citrix.com> wrote:
>> Jan Beulich writes ("Re: [Xen-devel] RFC: LTS and stable release scheme"):
>>> Perhaps there's room for further automation here, yet as with
>>> any automation the question is how quickly getting this in place
>>> will amortize itself.
>>
>> Even with the current situation I think much more automation would be
>> good.  (But then I'm someone who really (a) likes automating things
>> (b) likes sitting back and watching the automation do its thing and
>> even (c) likes debugging the automation when it goes wrong.)
>>
>> I think that maybe as a starting point, Jan and I could agree that
>> instead of build-testing our backports locally, we will throw them at
>> osstest and see what sticks.
>
> Well, yes, we could. Otoh the overhead of fixing something that
> didn't build but got committed already means more mechanical
> work (revert, or create a fixup patch) compared to fixing it before
> pushing to the respective staging tree.
>
> What I would see as possibly useful would be a queue like thing
> where backports could be added, and automation would take
> care of committing and pushing as much of it as it can validate
> to build (more severe problems are pretty rare in stable trees,
> and hence relying on the normal osstest there like we do now
> would seem reasonable). Yet again this would mean one may
> have to turn attention to the respective tree more often (since
> right now this is needed just once for each batch of backports,
> unless something really odd happens).

Couldn't that purely mechanical work be spread to others? I can't
believe this would require exceptional skills and I think your
time is to precious for stuff like that.

In the beginning the workflow could be the same as yours today,
there would be just the queue you mentioned and someone either
doing the builds and committing or just look after the results
of any automatism. It just wouldn't be you.


Just my 0.02€, Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 08.10.15 at 13:10, <wei.liu2@citrix.com> wrote:
> On Thu, Oct 08, 2015 at 02:05:51AM -0600, Jan Beulich wrote:
>> Perhaps there's room for further automation here, yet as with
>> any automation the question is how quickly getting this in place
>> will amortize itself.
>> 
> 
> Building every commit can be easily implemented. I don't think it takes
> more than a day to write a script.  What do you mean by "in place"? Are
> there any other considerations?

No. "In place" simply means this needs to be written and tested.
And as things stand what will fit Ian may not fit me or vice versa.
I.e. unless this would be a server side thing (see the queue thing
I mentioned in another reply), multiple scripts may end up being
needed. And a day is a day - if it saves, say, 15 minutes per batch,
it would take 32 batches for the work to amortize.

> But for now, let's say 3 years of security support is good enough. Do
> you have proposal that would fit 6 months release cycle? A majority of
> developers seem to think 9 months is too long and 6 months is worth
> trying.

And if there is a majority, I'm sure we will try it. I'm not arguing
because I mean to try to veto any such decision, I'm only trying
to point out that there also are reasons to stick with the current
model.

>> So using the 4 month stable release cadence I get currently 6
>> stable releases a year (two maintained branches, ignoring short
>> periods of overlap), compared to 6 ordinary stable releases (two
>> ordinary branches) plus 6 LTS releases (two LTS branches) a year,
>> i.e. doubling the amount.
> 
> So this seems to be your major concern. My interpretation is that you
> kept stressing workload because you think it is a major issue. 

Thing is - I'm not of the opinion that it's impossible to do, but I am
of the opinion that any amount of time spent on mechanical things
is worthwhile trying to avoid or at least limit as much as possible. I
don't know about you, but I prefer to do productive things...

> I fail to get the idea why this would be a problem. Maybe you're seeing
> every backport as your sole responsibility?  From Xen project's point of
> view, this amount can be absorbed by a team of stable release
> maintainers. Then this leads to your concern of varying quality of
> different branches? But again, I fail to grasp that idea. All stable
> trees maintained by Xen project are equally good.

Because right now they're all being managed by the same people.
The main reason for questioning the quality of some Linux stable
trees was that the criteria based upon which selection of backports
happened appeared to be quite different between individual people.
I'd be much less worried of the quality of the actual backports.

>> > Personally I like the LTS model.  I think arguably if we use the
>> > numbers I chose above, it shouldn't be any more actual work than it is
>> > now (as there are still only 4 releases every 3-month maintenance
>> > window), and I don't understand the concern with "treating releases
>> > differently".
>> 
>> If what becomes an LTS release is known up front, the pressure to
>> get things into such a release may (and, looking at what I got to
>> notice on Linux, will) be quite a bit higher. After all the expectation
>> and intention is for distros to particularly use those releases when
>> caring for long term maintenance. And with that why would
>> contributors bother much about getting things into non-LTS
>> releases when those won't get used by many / for a reasonable
>> time period anyway?
> 
> I disagree. Many developers care about getting their patches upstream
> but are not interested in which releases their features are going to be.

Likely there are both. It's pretty clear to me that developers driven
by product release cycles, and with product releases preferring (and
as was expressed earlier, perhaps even expected to prefer) LTS
releases, focus would be on those releases that end up being used
in products. Of course that's not going to be the exclusive case,
but do you really think a large scale contributor would see much point
in starting work on a feature when they can be sure their code won't
land in any major distro for the next almost 2 years?

>> If what becomes an LTS release is to be determined only at the
>> end of a branch's ordinary life cycle, the above problem can be
>> avoided, but downstream consumers having picked the "wrong"
>> release will be penalized. This is what we've been getting
>> complaints on by openSUSE users relative to the kernel versions
>> (and which keeps coming up the discussion of up-rev-ing the
>> kernel version post release, which as you surely can imagine has
>> all kinds of up and down sides).
> 
> As you mentioned above, there are always upside and downside with *any*
> change. We won't know until we try.
> 
> But what we do know at hand is that current scheme is not perfect, hence
> the whole discussion.

Sure. My main point really is that it's not just the current model that
isn't perfect, yet I'm getting the impression that staying with it has
been (almost) ruled out from the beginning.

>> Plus depending on who gets to
>> maintain the branch from that point on (if not handed to always
>> the same person), quality and hence value to the downstreams
>> may heavily vary.
> 
> This is really irrelevant to this discussion. If we hand that to
> external people, that means it's out of our control and by definition we
> don't care that much -- no matter what scheme we use.

That's a strange reply, considering that there is no strict boundary
between who "represents" XenProject and who does not. Or am I
missing something here?

Jan

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 08.10.15 at 13:49, <JGross@suse.com> wrote:
> On 10/08/2015 01:34 PM, Jan Beulich wrote:
>>>>> On 08.10.15 at 12:59, <Ian.Jackson@eu.citrix.com> wrote:
>>> Jan Beulich writes ("Re: [Xen-devel] RFC: LTS and stable release scheme"):
>>>> Perhaps there's room for further automation here, yet as with
>>>> any automation the question is how quickly getting this in place
>>>> will amortize itself.
>>>
>>> Even with the current situation I think much more automation would be
>>> good.  (But then I'm someone who really (a) likes automating things
>>> (b) likes sitting back and watching the automation do its thing and
>>> even (c) likes debugging the automation when it goes wrong.)
>>>
>>> I think that maybe as a starting point, Jan and I could agree that
>>> instead of build-testing our backports locally, we will throw them at
>>> osstest and see what sticks.
>>
>> Well, yes, we could. Otoh the overhead of fixing something that
>> didn't build but got committed already means more mechanical
>> work (revert, or create a fixup patch) compared to fixing it before
>> pushing to the respective staging tree.
>>
>> What I would see as possibly useful would be a queue like thing
>> where backports could be added, and automation would take
>> care of committing and pushing as much of it as it can validate
>> to build (more severe problems are pretty rare in stable trees,
>> and hence relying on the normal osstest there like we do now
>> would seem reasonable). Yet again this would mean one may
>> have to turn attention to the respective tree more often (since
>> right now this is needed just once for each batch of backports,
>> unless something really odd happens).
> 
> Couldn't that purely mechanical work be spread to others? I can't
> believe this would require exceptional skills and I think your
> time is to precious for stuff like that.
> 
> In the beginning the workflow could be the same as yours today,
> there would be just the queue you mentioned and someone either
> doing the builds and committing or just look after the results
> of any automatism. It just wouldn't be you.

I really dislike considering my time more precious than that of
other people. Hence I'm rather hesitant to push work onto
others, albeit I've learned that I can't do entirely without (but 
then on the basis of them being more knowledgeable about
things or it really being their responsibility, not their time being
less valuable).

Jan

Re: RFC: LTS and stable release scheme

[Juergen Gross]

On 10/08/2015 02:22 PM, Jan Beulich wrote:
>>>> On 08.10.15 at 13:49, <JGross@suse.com> wrote:
>> On 10/08/2015 01:34 PM, Jan Beulich wrote:
>>>>>> On 08.10.15 at 12:59, <Ian.Jackson@eu.citrix.com> wrote:
>>>> Jan Beulich writes ("Re: [Xen-devel] RFC: LTS and stable release scheme"):
>>>>> Perhaps there's room for further automation here, yet as with
>>>>> any automation the question is how quickly getting this in place
>>>>> will amortize itself.
>>>>
>>>> Even with the current situation I think much more automation would be
>>>> good.  (But then I'm someone who really (a) likes automating things
>>>> (b) likes sitting back and watching the automation do its thing and
>>>> even (c) likes debugging the automation when it goes wrong.)
>>>>
>>>> I think that maybe as a starting point, Jan and I could agree that
>>>> instead of build-testing our backports locally, we will throw them at
>>>> osstest and see what sticks.
>>>
>>> Well, yes, we could. Otoh the overhead of fixing something that
>>> didn't build but got committed already means more mechanical
>>> work (revert, or create a fixup patch) compared to fixing it before
>>> pushing to the respective staging tree.
>>>
>>> What I would see as possibly useful would be a queue like thing
>>> where backports could be added, and automation would take
>>> care of committing and pushing as much of it as it can validate
>>> to build (more severe problems are pretty rare in stable trees,
>>> and hence relying on the normal osstest there like we do now
>>> would seem reasonable). Yet again this would mean one may
>>> have to turn attention to the respective tree more often (since
>>> right now this is needed just once for each batch of backports,
>>> unless something really odd happens).
>>
>> Couldn't that purely mechanical work be spread to others? I can't
>> believe this would require exceptional skills and I think your
>> time is to precious for stuff like that.
>>
>> In the beginning the workflow could be the same as yours today,
>> there would be just the queue you mentioned and someone either
>> doing the builds and committing or just look after the results
>> of any automatism. It just wouldn't be you.
>
> I really dislike considering my time more precious than that of
> other people. Hence I'm rather hesitant to push work onto
> others, albeit I've learned that I can't do entirely without (but
> then on the basis of them being more knowledgeable about
> things or it really being their responsibility, not their time being
> less valuable).

Okay, let me rephrase this:

You are already doing quite a lot for the Xen project (committer, x86
maintainer, a huge amount of reviews) resulting in your time being
available to productive topics seems to be of higher priority than
not spreading more or less mechanical work to others. I can
understand you are feeling a little bit uneasy letting others do this
maybe even dumb work (no offence), but I hope there would be someone
volunteering for that task. If not, this discussion is moot, of course.

You can put it this way: you are seeing a problem with a shorter release
cycle due to the suspected higher workload required doing purely
mechanical work. Maybe the desire for a shorter release cycle is so high
that someone steps up and says: "hey, no problem, let me do that purely
mechanical work, so your problem isn't existing any more."


Juergen

Re: RFC: LTS and stable release scheme

[Wei Liu]

On Thu, Oct 08, 2015 at 02:39:40PM +0200, Juergen Gross wrote:
> On 10/08/2015 02:22 PM, Jan Beulich wrote:
> >>>>On 08.10.15 at 13:49, <JGross@suse.com> wrote:
> >>On 10/08/2015 01:34 PM, Jan Beulich wrote:
> >>>>>>On 08.10.15 at 12:59, <Ian.Jackson@eu.citrix.com> wrote:
> >>>>Jan Beulich writes ("Re: [Xen-devel] RFC: LTS and stable release scheme"):
> >>>>>Perhaps there's room for further automation here, yet as with
> >>>>>any automation the question is how quickly getting this in place
> >>>>>will amortize itself.
> >>>>
> >>>>Even with the current situation I think much more automation would be
> >>>>good.  (But then I'm someone who really (a) likes automating things
> >>>>(b) likes sitting back and watching the automation do its thing and
> >>>>even (c) likes debugging the automation when it goes wrong.)
> >>>>
> >>>>I think that maybe as a starting point, Jan and I could agree that
> >>>>instead of build-testing our backports locally, we will throw them at
> >>>>osstest and see what sticks.
> >>>
> >>>Well, yes, we could. Otoh the overhead of fixing something that
> >>>didn't build but got committed already means more mechanical
> >>>work (revert, or create a fixup patch) compared to fixing it before
> >>>pushing to the respective staging tree.
> >>>
> >>>What I would see as possibly useful would be a queue like thing
> >>>where backports could be added, and automation would take
> >>>care of committing and pushing as much of it as it can validate
> >>>to build (more severe problems are pretty rare in stable trees,
> >>>and hence relying on the normal osstest there like we do now
> >>>would seem reasonable). Yet again this would mean one may
> >>>have to turn attention to the respective tree more often (since
> >>>right now this is needed just once for each batch of backports,
> >>>unless something really odd happens).
> >>
> >>Couldn't that purely mechanical work be spread to others? I can't
> >>believe this would require exceptional skills and I think your
> >>time is to precious for stuff like that.
> >>
> >>In the beginning the workflow could be the same as yours today,
> >>there would be just the queue you mentioned and someone either
> >>doing the builds and committing or just look after the results
> >>of any automatism. It just wouldn't be you.
> >
> >I really dislike considering my time more precious than that of
> >other people. Hence I'm rather hesitant to push work onto
> >others, albeit I've learned that I can't do entirely without (but
> >then on the basis of them being more knowledgeable about
> >things or it really being their responsibility, not their time being
> >less valuable).
> 
> Okay, let me rephrase this:
> 
> You are already doing quite a lot for the Xen project (committer, x86
> maintainer, a huge amount of reviews) resulting in your time being
> available to productive topics seems to be of higher priority than
> not spreading more or less mechanical work to others. I can
> understand you are feeling a little bit uneasy letting others do this
> maybe even dumb work (no offence), but I hope there would be someone
> volunteering for that task. If not, this discussion is moot, of course.
> 

+1.

I think it's whole community's burden to run a community driven project.
Nobody would think that you push work to others because you think their
time is less valuable than yours.

> You can put it this way: you are seeing a problem with a shorter release
> cycle due to the suspected higher workload required doing purely
> mechanical work. Maybe the desire for a shorter release cycle is so high
> that someone steps up and says: "hey, no problem, let me do that purely
> mechanical work, so your problem isn't existing any more."
> 

+1.

That's why various people mean  team of stable release maintainers,
automation tools and other technical / procedural solutions.

But until we figure out what the real pain points are item by item,
there is no answers to how to solve them.

In any case, I'm all for spreading the burden.

Wei.

> 
> Juergen

Re: RFC: LTS and stable release scheme

[Wei Liu]

On Thu, Oct 08, 2015 at 06:13:27AM -0600, Jan Beulich wrote:
> >>> On 08.10.15 at 13:10, <wei.liu2@citrix.com> wrote:
> > On Thu, Oct 08, 2015 at 02:05:51AM -0600, Jan Beulich wrote:
> >> Perhaps there's room for further automation here, yet as with
> >> any automation the question is how quickly getting this in place
> >> will amortize itself.
> >> 
> > 
> > Building every commit can be easily implemented. I don't think it takes
> > more than a day to write a script.  What do you mean by "in place"? Are
> > there any other considerations?
> 
> No. "In place" simply means this needs to be written and tested.
> And as things stand what will fit Ian may not fit me or vice versa.
> I.e. unless this would be a server side thing (see the queue thing
> I mentioned in another reply), multiple scripts may end up being
> needed. And a day is a day - if it saves, say, 15 minutes per batch,
> it would take 32 batches for the work to amortize.
> 

This is still based on the assumption that only you alone is going to do
the work. If you agree we can solve this by having more helping hand,
I'm sure we can try to call for more people to help -- that's a possible
solution to this issue.

> > But for now, let's say 3 years of security support is good enough. Do
> > you have proposal that would fit 6 months release cycle? A majority of
> > developers seem to think 9 months is too long and 6 months is worth
> > trying.
> 
> And if there is a majority, I'm sure we will try it. I'm not arguing
> because I mean to try to veto any such decision, I'm only trying
> to point out that there also are reasons to stick with the current
> model.
> 
> >> So using the 4 month stable release cadence I get currently 6
> >> stable releases a year (two maintained branches, ignoring short
> >> periods of overlap), compared to 6 ordinary stable releases (two
> >> ordinary branches) plus 6 LTS releases (two LTS branches) a year,
> >> i.e. doubling the amount.
> > 
> > So this seems to be your major concern. My interpretation is that you
> > kept stressing workload because you think it is a major issue. 
> 
> Thing is - I'm not of the opinion that it's impossible to do, but I am
> of the opinion that any amount of time spent on mechanical things
> is worthwhile trying to avoid or at least limit as much as possible. I
> don't know about you, but I prefer to do productive things...
> 

I understand. I, too, prefer productive things over tedious tasks.

> > I fail to get the idea why this would be a problem. Maybe you're seeing
> > every backport as your sole responsibility?  From Xen project's point of
> > view, this amount can be absorbed by a team of stable release
> > maintainers. Then this leads to your concern of varying quality of
> > different branches? But again, I fail to grasp that idea. All stable
> > trees maintained by Xen project are equally good.
> 
> Because right now they're all being managed by the same people.
> The main reason for questioning the quality of some Linux stable
> trees was that the criteria based upon which selection of backports
> happened appeared to be quite different between individual people.
> I'd be much less worried of the quality of the actual backports.
> 

So your point here is that you would like to have all branches manage by
same person so that their quality are all the same because they always
have the same set of backports, or at least all patches are considered
with same standard.

This is understandable but not always applicable. As you noted, when you
backport stuff you also consider the difficulty and risk of backport.
So in the end even if there is only you who manage all the trees they
will still have different sets of backports. I don't think there is
difference in the end result when you have multiple people maintaining
different trees. True, their criteria for deciding if a patch is worth
backporting are different, but we can perhaps write down a guide to
stable tree maintainers to minimise the differences.

> >> > Personally I like the LTS model.  I think arguably if we use the
> >> > numbers I chose above, it shouldn't be any more actual work than it is
> >> > now (as there are still only 4 releases every 3-month maintenance
> >> > window), and I don't understand the concern with "treating releases
> >> > differently".
> >> 
> >> If what becomes an LTS release is known up front, the pressure to
> >> get things into such a release may (and, looking at what I got to
> >> notice on Linux, will) be quite a bit higher. After all the expectation
> >> and intention is for distros to particularly use those releases when
> >> caring for long term maintenance. And with that why would
> >> contributors bother much about getting things into non-LTS
> >> releases when those won't get used by many / for a reasonable
> >> time period anyway?
> > 
> > I disagree. Many developers care about getting their patches upstream
> > but are not interested in which releases their features are going to be.
> 
> Likely there are both. It's pretty clear to me that developers driven
> by product release cycles, and with product releases preferring (and
> as was expressed earlier, perhaps even expected to prefer) LTS
> releases, focus would be on those releases that end up being used
> in products. Of course that's not going to be the exclusive case,
> but do you really think a large scale contributor would see much point
> in starting work on a feature when they can be sure their code won't
> land in any major distro for the next almost 2 years?
> 

I can't say for sure. There is no scientific survey to determine that.

But then the reverse argument applies to longer release cycle, why would
you develop your feature for a project knowing that it would definitely
not be available at all for 9 months and even longer (if its release is
delayed) -- you would miss precious window to release you hardware /
software platform.

Again, I'm not dismissing your argument. But my empirical observation
suggests that some developers just want their features to in _some_
version.

> >> If what becomes an LTS release is to be determined only at the
> >> end of a branch's ordinary life cycle, the above problem can be
> >> avoided, but downstream consumers having picked the "wrong"
> >> release will be penalized. This is what we've been getting
> >> complaints on by openSUSE users relative to the kernel versions
> >> (and which keeps coming up the discussion of up-rev-ing the
> >> kernel version post release, which as you surely can imagine has
> >> all kinds of up and down sides).
> > 
> > As you mentioned above, there are always upside and downside with *any*
> > change. We won't know until we try.
> > 
> > But what we do know at hand is that current scheme is not perfect, hence
> > the whole discussion.
> 
> Sure. My main point really is that it's not just the current model that
> isn't perfect, yet I'm getting the impression that staying with it has
> been (almost) ruled out from the beginning.
> 

No. That was not my intent.

It's just my misinterpretation of constraints. Sorry about that.

> >> Plus depending on who gets to
> >> maintain the branch from that point on (if not handed to always
> >> the same person), quality and hence value to the downstreams
> >> may heavily vary.
> > 
> > This is really irrelevant to this discussion. If we hand that to
> > external people, that means it's out of our control and by definition we
> > don't care that much -- no matter what scheme we use.
> 
> That's a strange reply, considering that there is no strict boundary
> between who "represents" XenProject and who does not. Or am I
> missing something here?
> 

So what do you mean by "from that point on"? I thought it mean "after
retirement of a particular branch" -- retiring a branch means we don't
care that much about it anymore. That happens no matter what scheme we
use.

Wei.

> Jan

Re: RFC: LTS and stable release scheme

[Jan Beulich]

>>> On 08.10.15 at 16:23, <wei.liu2@citrix.com> wrote:
> On Thu, Oct 08, 2015 at 06:13:27AM -0600, Jan Beulich wrote:
>> >>> On 08.10.15 at 13:10, <wei.liu2@citrix.com> wrote:
>> > I fail to get the idea why this would be a problem. Maybe you're seeing
>> > every backport as your sole responsibility?  From Xen project's point of
>> > view, this amount can be absorbed by a team of stable release
>> > maintainers. Then this leads to your concern of varying quality of
>> > different branches? But again, I fail to grasp that idea. All stable
>> > trees maintained by Xen project are equally good.
>> 
>> Because right now they're all being managed by the same people.
>> The main reason for questioning the quality of some Linux stable
>> trees was that the criteria based upon which selection of backports
>> happened appeared to be quite different between individual people.
>> I'd be much less worried of the quality of the actual backports.
>> 
> 
> So your point here is that you would like to have all branches manage by
> same person so that their quality are all the same because they always
> have the same set of backports, or at least all patches are considered
> with same standard.
> 
> This is understandable but not always applicable. As you noted, when you
> backport stuff you also consider the difficulty and risk of backport.
> So in the end even if there is only you who manage all the trees they
> will still have different sets of backports.

Yes and no. The main criteria for me is that 4.<n>.x should never have
a fix that 4.<n+1>.y doesn't have. I.e. (quite naturally I would say) if
I decide to drop a patch due to being too difficult to backport, I'll surely
not re-introduce it on an older branch. Maintaining all branches this is
actually easily enforced: When doing backports, I (again quite naturally)
do them in reverse order of releases, and hence once a patch got
dropped it won't re-appear going further backwards.

And yes, to preempt your respective argument, this can be documented
and people can be told to obey to this rule. But there are cases where
I push back certain desirable, but more risky backports, with the plan to
perhaps apply them once they've seen more testing on unstable or even
in a release. This is quite a bit more cumbersome (but not impossible,
agreed) to coordinate.

> I don't think there is
> difference in the end result when you have multiple people maintaining
> different trees. True, their criteria for deciding if a patch is worth
> backporting are different, but we can perhaps write down a guide to
> stable tree maintainers to minimise the differences.

I'm sure Linux has criteria too. But you know what - people tend to
ignore such when they think they know better (and I can't exclude
myself here).

>> >> Plus depending on who gets to
>> >> maintain the branch from that point on (if not handed to always
>> >> the same person), quality and hence value to the downstreams
>> >> may heavily vary.
>> > 
>> > This is really irrelevant to this discussion. If we hand that to
>> > external people, that means it's out of our control and by definition we
>> > don't care that much -- no matter what scheme we use.
>> 
>> That's a strange reply, considering that there is no strict boundary
>> between who "represents" XenProject and who does not. Or am I
>> missing something here?
> 
> So what do you mean by "from that point on"? I thought it mean "after
> retirement of a particular branch" -- retiring a branch means we don't
> care that much about it anymore. That happens no matter what scheme we
> use.

Oh, sorry, "from that point on" was meant to refer to the transition
from ordinary to LTS maintenance (at which point a transfer of
responsibilities seems quite reasonable a possibility).

Jan