[RFC] -stable, how it's going to work.

[RFC] -stable, how it's going to work.

[Greg KH]

So here's a first cut at how this 2.6 -stable release process is going
to work that Chris and I have come up with.  Does anyone have any
problems/issues/questions with this?

thanks,

greg k-h

-------------------

Everything you ever wanted to know about Linux 2.6 -stable releases.


Rules on what kind of patches are accepted, and what ones are not, into
the "-stable" tree:
 - It must be obviously correct and tested.
 - It can not bigger than 100 lines, with context.
 - It must fix only one thing.
 - It must fix a real bug that bothers people (not a, "This could be a
   problem..." type thing.)
 - It must fix a problem that causes a build error (but not for things
   marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
   security issue, or some "oh, that's not good" issue.  In short,
   something critical.
 - No "theoretical race condition" issues, unless an explanation of how
   the race can be exploited.
 - It can not contain any "trivial" fixes in it (spelling changes,
   whitespace cleanups, etc.)
 - It must be accepted by the relevant subsystem maintainer.
 - It must follow Documentation/SubmittingPatches rules.

Procedure for submitting patches to the -stable tree:
 - Send the patch, after verifying that it follows the above rules, to
   stable@kernel.org.
 - The sender will receive an ack when the patch has been accepted into
   the queue, or a nak if the patch is rejected.  This response might
   take a few days, according to the developer's schedules.
 - If accepted, the patch will be added to the -stable queue, for review
   by other developers.
 - Security patches should not be sent to this alias, but instead to the
   documented security@kernel.org.  
   
Review cycle:
 - When the -stable maintainers decide for a review cycle, the patches
   will be sent to the review committee, and the maintainer of the
   affected area of the patch (unless the submitter is the maintainer of
   the area) and CC: to the linux-kernel mailing list.
 - The review committee has 48 hours in which to ack or nak the patch.
 - If the patch is rejected by a member of the committee, or linux-kernel
   members object to the patch by bringing up issues that the maintainer
   and members did not realize, the patch will be dropped from the
   queue.
 - At the end of the review cycle, the acked patches will be added to
   the latest -stable release, and a new -stable release will happen.
 - Security patches will be accepted into the -stable tree directly from
   the security kernel team, and not go through the normal review cycle.
   Contact the kernel security team for more details on this procedure.

Review committe:
 - This will be made up of a number of kernel developers who have
   volunteered for this task, and a few that haven't.

Re: [RFC] -stable, how it's going to work.

[Andi Kleen]

Greg KH <greg@kroah.com> writes:
>
> Rules on what kind of patches are accepted, and what ones are not, into
> the "-stable" tree:
>  - It must be obviously correct and tested.
>  - It can not bigger than 100 lines, with context.

This rule seems silly. What happens when a security fix needs 150 lines? 

Better maybe a rule like "The patch should be the minimal and safest 
change to fix an issue". But see below for an exception.

>  - It must fix only one thing.
>  - It must fix a real bug that bothers people (not a, "This could be a
>    problem..." type thing.)
>  - It must fix a problem that causes a build error (but not for things
>    marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
>    security issue, or some "oh, that's not good" issue.  In short,
>    something critical.
>  - No "theoretical race condition" issues, unless an explanation of how
>    the race can be exploited.
>  - It can not contain any "trivial" fixes in it (spelling changes,
>    whitespace cleanups, etc.)
>  - It must be accepted by the relevant subsystem maintainer.



>  - It must follow Documentation/SubmittingPatches rules.

One rule I'm missing:

- It must be accepted to mainline. 

That is what big enterprise distributions often require and I think
it's a good rule. Otherwise you risk code and feature set drift
and we don't want to repeat the 2.4 mistakes again where some 
subsystems had more fixes in 2.4 than 2.6.

Also your rules encourage to do different patches for -stable
(e.g. with less comment changes etc.) than for mainline. I don't
think that's a very good thing. Sometimes it is unavoidable
and sometimes the mainline patches are just too big and intrusive,
but in general it's imho best to apply the same patches
to mainline and backport trees.  This has also the advantage
that the patch is best tested as possible; slimmed down patches
usually have a risk of malfunction.

If a mainline patch violates too many of your other rules
("Fixes one thing; doesn't do cosmetic changes etc.") perhaps
the mainline patch just needs to be improved.

So in general there should be a preference to apply the same
patch as mainline, unless it is very big.

>  - Security patches will be accepted into the -stable tree directly from
>    the security kernel team, and not go through the normal review cycle.
>    Contact the kernel security team for more details on this procedure.

This also sounds like a bad rule. How come the security team has more
competence to review patches than the subsystem maintainers?  I can
see the point of overruling maintainers on security issues when they
are not responsive, but if they are I think the should be still the
main point of contact.

-Andi

Re: [RFC] -stable, how it's going to work.

[Arjan van de Ven]

On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> One rule I'm missing:
> 
> - It must be accepted to mainline. 
> 

I absolutely agree with Andi on this one.


> If a mainline patch violates too many of your other rules
> ("Fixes one thing; doesn't do cosmetic changes etc.") perhaps
> the mainline patch just needs to be improved.
> 

I can see this as getting an exception occasionally, but it should be a
well thought out exception and not a general rule

> >  - Security patches will be accepted into the -stable tree directly from
> >    the security kernel team, and not go through the normal review cycle.
> >    Contact the kernel security team for more details on this procedure.
> 
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers?  I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

yeah; the security patch is public anyway, so why not have the regular
review on it as well? Why would such a patch be special?

Re: [RFC] -stable, how it's going to work.

[Russell King]

On Wed, Mar 09, 2005 at 11:10:59AM +0100, Arjan van de Ven wrote:
> On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> > One rule I'm missing:
> > 
> > - It must be accepted to mainline. 
> > 
> 
> I absolutely agree with Andi on this one.

What about the case (as highlighted in previous discussions) that the
stable tree needs a simple "dirty" fix, whereas mainline takes the
complex "clean" fix?

>From what's been said above, this means that the stable tree doesn't
get the simple "dirty" fix, because that isn't what's been accepted
into mainline.  However, it could take the complex "clean" fix, but
that may have too high an impact.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: [RFC] -stable, how it's going to work.

[Arjan van de Ven]

On Wed, 2005-03-09 at 10:17 +0000, Russell King wrote:
> On Wed, Mar 09, 2005 at 11:10:59AM +0100, Arjan van de Ven wrote:
> > On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> > > One rule I'm missing:
> > > 
> > > - It must be accepted to mainline. 
> > > 
> > 
> > I absolutely agree with Andi on this one.
> 
> What about the case (as highlighted in previous discussions) that the
> stable tree needs a simple "dirty" fix, whereas mainline takes the
> complex "clean" fix?

that's the exception I talked about a bit later in my mail. It should be
rare and very deliberate. And in fact once the mainline change ripples
out into maturity I rather replace the -stable one with that later on,
even if it's a bit more invasive. 

Re: [RFC] -stable, how it's going to work.

[Andi Kleen]

On Wed, Mar 09, 2005 at 10:17:28AM +0000, Russell King wrote:
> On Wed, Mar 09, 2005 at 11:10:59AM +0100, Arjan van de Ven wrote:
> > On Wed, 2005-03-09 at 10:56 +0100, Andi Kleen wrote:
> > > One rule I'm missing:
> > > 
> > > - It must be accepted to mainline. 
> > > 
> > 
> > I absolutely agree with Andi on this one.
> 
> What about the case (as highlighted in previous discussions) that the
> stable tree needs a simple "dirty" fix, whereas mainline takes the
> complex "clean" fix?

That's ok, as long as the maintainers agree it's an equivalent fix

What should just be ruled out is something getting fixed in stable
and not getting fixed in mainline. And the mainline patch should
always go in first and preferably tested for a short time there.

But in general it is a judgement call: if the "big fix" is not too big
or risky I would prefer the big fix just to avoid code drift.
A custom fix for stable should be more the exception than the rule.

-Andi

Re: [RFC] -stable, how it's going to work.

[Russell King]

On Wed, Mar 09, 2005 at 11:24:58AM +0100, Arjan van de Ven wrote:
> On Wed, 2005-03-09 at 10:17 +0000, Russell King wrote:
> > What about the case (as highlighted in previous discussions) that the
> > stable tree needs a simple "dirty" fix, whereas mainline takes the
> > complex "clean" fix?
> 
> that's the exception I talked about a bit later in my mail. It should be
> rare and very deliberate. And in fact once the mainline change ripples
> out into maturity I rather replace the -stable one with that later on,
> even if it's a bit more invasive. 

Is that really necessary with a stable tree which may only be around
for about 2 months before the next stable tree is forked (which would
have the mature mainline fix in) ?

There is another point here though, which I think is much more important.
Remember that the original issue which caused the -stable tree to be
created was a concern over the testing that Linus' kernels were getting.
Also, realise that by creating a -stable tree, we haven't increased the
number of testers which Linus' kernels are seeing.

Given that, how can we decide that a complex fix has matured enough
in Linus' kernel to warrant replacing a (proven) fix which users are
perfectly happy with in the corresponding -stable tree?

I thought the -stable tree is targeted towards stability, not towards
"lets replace this change with some other because we as developers think
it's better".

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: [RFC] -stable, how it's going to work.

[Marcelo Tosatti]

> >  - Security patches will be accepted into the -stable tree directly from
> >    the security kernel team, and not go through the normal review cycle.
> >    Contact the kernel security team for more details on this procedure.
> 
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers?  I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

The security team is going to work with the subsystem maintainers,
not overrule them.

That would be indeed insane.

Re: [RFC] -stable, how it's going to work.

[Alan Cox]

On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> - It must be accepted to mainline. 

Strongly disagree. What if the mainline fix is a rewrite of the core API
involved. Some times you need to put in the short term fix. What must
never happen is people accepting that fix as long term.

How about

 - It must be accepted to mainline, or the accepted mainline patch be
deemed too complex or risky to backport and thus a simple obvious
alternative fix applied to stable ONLY.

Re: [RFC] -stable, how it's going to work.

[Chris Wright]

* Andi Kleen (ak@muc.de) wrote:
> Greg KH <greg@kroah.com> writes:
> >
> > Rules on what kind of patches are accepted, and what ones are not, into
> > the "-stable" tree:
> >  - It must be obviously correct and tested.
> >  - It can not bigger than 100 lines, with context.
> 
> This rule seems silly. What happens when a security fix needs 150 lines? 
> 
> Better maybe a rule like "The patch should be the minimal and safest 
> change to fix an issue". But see below for an exception.

It's just a guideline to scope the work.  But a fixed size is probably
less meaningful than your wording.

> >  - It must fix only one thing.
> >  - It must fix a real bug that bothers people (not a, "This could be a
> >    problem..." type thing.)
> >  - It must fix a problem that causes a build error (but not for things
> >    marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
> >    security issue, or some "oh, that's not good" issue.  In short,
> >    something critical.
> >  - No "theoretical race condition" issues, unless an explanation of how
> >    the race can be exploited.
> >  - It can not contain any "trivial" fixes in it (spelling changes,
> >    whitespace cleanups, etc.)
> >  - It must be accepted by the relevant subsystem maintainer.
> 
> >  - It must follow Documentation/SubmittingPatches rules.
> 
> One rule I'm missing:
> 
> - It must be accepted to mainline. 

This can violate the principle of keeping fixes simple for -stable tree.
And Linus/Andrew don't want to litter mainline with patch series that
do simple fix followed by complete fix meant for developement branch.

> That is what big enterprise distributions often require and I think
> it's a good rule. Otherwise you risk code and feature set drift
> and we don't want to repeat the 2.4 mistakes again where some 
> subsystems had more fixes in 2.4 than 2.6.

I agree, it's a good rule, but these should be small, temporal diffs
from mainline.  For example, -ac tree will sometimes do the simpler fix,
whereas mainline does proper complete fix.

> Also your rules encourage to do different patches for -stable
> (e.g. with less comment changes etc.) than for mainline. I don't
> think that's a very good thing. Sometimes it is unavoidable
> and sometimes the mainline patches are just too big and intrusive,
> but in general it's imho best to apply the same patches
> to mainline and backport trees.  This has also the advantage
> that the patch is best tested as possible; slimmed down patches
> usually have a risk of malfunction.
> 
> If a mainline patch violates too many of your other rules
> ("Fixes one thing; doesn't do cosmetic changes etc.") perhaps
> the mainline patch just needs to be improved.

Good point.

> So in general there should be a preference to apply the same
> patch as mainline, unless it is very big.

Agreed.

> >  - Security patches will be accepted into the -stable tree directly from
> >    the security kernel team, and not go through the normal review cycle.
> >    Contact the kernel security team for more details on this procedure.
> 
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers?  I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

They don't, the security patches should still be reviewed by subsystem
maintainer.  Point here is, sometimes there's disclosure coordination
happening as well.

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: [RFC] -stable, how it's going to work.

[Greg KH]

On Wed, Mar 09, 2005 at 06:00:45PM +0000, Alan Cox wrote:
> On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> > - It must be accepted to mainline. 
> 
> Strongly disagree. What if the mainline fix is a rewrite of the core API
> involved. Some times you need to put in the short term fix. What must
> never happen is people accepting that fix as long term.
> 
> How about
> 
>  - It must be accepted to mainline, or the accepted mainline patch be
> deemed too complex or risky to backport and thus a simple obvious
> alternative fix applied to stable ONLY.

Yes, that's acceptable, I agree that Andi's statment is not ok (see the
.1 release for 2 patches that were accepted in a "simple" way because
the real fix was too complex.)

I'll go update the document with this.

thanks,

greg k-h

Re: [RFC] -stable, how it's going to work.

[Chris Wright]

* Alan Cox (alan@lxorguk.ukuu.org.uk) wrote:
> On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> > - It must be accepted to mainline. 
> 
> Strongly disagree. What if the mainline fix is a rewrite of the core API
> involved. Some times you need to put in the short term fix. What must
> never happen is people accepting that fix as long term.
> 
> How about
> 
>  - It must be accepted to mainline, or the accepted mainline patch be
> deemed too complex or risky to backport and thus a simple obvious
> alternative fix applied to stable ONLY.

Yes.

Re: [RFC] -stable, how it's going to work.

[Greg KH]

On Wed, Mar 09, 2005 at 10:56:33AM +0100, Andi Kleen wrote:
> Greg KH <greg@kroah.com> writes:
> >
> > Rules on what kind of patches are accepted, and what ones are not, into
> > the "-stable" tree:
> >  - It must be obviously correct and tested.
> >  - It can not bigger than 100 lines, with context.
> 
> This rule seems silly. What happens when a security fix needs 150 lines? 

Then we bend the rules and accept it :)

We'll take these as a case-by-case basis...

> >  - Security patches will be accepted into the -stable tree directly from
> >    the security kernel team, and not go through the normal review cycle.
> >    Contact the kernel security team for more details on this procedure.
> 
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers?  I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

Security fixes go from the security team to Linus's tree directly, and
usually the subsystem maintainer has already been notified and has
reviewedit.  At that point in time, they are public and accepted into
mainline, and need to be made availble to the -stable users as soon as
possible.

That is why the "fast track" is going to happen, the patch really was
reviewed properly, just not in public :)

thanks,

greg k-h

Re: [RFC] -stable, how it's going to work.

[Andi Kleen]

On Wed, Mar 09, 2005 at 06:00:45PM +0000, Alan Cox wrote:
> On Mer, 2005-03-09 at 09:56, Andi Kleen wrote:
> > - It must be accepted to mainline. 
> 
> Strongly disagree. What if the mainline fix is a rewrite of the core API
> involved. Some times you need to put in the short term fix. What must
> never happen is people accepting that fix as long term.
> 
> How about
> 
>  - It must be accepted to mainline, or the accepted mainline patch be
> deemed too complex or risky to backport and thus a simple obvious
> alternative fix applied to stable ONLY.

That is what I wrote later in my mail anyways (did you really read it completely?:)  See also the followup discussion with Russel and Arjan.

In general stable specific fixes should be the exception, not the rule though.

-Andi

Re: [RFC] -stable, how it's going to work.

[Andi Kleen]

On Wed, Mar 09, 2005 at 10:34:08AM -0800, Greg KH wrote:
> On Wed, Mar 09, 2005 at 10:56:33AM +0100, Andi Kleen wrote:
> > Greg KH <greg@kroah.com> writes:
> > >
> > > Rules on what kind of patches are accepted, and what ones are not, into
> > > the "-stable" tree:
> > >  - It must be obviously correct and tested.
> > >  - It can not bigger than 100 lines, with context.
> > 
> > This rule seems silly. What happens when a security fix needs 150 lines? 
> 
> Then we bend the rules and accept it :)
> 
> We'll take these as a case-by-case basis...
> 
> > >  - Security patches will be accepted into the -stable tree directly from
> > >    the security kernel team, and not go through the normal review cycle.
> > >    Contact the kernel security team for more details on this procedure.
> > 
> > This also sounds like a bad rule. How come the security team has more
> > competence to review patches than the subsystem maintainers?  I can
> > see the point of overruling maintainers on security issues when they
> > are not responsive, but if they are I think the should be still the
> > main point of contact.
> 
> Security fixes go from the security team to Linus's tree directly, and
> usually the subsystem maintainer has already been notified and has
> reviewedit.  At that point in time, they are public and accepted into

What guarantees that?

Basically what I would like to avoid is that the security team
merges something through the backdoor that the maintainer considers crap.

If anything you should have a rule like

"Send to maintainer. If he doesn't ACK in 24h send it directly"


> mainline, and need to be made availble to the -stable users as soon as
> possible.
> 
> That is why the "fast track" is going to happen, the patch really was
> reviewed properly, just not in public :)

Well, when you really want to have such formal rules (which is a novelty in 
Linux space BTW, for many years we did fine with unwritten rules)  then you
should spell it out completely. Or alternatively drop all the formal
rules and do it informally like it was always done.


-Andi

Re: [RFC] -stable, how it's going to work.

[Andi Kleen]

On Wed, Mar 09, 2005 at 10:28:22AM -0800, Chris Wright wrote:
> * Andi Kleen (ak@muc.de) wrote:
> > Greg KH <greg@kroah.com> writes:
> > >
> > > Rules on what kind of patches are accepted, and what ones are not, into
> > > the "-stable" tree:
> > >  - It must be obviously correct and tested.
> > >  - It can not bigger than 100 lines, with context.
> > 
> > This rule seems silly. What happens when a security fix needs 150 lines? 
> > 
> > Better maybe a rule like "The patch should be the minimal and safest 
> > change to fix an issue". But see below for an exception.
> 
> It's just a guideline to scope the work.  But a fixed size is probably
> less meaningful than your wording.
> 
> > >  - It must fix only one thing.
> > >  - It must fix a real bug that bothers people (not a, "This could be a
> > >    problem..." type thing.)
> > >  - It must fix a problem that causes a build error (but not for things
> > >    marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
> > >    security issue, or some "oh, that's not good" issue.  In short,
> > >    something critical.
> > >  - No "theoretical race condition" issues, unless an explanation of how
> > >    the race can be exploited.
> > >  - It can not contain any "trivial" fixes in it (spelling changes,
> > >    whitespace cleanups, etc.)
> > >  - It must be accepted by the relevant subsystem maintainer.
> > 
> > >  - It must follow Documentation/SubmittingPatches rules.
> > 
> > One rule I'm missing:
> > 
> > - It must be accepted to mainline. 
> 
> This can violate the principle of keeping fixes simple for -stable tree.
> And Linus/Andrew don't want to litter mainline with patch series that
> do simple fix followed by complete fix meant for developement branch.

But it risks code drift like we had in 2.4 with older kernels 
having more fixes than the newer kernel. And that way lies madness.

I think it is very very important to avoid this.

If you prefer you can rewrite the rule like

"Fix must in mainline first. In exceptional cases when the fix 
in mainline is too intrusive or risky a simpler version of the patch
can be applied to stable. In this case the mainline fix must be already
accepted. For most cases the full fix should be applied to avoid code drift"


> I agree, it's a good rule, but these should be small, temporal diffs
> from mainline.  For example, -ac tree will sometimes do the simpler fix,
> whereas mainline does proper complete fix.

You make it sound like all patches are super complicated and 
not suitable for backporting.

>From my experiences maintaining distribution kernel most mainline changes
can be just completely backported. 

> 
> > >    the security kernel team, and not go through the normal review cycle.
> > >    Contact the kernel security team for more details on this procedure.
> > 
> > This also sounds like a bad rule. How come the security team has more
> > competence to review patches than the subsystem maintainers?  I can
> > see the point of overruling maintainers on security issues when they
> > are not responsive, but if they are I think the should be still the
> > main point of contact.
> 
> They don't, the security patches should still be reviewed by subsystem
> maintainer.  Point here is, sometimes there's disclosure coordination
> happening as well.

Ok, how does it coordinate with the vendor-sec process? 
And at what point is the subsystem maintainer notified.

The security thing seems to be still quite half backed to me...

-Andi

Re: [RFC] -stable, how it's going to work.

[Greg KH]

On Wed, Mar 09, 2005 at 08:39:36PM +0100, Andi Kleen wrote:
> On Wed, Mar 09, 2005 at 10:34:08AM -0800, Greg KH wrote:
> > On Wed, Mar 09, 2005 at 10:56:33AM +0100, Andi Kleen wrote:
> > > Greg KH <greg@kroah.com> writes:
> > > >
> > > > Rules on what kind of patches are accepted, and what ones are not, into
> > > > the "-stable" tree:
> > > >  - It must be obviously correct and tested.
> > > >  - It can not bigger than 100 lines, with context.
> > > 
> > > This rule seems silly. What happens when a security fix needs 150 lines? 
> > 
> > Then we bend the rules and accept it :)
> > 
> > We'll take these as a case-by-case basis...
> > 
> > > >  - Security patches will be accepted into the -stable tree directly from
> > > >    the security kernel team, and not go through the normal review cycle.
> > > >    Contact the kernel security team for more details on this procedure.
> > > 
> > > This also sounds like a bad rule. How come the security team has more
> > > competence to review patches than the subsystem maintainers?  I can
> > > see the point of overruling maintainers on security issues when they
> > > are not responsive, but if they are I think the should be still the
> > > main point of contact.
> > 
> > Security fixes go from the security team to Linus's tree directly, and
> > usually the subsystem maintainer has already been notified and has
> > reviewedit.  At that point in time, they are public and accepted into
> 
> What guarantees that?

The kernel security team's proceedures.

> Basically what I would like to avoid is that the security team
> merges something through the backdoor that the maintainer considers crap.
> 
> If anything you should have a rule like
> 
> "Send to maintainer. If he doesn't ACK in 24h send it directly"
> 
> 
> > mainline, and need to be made availble to the -stable users as soon as
> > possible.
> > 
> > That is why the "fast track" is going to happen, the patch really was
> > reviewed properly, just not in public :)
> 
> Well, when you really want to have such formal rules (which is a novelty in 
> Linux space BTW, for many years we did fine with unwritten rules)  then you
> should spell it out completely. Or alternatively drop all the formal
> rules and do it informally like it was always done.

I'd love to do it informally, but the rules are going to be used to make
our lives easier, by having something to point to when we want to reject
something, and having something that everyone can refer to when trying
to understand what it is we are attempting to do here.

If they get too complex, or large, we will have to revisit them.

So, let's stop arguing about the semantics of the rules, and see if what
we have proposed actually works in real-life.  If that doesn't work out,
we can revisit it then.

thanks,

greg k-h

Re: [RFC] -stable, how it's going to work.

[Chris Wright]

* Andi Kleen (ak@muc.de) wrote:
> On Wed, Mar 09, 2005 at 10:28:22AM -0800, Chris Wright wrote:
> > * Andi Kleen (ak@muc.de) wrote:
> > > Greg KH <greg@kroah.com> writes:

> > > One rule I'm missing:
> > > 
> > > - It must be accepted to mainline. 
> > 
> > This can violate the principle of keeping fixes simple for -stable tree.
> > And Linus/Andrew don't want to litter mainline with patch series that
> > do simple fix followed by complete fix meant for developement branch.
> 
> But it risks code drift like we had in 2.4 with older kernels 
> having more fixes than the newer kernel. And that way lies madness.
> 
> I think it is very very important to avoid this.
> 
> If you prefer you can rewrite the rule like
> 
> "Fix must in mainline first. In exceptional cases when the fix 
> in mainline is too intrusive or risky a simpler version of the patch
> can be applied to stable. In this case the mainline fix must be already
> accepted. For most cases the full fix should be applied to avoid code drift"

I think we've all agreed that's the intention.

> > I agree, it's a good rule, but these should be small, temporal diffs
> > from mainline.  For example, -ac tree will sometimes do the simpler fix,
> > whereas mainline does proper complete fix.
> 
> You make it sound like all patches are super complicated and 
> not suitable for backporting.

I didn't think I did, that's why I said 'sometimes'.  Just acknowledging
what does really happen.

> > They don't, the security patches should still be reviewed by subsystem
> > maintainer.  Point here is, sometimes there's disclosure coordination
> > happening as well.
> 
> Ok, how does it coordinate with the vendor-sec process? 
> And at what point is the subsystem maintainer notified.

That's part of the vendor coordination mentioned in the policy.  And
subsystem maintainer is notified as part of vetting the issue/solution,
as stated in the policy.

> The security thing seems to be still quite half backed to me...

Take a look at the policy I posted last night and give me suggestions
for improvements.

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: [RFC] -stable, how it's going to work.

[Bill Davidsen]

http://localhost/blogAndi Kleen wrote:
> Greg KH <greg@kroah.com> writes:
> 
>>Rules on what kind of patches are accepted, and what ones are not, into
>>the "-stable" tree:
>> - It must be obviously correct and tested.
>> - It can not bigger than 100 lines, with context.
> 
> 
> This rule seems silly. What happens when a security fix needs 150 lines? 
> 
> Better maybe a rule like "The patch should be the minimal and safest 
 > change to fix an issue". But see below for an exception.

I was willing to accept that there might be an exploitable security 
issue taking more than 100 lines to plug the hole (remember, check your 
elegance at the door), and I like your suggestion regardless of length. 
But I don't think any more exceptions are desirable. As you said, "see 
below."
> 
>> - It must fix only one thing.
>> - It must fix a real bug that bothers people (not a, "This could be a
>>   problem..." type thing.)
>> - It must fix a problem that causes a build error (but not for things
>>   marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
>>   security issue, or some "oh, that's not good" issue.  In short,
>>   something critical.
>> - No "theoretical race condition" issues, unless an explanation of how
>>   the race can be exploited.
>> - It can not contain any "trivial" fixes in it (spelling changes,
>>   whitespace cleanups, etc.)
>> - It must be accepted by the relevant subsystem maintainer.

I have the feeling that drivers will need some special consideration, 
like allowing additions to an existing blacklist to prevent "almost 
working" with similar but not identical chips and the like.

> One rule I'm missing:
> 
> - It must be accepted to mainline. 
> 
> That is what big enterprise distributions often require and I think
> it's a good rule. Otherwise you risk code and feature set drift
> and we don't want to repeat the 2.4 mistakes again where some 
> subsystems had more fixes in 2.4 than 2.6.

Right problem, wrong solution. What you is that SOME solution go in 
mainline. My example above of adding a chipset to a blacklist is a 
perfect case, where you want the chipset to become supported, not long 
term "officially unsupported." And band-aids like using the BKL or long 
linear searches just to prevent something which causes a hard hang or 
oops should not be blessed and included in mainline.

I know you understand this, I just think it's worth saying more clearly.
> 
> Also your rules encourage to do different patches for -stable
> (e.g. with less comment changes etc.) than for mainline. I don't
> think that's a very good thing. Sometimes it is unavoidable
> and sometimes the mainline patches are just too big and intrusive,
> but in general it's imho best to apply the same patches
> to mainline and backport trees.  This has also the advantage
> that the patch is best tested as possible; slimmed down patches
> usually have a risk of malfunction.

Different in function as well as style. Testing complex patches can be 
done in -mm, reliable is often easier than optimal.

> So in general there should be a preference to apply the same
> patch as mainline, unless it is very big.

I can't imagine anyone not using a good patch in mainline, I can imagine 
some really brute force patches in stable, because they are easier to 
vat than a really correct solution.
> 
> 
>> - Security patches will be accepted into the -stable tree directly from
>>   the security kernel team, and not go through the normal review cycle.
>>   Contact the kernel security team for more details on this procedure.
> 
> 
> This also sounds like a bad rule. How come the security team has more
> competence to review patches than the subsystem maintainers?  I can
> see the point of overruling maintainers on security issues when they
> are not responsive, but if they are I think the should be still the
> main point of contact.

I was actually thinking that the normal patch might want a fast path in 
practice. Some things are clearly wrong, like off by one errors in array 
handling and the like, using pointers to freed data or which could be 
null, that kind of thing is pretty unsubtle, and I would think would 
need fewer eyes-on. But until we see how this works in practice, let's 
not fix it. The stable kernel should have a stable process, which should 
not be changed unless there's a demonstrated problem.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: [RFC] -stable, how it's going to work.

[Russell King]

On Wed, Mar 09, 2005 at 08:44:01PM +0100, Andi Kleen wrote:
> But it risks code drift like we had in 2.4 with older kernels 
> having more fixes than the newer kernel. And that way lies madness.

I believe it's going to work like this:

* simple fixes are submitted to Linus and -stable, and are appropriately
  merged.
* complex "correct" fixes too large for -stable are submitted to Linus,
  with a simplified version for -stable.

When the next Linus kernel is released, anything in the previous -stable
series is effectively discarded, and the next -stable series is produced
from the new release point.

Obviously, the -stable sucker can continue with his existing -stable
series if he so wishes, but I would expect that any fixes in, eg,
2.6.11.x would not be propagated by the -stable sucker to 2.6.12.x.

This may be a good thing - it encourages people to get the fixes into
the Linus tree, thereby keeping the code drift to a minimum.  Any
drift would only exist for one of these -stable branches, which may
only survive for maybe one Linus kernel release cycle.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: [RFC] -stable, how it's going to work.

[Neil Brown]

On Tuesday March 8, greg@kroah.com wrote:
> So here's a first cut at how this 2.6 -stable release process is going
> to work that Chris and I have come up with.  Does anyone have any
> problems/issues/questions with this?

One rule that I thought would make sense, but that I don't see listed
is:

 - must fix a regression

If some problem was in 2.6.X and is still there in 2.6.X+1, then there
is no great rush to fix it for 2.6.(X+1).1, it can wait for 2.6.(X+2).

Security issues could be an exception to this, or it could be seen
that the publication of a security vulnerability causes a regression
which needs to be fixed...

NeilBrown

Re: [RFC] -stable, how it's going to work.

[Arjan van de Ven]

On Thu, 2005-03-10 at 21:00 +1100, Neil Brown wrote:
> On Tuesday March 8, greg@kroah.com wrote:
> > So here's a first cut at how this 2.6 -stable release process is going
> > to work that Chris and I have come up with.  Does anyone have any
> > problems/issues/questions with this?
> 
> One rule that I thought would make sense, but that I don't see listed
> is:
> 
>  - must fix a regression
> 
> If some problem was in 2.6.X and is still there in 2.6.X+1, then there
> is no great rush to fix it for 2.6.(X+1).1, it can wait for 2.6.(X+2).

this is tricky. I mean, what if it's a datacorruption thing? Sure older
kernels may have it too... yet at the same time it'd be nice to get it
fixed. And once you go down this slipperly slope you end up with the
original ruleset again, eg it must be "somehow relatively important".

Re: [RFC] -stable, how it's going to work.

[Greg KH]

On Thu, Mar 10, 2005 at 09:00:51PM +1100, Neil Brown wrote:
> On Tuesday March 8, greg@kroah.com wrote:
> > So here's a first cut at how this 2.6 -stable release process is going
> > to work that Chris and I have come up with.  Does anyone have any
> > problems/issues/questions with this?
> 
> One rule that I thought would make sense, but that I don't see listed
> is:
> 
>  - must fix a regression

That, and a zillion other specific wordings that people suggested fall
under the:
	or some "oh, that's not good" issue
rule.

I didn't feel like being all lawyer-like and explicitly spelling out all
of the different kinds of bugs that we would be accepting patches for :)

So yes, I don't have a problem with patches to fix regressions.

thanks,

greg k-h

Re: [RFC] -stable, how it's going to work.

[Lee Revell]

On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> That, and a zillion other specific wordings that people suggested fall
> under the:
> 	or some "oh, that's not good" issue
> rule.

So just to be 100% clear, no sound with 2.6.N where the sound worked
with 2.6.N-1 absolutely does qualify.  Right?

Lee

Re: [RFC] -stable, how it's going to work.

[Greg KH]

On Thu, Mar 10, 2005 at 12:27:23PM -0500, Lee Revell wrote:
> On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > That, and a zillion other specific wordings that people suggested fall
> > under the:
> > 	or some "oh, that's not good" issue
> > rule.
> 
> So just to be 100% clear, no sound with 2.6.N where the sound worked
> with 2.6.N-1 absolutely does qualify.  Right?

Hm, do you think that is a "good" thing to have happen?...

greg k-h

Re: [RFC] -stable, how it's going to work.

[Chris Wright]

* Lee Revell (rlrevell@joe-job.com) wrote:
> On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > That, and a zillion other specific wordings that people suggested fall
> > under the:
> > 	or some "oh, that's not good" issue
> > rule.
> 
> So just to be 100% clear, no sound with 2.6.N where the sound worked
> with 2.6.N-1 absolutely does qualify.  Right?

Depends, is listening to music while you work critical...? j/k ;-)
Yeah, that's a driver regression...used to work, now it's broken.
If fix is back out all changes, that's not so nice, if it's a
'one-liner' then definitely.  Have a concrete example and patch?

thanks
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: [RFC] -stable, how it's going to work.

[Linus Torvalds]

On Thu, 10 Mar 2005, Lee Revell wrote:
> 
> So just to be 100% clear, no sound with 2.6.N where the sound worked
> with 2.6.N-1 absolutely does qualify.  Right?

If you can send in a patch that fixes it in an obvious way and in less
than 100 lines of context diff, hell yes.

Remember: all the other constraints still hold. Don't fall into the trap 
of believing that "if it fixes a regression, it's for -stable". It needs 
to be _obvious_, and it needs to be small enough that bugs are unlikely.

And that "small enough" is really important. Bugs do happen. Even in 
"obvious" patches. The whole _point_ of -stable is to try to make them 
less likely, and the strict constraints are very much a part of that.

		Linus

Re: [RFC] -stable, how it's going to work.

[Lee Revell]

On Thu, 2005-03-10 at 09:43 -0800, Chris Wright wrote:
> * Lee Revell (rlrevell@joe-job.com) wrote:
> > On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > > That, and a zillion other specific wordings that people suggested fall
> > > under the:
> > > 	or some "oh, that's not good" issue
> > > rule.
> > 
> > So just to be 100% clear, no sound with 2.6.N where the sound worked
> > with 2.6.N-1 absolutely does qualify.  Right?
> 
> Depends, is listening to music while you work critical...? j/k ;-)
> Yeah, that's a driver regression...used to work, now it's broken.
> If fix is back out all changes, that's not so nice, if it's a
> 'one-liner' then definitely.  Have a concrete example and patch?

Not yet.  We are still trying to figure out whether 2.6.11 introduced an
ALSA regression or not.  See the "intel 8x0 went silent" thread.

Lee

Re: [RFC] -stable, how it's going to work.

[Lee Revell]

On Thu, 2005-03-10 at 09:31 -0800, Greg KH wrote:
> On Thu, Mar 10, 2005 at 12:27:23PM -0500, Lee Revell wrote:
> > On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > > That, and a zillion other specific wordings that people suggested fall
> > > under the:
> > > 	or some "oh, that's not good" issue
> > > rule.
> > 
> > So just to be 100% clear, no sound with 2.6.N where the sound worked
> > with 2.6.N-1 absolutely does qualify.  Right?
> 
> Hm, do you think that is a "good" thing to have happen?...

OK, so it sounds like scheduling latency regressions also qualify.  This
could make a system that worked on 2.6.N-1 unusable on 2.6.N, and the
fixes here (usually restoring a lockbreak) are almost always small and
obvious.  And users do report this, usualy in the form of "JACK was
usable under foo kernel but I get xruns with the same config under bar
kernel".

Lee

Re: [RFC] -stable, how it's going to work.

[Neil Brown]

On Thursday March 10, greg@kroah.com wrote:
> On Thu, Mar 10, 2005 at 09:00:51PM +1100, Neil Brown wrote:
> > On Tuesday March 8, greg@kroah.com wrote:
> > > So here's a first cut at how this 2.6 -stable release process is going
> > > to work that Chris and I have come up with.  Does anyone have any
> > > problems/issues/questions with this?
> > 
> > One rule that I thought would make sense, but that I don't see listed
> > is:
> > 
> >  - must fix a regression
> 
> That, and a zillion other specific wordings that people suggested fall
> under the:
> 	or some "oh, that's not good" issue
> rule.
> 
> I didn't feel like being all lawyer-like and explicitly spelling out all
> of the different kinds of bugs that we would be accepting patches for :)
> 
> So yes, I don't have a problem with patches to fix regressions.

I think you did not get the meaning that I was trying to convey...

I didn't mean "If it fixes a regression, it should be accepted."
I meant "If it does not fix a regression, it should not be accepted."

I have the impression that the 2.6.x.y series were suggested because
of regressions in 2.6.11 over 2.6.10 and we needed a way to respond to
that.   I think it should purely be a response to that.


The issue that made me think through this is the locking bug in nfs
(there is a significant delay in getting a contended lock after the
holder drops it).  It has been suggested at least twice for 2.6.11.y.
While it would be nice to have it fixed, I really don't think it
should be a candidate for 2.6.11.y.  It should go into 2.6.12
(which it will) and that should be released 6-10 weeks post 2.6.11
which, given the bug as been around for a lot longer than that without
being widely noticed, should be soon enough.

I fear that if too many bug fixes go into 2.6.11.y, it might seem to
take the pressure of 2.6.12 coming out in a timely fashion, and I
wouldn't like to see that.

NeilBrown

Re: [RFC] -stable, how it's going to work.

[Neil Brown]

On Thursday March 10, arjan@infradead.org wrote:
> On Thu, 2005-03-10 at 21:00 +1100, Neil Brown wrote:
> > One rule that I thought would make sense, but that I don't see listed
> > is:
> > 
> >  - must fix a regression
> > 
> > If some problem was in 2.6.X and is still there in 2.6.X+1, then there
> > is no great rush to fix it for 2.6.(X+1).1, it can wait for 2.6.(X+2).
> 
> this is tricky. I mean, what if it's a datacorruption thing? Sure older
> kernels may have it too... yet at the same time it'd be nice to get it
> fixed. And once you go down this slipperly slope you end up with the
> original ruleset again, eg it must be "somehow relatively important".
> 

My position is that
 - there are lots of things that "it'd be nice to get it fixed", and
    that is what 2.6.X is for
 - It is a very slippery slope 
 - No one expect Linux to be perfect. But people do expect it to get
   better without getting worse.  It is precisely to fix this
   expectation that (I understand) 2.6.x.y was created.

If a data corruption bug has been there for 10 weeks without being
noticed, then the real risk is not that great.  We are calling it
"-release", not "-hardened".

Just my shilling-and-six worth.

NeilBrown

Re: [RFC] -stable, how it's going to work.

[J. Bruce Fields]

On Fri, Mar 11, 2005 at 11:10:37AM +1100, Neil Brown wrote:
> I didn't mean "If it fixes a regression, it should be accepted."
> I meant "If it does not fix a regression, it should not be accepted."

... Presumably with the obvious exception for security fixes.--b.

Re: [RFC] -stable, how it's going to work.

[Chris Friesen]

Neil Brown wrote:

> If a data corruption bug has been there for 10 weeks without being
> noticed, then the real risk is not that great.  We are calling it
> "-release", not "-hardened".

I disagree.  If there's a simple, obvious, small fix that passes all the 
other criteria, it should go into -stable ASAP after passing review. 
Then the -stable maintainers will push the fix to Andrew/Linux, and it 
will go into the next 2.6.x.

Let's keep -stable as good as possible, while still abiding by all the 
other rules.

Chris

Re: [RFC] -stable, how it's going to work.

[Andi Kleen]

Chris Friesen <cfriesen@nortel.com> writes:

> Neil Brown wrote:
>
>> If a data corruption bug has been there for 10 weeks without being
>> noticed, then the real risk is not that great.  We are calling it
>> "-release", not "-hardened".
>
> I disagree.  If there's a simple, obvious, small fix that passes all
> the other criteria, it should go into -stable ASAP after passing
> review. Then the -stable maintainers will push the fix to
> Andrew/Linux, and it will go into the next 2.6.x.

No way, it needs to go into mainline first and then maybe later
into stable. Doing stable first would lead to code drift because
a lot of people would only care about stable and we would be back
in the bad old days when older kernels had more fixes than newer
ones.

-Andi

Re: [RFC] -stable, how it's going to work.

[Pavel Machek]

On Čt 10-03-05 13:25:19, Lee Revell wrote:
> On Thu, 2005-03-10 at 09:31 -0800, Greg KH wrote:
> > On Thu, Mar 10, 2005 at 12:27:23PM -0500, Lee Revell wrote:
> > > On Thu, 2005-03-10 at 08:43 -0800, Greg KH wrote:
> > > > That, and a zillion other specific wordings that people suggested fall
> > > > under the:
> > > > 	or some "oh, that's not good" issue
> > > > rule.
> > > 
> > > So just to be 100% clear, no sound with 2.6.N where the sound worked
> > > with 2.6.N-1 absolutely does qualify.  Right?
> > 
> > Hm, do you think that is a "good" thing to have happen?...
> 
> OK, so it sounds like scheduling latency regressions also qualify.  This
> could make a system that worked on 2.6.N-1 unusable on 2.6.N, and the
> fixes here (usually restoring a lockbreak) are almost always small and
> obvious.  And users do report this, usualy in the form of "JACK was
> usable under foo kernel but I get xruns with the same config under bar
> kernel".

No, I do not think we want to extend it that far. Latency regression
is more of "oh, who cares" issue ;-).
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!