Proposed stable release changes

Proposed stable release changes

[Greg KH]

Hi all,

Given that I had to just revert a patch in the recent stable releases
that didn't get enough time to "bake" in Linus's tree (or in -next), I
figured it was worth discussing some possible changes with how "fast" I
pick up patches for stable releases.

So, how about this proposal:

- I will wait for a -rc to come out with the patch in it before putting
  it into a stable release, unless:
	- the maintainer ACKs it, or sends it directly (like DaveM does
	  for networking patches)
	- I have seen enough discussion about a patch to show that it
	  really does fix something / is good / doesn't cause problems.
	- obviously safe, i.e. "add a device id" type thing.

Given that we have -rc releases every week, except for the initial -rc1
release, I don't think this will really cause any major delays.

Also, now that we are about to head into my busy "travel season", odds
are, I'll be at least a week behind anyway, so this would probably start
happening without an "official" change.  It's been a boring summer, I've
been able to keep up with the stable stuff really easily, causing
problems like this :)

Objections?  Comments?

thanks,

greg k-h

Re: Proposed stable release changes

[Willy Tarreau]

Hi Greg,

On Tue, Aug 20, 2013 at 03:40:32PM -0700, Greg KH wrote:
> Hi all,
> 
> Given that I had to just revert a patch in the recent stable releases
> that didn't get enough time to "bake" in Linus's tree (or in -next), I
> figured it was worth discussing some possible changes with how "fast" I
> pick up patches for stable releases.
> 
> So, how about this proposal:
> 
> - I will wait for a -rc to come out with the patch in it before putting
>   it into a stable release, unless:
> 	- the maintainer ACKs it, or sends it directly (like DaveM does
> 	  for networking patches)
> 	- I have seen enough discussion about a patch to show that it
> 	  really does fix something / is good / doesn't cause problems.
> 	- obviously safe, i.e. "add a device id" type thing.
> 
> Given that we have -rc releases every week, except for the initial -rc1
> release, I don't think this will really cause any major delays.

In the last discussion you initiated on the subject, I proposed something
even more conservative which was the same as above except instead of
"wait for a -rc", it was "wait for rc1 after a full release containing
the patch", unless one of the conditions you proposed, or another one
which would be a tag "urgent" or something like this in the patch. I
tend to think it will be hard at the beginning but will quickly motivate
maintainers to care more about their fixes flow towards -stable depending
on their severity.

Best regards,
Willy

Re: Proposed stable release changes

[Greg KH]

On Wed, Aug 21, 2013 at 12:58:15AM +0200, Willy Tarreau wrote:
> Hi Greg,
> 
> On Tue, Aug 20, 2013 at 03:40:32PM -0700, Greg KH wrote:
> > Hi all,
> > 
> > Given that I had to just revert a patch in the recent stable releases
> > that didn't get enough time to "bake" in Linus's tree (or in -next), I
> > figured it was worth discussing some possible changes with how "fast" I
> > pick up patches for stable releases.
> > 
> > So, how about this proposal:
> > 
> > - I will wait for a -rc to come out with the patch in it before putting
> >   it into a stable release, unless:
> > 	- the maintainer ACKs it, or sends it directly (like DaveM does
> > 	  for networking patches)
> > 	- I have seen enough discussion about a patch to show that it
> > 	  really does fix something / is good / doesn't cause problems.
> > 	- obviously safe, i.e. "add a device id" type thing.
> > 
> > Given that we have -rc releases every week, except for the initial -rc1
> > release, I don't think this will really cause any major delays.
> 
> In the last discussion you initiated on the subject, I proposed something
> even more conservative which was the same as above except instead of
> "wait for a -rc", it was "wait for rc1 after a full release containing
> the patch", unless one of the conditions you proposed, or another one
> which would be a tag "urgent" or something like this in the patch.

Waiting 3 months is too long, in my opinion, sorry.

thanks,

greg k-h

Re: Proposed stable release changes

[Guenter Roeck]

On Tue, Aug 20, 2013 at 04:12:32PM -0700, Greg KH wrote:
> On Wed, Aug 21, 2013 at 12:58:15AM +0200, Willy Tarreau wrote:
> > Hi Greg,
> > 
> > On Tue, Aug 20, 2013 at 03:40:32PM -0700, Greg KH wrote:
> > > Hi all,
> > > 
> > > Given that I had to just revert a patch in the recent stable releases
> > > that didn't get enough time to "bake" in Linus's tree (or in -next), I
> > > figured it was worth discussing some possible changes with how "fast" I
> > > pick up patches for stable releases.
> > > 
> > > So, how about this proposal:
> > > 
> > > - I will wait for a -rc to come out with the patch in it before putting
> > >   it into a stable release, unless:
> > > 	- the maintainer ACKs it, or sends it directly (like DaveM does
> > > 	  for networking patches)
> > > 	- I have seen enough discussion about a patch to show that it
> > > 	  really does fix something / is good / doesn't cause problems.
> > > 	- obviously safe, i.e. "add a device id" type thing.
> > > 
> > > Given that we have -rc releases every week, except for the initial -rc1
> > > release, I don't think this will really cause any major delays.
> > 
> > In the last discussion you initiated on the subject, I proposed something
> > even more conservative which was the same as above except instead of
> > "wait for a -rc", it was "wait for rc1 after a full release containing
> > the patch", unless one of the conditions you proposed, or another one
> > which would be a tag "urgent" or something like this in the patch.
> 
> Waiting 3 months is too long, in my opinion, sorry.
> 
I definitely agree.

Guenter

Re: Proposed stable release changes

[Willy Tarreau]

On Tue, Aug 20, 2013 at 04:12:32PM -0700, Greg KH wrote:
> On Wed, Aug 21, 2013 at 12:58:15AM +0200, Willy Tarreau wrote:
> > Hi Greg,
> > 
> > On Tue, Aug 20, 2013 at 03:40:32PM -0700, Greg KH wrote:
> > > Hi all,
> > > 
> > > Given that I had to just revert a patch in the recent stable releases
> > > that didn't get enough time to "bake" in Linus's tree (or in -next), I
> > > figured it was worth discussing some possible changes with how "fast" I
> > > pick up patches for stable releases.
> > > 
> > > So, how about this proposal:
> > > 
> > > - I will wait for a -rc to come out with the patch in it before putting
> > >   it into a stable release, unless:
> > > 	- the maintainer ACKs it, or sends it directly (like DaveM does
> > > 	  for networking patches)
> > > 	- I have seen enough discussion about a patch to show that it
> > > 	  really does fix something / is good / doesn't cause problems.
> > > 	- obviously safe, i.e. "add a device id" type thing.
> > > 
> > > Given that we have -rc releases every week, except for the initial -rc1
> > > release, I don't think this will really cause any major delays.
> > 
> > In the last discussion you initiated on the subject, I proposed something
> > even more conservative which was the same as above except instead of
> > "wait for a -rc", it was "wait for rc1 after a full release containing
> > the patch", unless one of the conditions you proposed, or another one
> > which would be a tag "urgent" or something like this in the patch.
> 
> Waiting 3 months is too long, in my opinion, sorry.

I meant only for the non-important ones. Their authors will qualify the
ones that are important and must not wait. The same way as now many patches
are correctly tagged "cc: stable", I suspect that we could end up with maybe
80% of patches tagged as "must not wait", and the remaining 20% would indeed
wait up to 3 months, but if their authors think they should wait maybe we
should trust them.

Willy

Re: Proposed stable release changes

[Nicholas A. Bellinger]

On Tue, 2013-08-20 at 15:40 -0700, Greg KH wrote:
> Hi all,
> 
> Given that I had to just revert a patch in the recent stable releases
> that didn't get enough time to "bake" in Linus's tree (or in -next), I
> figured it was worth discussing some possible changes with how "fast" I
> pick up patches for stable releases.
> 
> So, how about this proposal:
> 
> - I will wait for a -rc to come out with the patch in it before putting
>   it into a stable release, unless:
> 	- the maintainer ACKs it, or sends it directly (like DaveM does
> 	  for networking patches)
> 	- I have seen enough discussion about a patch to show that it
> 	  really does fix something / is good / doesn't cause problems.
> 	- obviously safe, i.e. "add a device id" type thing.
> 
> Given that we have -rc releases every week, except for the initial -rc1
> release, I don't think this will really cause any major delays.
> 
> Also, now that we are about to head into my busy "travel season", odds
> are, I'll be at least a week behind anyway, so this would probably start
> happening without an "official" change.  It's been a boring summer, I've
> been able to keep up with the stable stuff really easily, causing
> problems like this :)
> 
> Objections?  Comments?

Sounds like a reasonable tradeoff, as long as the 'please include this
to stable ASAP' latency for critical fixes does not end up being too
large..

--nab

Re: Proposed stable release changes

[Greg KH]

On Tue, Aug 20, 2013 at 04:04:00PM -0700, Nicholas A. Bellinger wrote:
> On Tue, 2013-08-20 at 15:40 -0700, Greg KH wrote:
> > Hi all,
> > 
> > Given that I had to just revert a patch in the recent stable releases
> > that didn't get enough time to "bake" in Linus's tree (or in -next), I
> > figured it was worth discussing some possible changes with how "fast" I
> > pick up patches for stable releases.
> > 
> > So, how about this proposal:
> > 
> > - I will wait for a -rc to come out with the patch in it before putting
> >   it into a stable release, unless:
> > 	- the maintainer ACKs it, or sends it directly (like DaveM does
> > 	  for networking patches)
> > 	- I have seen enough discussion about a patch to show that it
> > 	  really does fix something / is good / doesn't cause problems.
> > 	- obviously safe, i.e. "add a device id" type thing.
> > 
> > Given that we have -rc releases every week, except for the initial -rc1
> > release, I don't think this will really cause any major delays.
> > 
> > Also, now that we are about to head into my busy "travel season", odds
> > are, I'll be at least a week behind anyway, so this would probably start
> > happening without an "official" change.  It's been a boring summer, I've
> > been able to keep up with the stable stuff really easily, causing
> > problems like this :)
> > 
> > Objections?  Comments?
> 
> Sounds like a reasonable tradeoff, as long as the 'please include this
> to stable ASAP' latency for critical fixes does not end up being too
> large..

Given the current number of those that I get are relativly small, it
shouldn't be that big of a deal.

All this really means is that I get a week vacation now, and then I'll
pick back up with stable releases :)

thanks,

greg k-h

Re: Proposed stable release changes

[Guenter Roeck]

On Tue, Aug 20, 2013 at 03:40:32PM -0700, Greg KH wrote:
> Hi all,
> 
> Given that I had to just revert a patch in the recent stable releases
> that didn't get enough time to "bake" in Linus's tree (or in -next), I
> figured it was worth discussing some possible changes with how "fast" I
> pick up patches for stable releases.
> 
> So, how about this proposal:
> 
> - I will wait for a -rc to come out with the patch in it before putting
>   it into a stable release, unless:
> 	- the maintainer ACKs it, or sends it directly (like DaveM does
> 	  for networking patches)
> 	- I have seen enough discussion about a patch to show that it
> 	  really does fix something / is good / doesn't cause problems.
> 	- obviously safe, i.e. "add a device id" type thing.
> 
> Given that we have -rc releases every week, except for the initial -rc1
> release, I don't think this will really cause any major delays.
> 
> Also, now that we are about to head into my busy "travel season", odds
> are, I'll be at least a week behind anyway, so this would probably start
> happening without an "official" change.  It's been a boring summer, I've
> been able to keep up with the stable stuff really easily, causing
> problems like this :)
> 
> Objections?  Comments?
> 
Makes sense to me.

It would be even better if you could find the time to push -rc releases
into the stable repository before you accept patches into a stable branch.
I am now running my test suite on stable/master, so that would give it
some time to catch new problems before they make their way into a stable
branch.

Guenter

Re: Proposed stable release changes

[Greg KH]

On Tue, Aug 20, 2013 at 04:11:17PM -0700, Guenter Roeck wrote:
> It would be even better if you could find the time to push -rc releases
> into the stable repository before you accept patches into a stable branch.

What do you mean by this?

The git tree?  I could do that, but when I have to drop a patch, it
would cause a mess if I had to always go forwards.

I could do branches for -rc releases, that end up as the
"end-of-the-line", and I create the next .y release on top of the
previous one, not the -rc release.

That's kind of what I do "internally" when I create the -rc releases in
the first place, but I just delete those throw-away trees, and never
push them publicly anywhere.

Does it really help anyone to do this, except for some automated
testing?  Doesn't the -rc patch work good enough for that?

> I am now running my test suite on stable/master, so that would give it
> some time to catch new problems before they make their way into a stable
> branch.

I don't understand what you mean here.

thanks,

greg k-h

Re: Proposed stable release changes

[Guenter Roeck]

On Tue, Aug 20, 2013 at 04:17:43PM -0700, Greg KH wrote:
> On Tue, Aug 20, 2013 at 04:11:17PM -0700, Guenter Roeck wrote:
> > It would be even better if you could find the time to push -rc releases
> > into the stable repository before you accept patches into a stable branch.
> 
> What do you mean by this?
>
> The git tree?  I could do that, but when I have to drop a patch, it
> would cause a mess if I had to always go forwards.
> 
That is not what I mean. I referred to the master branch of the stable
repository, which you normally update to the most recent -rc when you
publish a new -stable release.

> I could do branches for -rc releases, that end up as the
> "end-of-the-line", and I create the next .y release on top of the
> previous one, not the -rc release.
> 
I would not ask for creating any new branches. The existing ones are good
enough.

> That's kind of what I do "internally" when I create the -rc releases in
> the first place, but I just delete those throw-away trees, and never
> push them publicly anywhere.
> 
> Does it really help anyone to do this, except for some automated
> testing?  Doesn't the -rc patch work good enough for that?
> 
> > I am now running my test suite on stable/master, so that would give it
> > some time to catch new problems before they make their way into a stable
> > branch.
> 
> I don't understand what you mean here.
> 
The master branch of linux-stable.git on kernel.org currently points to
v3.11-rc5. All I asked for is to update it to the latest -rc prior to
accepting patches from this -rc into a stable release.

I am running my test suite on linux-stable:master to have a reference
and comparison against the various -stable branches. As soon as you update
the master branch of linux-stable.git (eg after you push v3.11-rc6 into it),
the test suite will automatically run on it.

Obviously I could track linux-kernel.git itself instead, but there is much more
activity on it and I am really only interested in the most recent tagged version.

Hope that describes it better.

Guenter

Re: Proposed stable release changes

[Greg KH]

On Tue, Aug 20, 2013 at 05:11:11PM -0700, Guenter Roeck wrote:
> On Tue, Aug 20, 2013 at 04:17:43PM -0700, Greg KH wrote:
> > On Tue, Aug 20, 2013 at 04:11:17PM -0700, Guenter Roeck wrote:
> > > It would be even better if you could find the time to push -rc releases
> > > into the stable repository before you accept patches into a stable branch.
> > 
> > What do you mean by this?
> >
> > The git tree?  I could do that, but when I have to drop a patch, it
> > would cause a mess if I had to always go forwards.
> > 
> That is not what I mean. I referred to the master branch of the stable
> repository, which you normally update to the most recent -rc when you
> publish a new -stable release.

Ah, ok, that makes sense.  I've updated it to 3.11-rc6 now.  Sorry for
the confusion.

greg k-h

Re: Proposed stable release changes

[Josh Boyer]

On Tue, Aug 20, 2013 at 6:40 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> Hi all,
>
> Given that I had to just revert a patch in the recent stable releases
> that didn't get enough time to "bake" in Linus's tree (or in -next), I
> figured it was worth discussing some possible changes with how "fast" I
> pick up patches for stable releases.
>
> So, how about this proposal:
>
> - I will wait for a -rc to come out with the patch in it before putting
>   it into a stable release, unless:
>         - the maintainer ACKs it, or sends it directly (like DaveM does
>           for networking patches)
>         - I have seen enough discussion about a patch to show that it
>           really does fix something / is good / doesn't cause problems.
>         - obviously safe, i.e. "add a device id" type thing.
>
> Given that we have -rc releases every week, except for the initial -rc1
> release, I don't think this will really cause any major delays.
>
> Also, now that we are about to head into my busy "travel season", odds
> are, I'll be at least a week behind anyway, so this would probably start
> happening without an "official" change.  It's been a boring summer, I've
> been able to keep up with the stable stuff really easily, causing
> problems like this :)
>
> Objections?  Comments?

I like this overall.  The only thing I might change is "wait for -rc2"
for patches tagged with CC: stable that go in during the merge window.
 It seems those are the ones that tend to bite us.

Overall though, I think waiting for the next -rc is a good balance.

josh

Re: Proposed stable release changes

[Greg KH]

On Tue, Aug 20, 2013 at 07:40:49PM -0400, Josh Boyer wrote:
> On Tue, Aug 20, 2013 at 6:40 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > Hi all,
> >
> > Given that I had to just revert a patch in the recent stable releases
> > that didn't get enough time to "bake" in Linus's tree (or in -next), I
> > figured it was worth discussing some possible changes with how "fast" I
> > pick up patches for stable releases.
> >
> > So, how about this proposal:
> >
> > - I will wait for a -rc to come out with the patch in it before putting
> >   it into a stable release, unless:
> >         - the maintainer ACKs it, or sends it directly (like DaveM does
> >           for networking patches)
> >         - I have seen enough discussion about a patch to show that it
> >           really does fix something / is good / doesn't cause problems.
> >         - obviously safe, i.e. "add a device id" type thing.
> >
> > Given that we have -rc releases every week, except for the initial -rc1
> > release, I don't think this will really cause any major delays.
> >
> > Also, now that we are about to head into my busy "travel season", odds
> > are, I'll be at least a week behind anyway, so this would probably start
> > happening without an "official" change.  It's been a boring summer, I've
> > been able to keep up with the stable stuff really easily, causing
> > problems like this :)
> >
> > Objections?  Comments?
> 
> I like this overall.  The only thing I might change is "wait for -rc2"
> for patches tagged with CC: stable that go in during the merge window.
>  It seems those are the ones that tend to bite us.

Maintainers can always tag their patches to have me hold off until -rc2
for that.

And, given the huge number of patches for stable that come in during
the -rc1 merge window, there's no way I can get to them all before -rc2
comes out, so this will probably end up being the case for most of them
anyway.

thanks,

greg k-h

Re: Proposed stable release changes

[Josh Boyer]

On Tue, Aug 20, 2013 at 7:57 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Tue, Aug 20, 2013 at 07:40:49PM -0400, Josh Boyer wrote:
>> On Tue, Aug 20, 2013 at 6:40 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
>> > Hi all,
>> >
>> > Given that I had to just revert a patch in the recent stable releases
>> > that didn't get enough time to "bake" in Linus's tree (or in -next), I
>> > figured it was worth discussing some possible changes with how "fast" I
>> > pick up patches for stable releases.
>> >
>> > So, how about this proposal:
>> >
>> > - I will wait for a -rc to come out with the patch in it before putting
>> >   it into a stable release, unless:
>> >         - the maintainer ACKs it, or sends it directly (like DaveM does
>> >           for networking patches)
>> >         - I have seen enough discussion about a patch to show that it
>> >           really does fix something / is good / doesn't cause problems.
>> >         - obviously safe, i.e. "add a device id" type thing.
>> >
>> > Given that we have -rc releases every week, except for the initial -rc1
>> > release, I don't think this will really cause any major delays.
>> >
>> > Also, now that we are about to head into my busy "travel season", odds
>> > are, I'll be at least a week behind anyway, so this would probably start
>> > happening without an "official" change.  It's been a boring summer, I've
>> > been able to keep up with the stable stuff really easily, causing
>> > problems like this :)
>> >
>> > Objections?  Comments?
>>
>> I like this overall.  The only thing I might change is "wait for -rc2"
>> for patches tagged with CC: stable that go in during the merge window.
>>  It seems those are the ones that tend to bite us.
>
> Maintainers can always tag their patches to have me hold off until -rc2
> for that.

They can (not immediately sure how though?), but they don't with the
exception of the few that don't tag at all and send you patches in
bundles.  I mean, that's what the huge thread about the stable trees
that hopefully leads to a conversation at KS is about, right?

> And, given the huge number of patches for stable that come in during
> the -rc1 merge window, there's no way I can get to them all before -rc2
> comes out, so this will probably end up being the case for most of them
> anyway.

Sheer numbers forcing some to be pushed out isn't really that great
when the problem is spread across the whole window.  I'm not saying
you aren't correct that a lot of them don't get pushed to stable
releases until post -rc2, but at that point you're playing catch up.
I was suggesting just sitting on them until -rc2 entirely.  Basically,
don't release a stable kernel until the next version is at -rc2.

Let me phrase this as a question instead.  Is there something we can
do to help catch the patches that get sucked into stable during the
merge window and then wind up causing issues and reverted/fixed after
things settle down in the -rc releases?

Reviewing the patches submitted for those stable kernels isn't
necessarily sufficient, since the issues I'm concerned about aren't
spotted until after the release anyway.  It's a two-fold problem.  The
first is one we're never going to fix in that people are human and
make mistakes and sometimes patches don't get tested well enough
before they're tagged for stable.  The second issue is a combination
of timing and volume.  I doubt we're going to fix the volume since we
keep touting the increasing change rate as a sign of success, but
could we fix the timing by forcing things to bake more in the -rc
releases?

I'm offering to help in whatever way you think is best.  It's your
workflow (and sanity) that are the most impacted.  However, I share
the pain whenever something breaks in stable through the wonderful
place that is Fedora bugzilla so I'm looking for ways to reduce that.

josh

Re: Proposed stable release changes

[Greg KH]

On Tue, Aug 20, 2013 at 08:41:23PM -0400, Josh Boyer wrote:
> On Tue, Aug 20, 2013 at 7:57 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> >> I like this overall.  The only thing I might change is "wait for -rc2"
> >> for patches tagged with CC: stable that go in during the merge window.
> >>  It seems those are the ones that tend to bite us.
> >
> > Maintainers can always tag their patches to have me hold off until -rc2
> > for that.
> 
> They can (not immediately sure how though?)

Some do:
	Cc: stable <stable@vger.kernel.org> # after -rc5 is out
or
	Cc: stable <stable@vger.kernel.org> # wait a -rc cycle
or
	Cc: stable <stable@vger.kernel.org> # wait a few weeks to bake

> , but they don't with the
> exception of the few that don't tag at all and send you patches in
> bundles.  I mean, that's what the huge thread about the stable trees
> that hopefully leads to a conversation at KS is about, right?

Hopefully, yes, but I don't know about that yet.

> Let me phrase this as a question instead.  Is there something we can
> do to help catch the patches that get sucked into stable during the
> merge window and then wind up causing issues and reverted/fixed after
> things settle down in the -rc releases?

Test linux-next and Linus's tree-of-the-day better.  If problems happen,
and a patch has a cc: stable@ on it, let stable@ know about it.

> I'm offering to help in whatever way you think is best.  It's your
> workflow (and sanity) that are the most impacted.  However, I share
> the pain whenever something breaks in stable through the wonderful
> place that is Fedora bugzilla so I'm looking for ways to reduce that.

Letting me know when something breaks is always good as well.  Right now
that doesn't seem to happen much, so either not much is breaking, or I'm
just not told about it, I don't know which.

thanks,

greg k-h

Re: Proposed stable release changes

[Josh Boyer]

On Tue, Aug 20, 2013 at 8:49 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Tue, Aug 20, 2013 at 08:41:23PM -0400, Josh Boyer wrote:
>> On Tue, Aug 20, 2013 at 7:57 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
>> >> I like this overall.  The only thing I might change is "wait for -rc2"
>> >> for patches tagged with CC: stable that go in during the merge window.
>> >>  It seems those are the ones that tend to bite us.
>> >
>> > Maintainers can always tag their patches to have me hold off until -rc2
>> > for that.
>>
>> They can (not immediately sure how though?)
>
> Some do:
>         Cc: stable <stable@vger.kernel.org> # after -rc5 is out
> or
>         Cc: stable <stable@vger.kernel.org> # wait a -rc cycle
> or
>         Cc: stable <stable@vger.kernel.org> # wait a few weeks to bake
>
>> , but they don't with the
>> exception of the few that don't tag at all and send you patches in
>> bundles.  I mean, that's what the huge thread about the stable trees
>> that hopefully leads to a conversation at KS is about, right?
>
> Hopefully, yes, but I don't know about that yet.
>
>> Let me phrase this as a question instead.  Is there something we can
>> do to help catch the patches that get sucked into stable during the
>> merge window and then wind up causing issues and reverted/fixed after
>> things settle down in the -rc releases?
>
> Test linux-next and Linus's tree-of-the-day better.  If problems happen,
> and a patch has a cc: stable@ on it, let stable@ know about it.

Ah, yes.  I forgot about the "look through linux-next for CC: stable".
 That could probably be automated to a degree even.

We already build a Linus kernel in rawhide at least once a day.  Two
flavors even, once with debug options enabled and once without.  The
three of us run it and have an autotest setup going and being
improved, but there is no substitute for wide-spread user testing.
Unfortunately, we have problems getting that with rawhide for various
reasons.

>> I'm offering to help in whatever way you think is best.  It's your
>> workflow (and sanity) that are the most impacted.  However, I share
>> the pain whenever something breaks in stable through the wonderful
>> place that is Fedora bugzilla so I'm looking for ways to reduce that.
>
> Letting me know when something breaks is always good as well.  Right now
> that doesn't seem to happen much, so either not much is breaking, or I'm
> just not told about it, I don't know which.

There's no need to reply specifically to these, but off the top of my
head just for 3.10.y:

brcmsmac explodes in 3.10.{6,7,8,9}.  I think you know about that one
already and Felix and Arend are working on it.

Suspend/Resume is broken on a variety of Thinkpad T400 and T500
machines in 3.10.  This was true with 3.10.0 afaik.  Current thinking
is that it's related to the Intel mei/mei_me driver(s).  Blacklisting
those seems to fix things for a number of users.  There are patches in
3.11-rcX, but the "fix" highlighted doesn't fix it.

There were various i915 backlight issues reported with 3.10.3/4.  I
believe they were fixed with 3.10.5.

USB storage was broken because of the mishandled VPD stuff.  Fixed in 3.10.7.

I'm aware I'm reporting issues that you either already knew about or
were already fixed.  The problem we have is that we roll out a new
stable release and then we get bug reports for 2 weeks because not
everyone updates as frequently as stable releases, etc.  So something
that may seem to impact a small number of users at the time winds up
actually impacting lots of users once it rolls out in a distro.  As
far as I know, Fedora is possibly the only distro actually pushing
stable release kernels out on a normal basis.  I'd love to be wrong on
that point.

In the future, if we can get the information from the end user in
time, I'll be happy to forward issues that aren't already reported
onwards.  Or if you still want to hear about it, I can chime in on the
existing threads with bugzilla numbers.  I'm also willing to do a
monthly "patches we're carrying not in stable" report if people find
that helpful.  I'll likely be doing that within Fedora already and I'm
happy to send it to stable@, even if those patches aren't exactly
stable-rules matching.  We did that when kernel.org went down and it
helped then, just not sure how much it would help now or if people
care.

josh

Re: Proposed stable release changes

[Guenter Roeck]

On Tue, Aug 20, 2013 at 09:03:12PM -0400, Josh Boyer wrote:
[ ... ]

> 
> Suspend/Resume is broken on a variety of Thinkpad T400 and T500
> machines in 3.10.  This was true with 3.10.0 afaik.  Current thinking
> is that it's related to the Intel mei/mei_me driver(s).  Blacklisting

Reminds me ... I had to disable MEI on my servers at home because 
it caused some instability (random system hangs) and the watchdog
didn't work. I thought that was only me, but maybe not.

Guenter

Re: Proposed stable release changes

[Greg KH]

On Tue, Aug 20, 2013 at 09:03:12PM -0400, Josh Boyer wrote:
> Suspend/Resume is broken on a variety of Thinkpad T400 and T500
> machines in 3.10.  This was true with 3.10.0 afaik.  Current thinking
> is that it's related to the Intel mei/mei_me driver(s).  Blacklisting
> those seems to fix things for a number of users.  There are patches in
> 3.11-rcX, but the "fix" highlighted doesn't fix it.

I have heard of mei issues recently, but no real "this is a problem"
type thing.  There are some patches queued up for 3.12 in that area, if
they are needed earlier, that would be great for me, as a subsystem
maintainer, to know.

> I'm aware I'm reporting issues that you either already knew about or
> were already fixed.  The problem we have is that we roll out a new
> stable release and then we get bug reports for 2 weeks because not
> everyone updates as frequently as stable releases, etc.  So something
> that may seem to impact a small number of users at the time winds up
> actually impacting lots of users once it rolls out in a distro.  As
> far as I know, Fedora is possibly the only distro actually pushing
> stable release kernels out on a normal basis.  I'd love to be wrong on
> that point.

The openSUSE Tumbleweed disto also pushes out these stable kernels.  But
there's only an "estimated" 8-10 thousand users of that openSUSE
"flavor", while smaller than what Fedora has, it's better than nothing.

> In the future, if we can get the information from the end user in
> time, I'll be happy to forward issues that aren't already reported
> onwards.  Or if you still want to hear about it, I can chime in on the
> existing threads with bugzilla numbers.  I'm also willing to do a
> monthly "patches we're carrying not in stable" report if people find
> that helpful.

I would love that report, one of the things I keep asking for is for
people to send the patches that distros have that are not in stable to
me, as those obviously are things that are needed for a valid reason
that everyone should be able to benifit from.

> I'll likely be doing that within Fedora already and I'm happy to send
> it to stable@, even if those patches aren't exactly stable-rules
> matching.

If they aren't "allowed" by the current rules, I'd be interested to know
why, unless it's the "add a new feature" type thing, which makes sense
why I couldn't take them.

> We did that when kernel.org went down and it helped then, just not
> sure how much it would help now or if people care.

I care :)

thanks,

greg k-h

Re: Proposed stable release changes

[Willy Tarreau]

On Tue, Aug 20, 2013 at 05:49:24PM -0700, Greg KH wrote:
> On Tue, Aug 20, 2013 at 08:41:23PM -0400, Josh Boyer wrote:
> > On Tue, Aug 20, 2013 at 7:57 PM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > >> I like this overall.  The only thing I might change is "wait for -rc2"
> > >> for patches tagged with CC: stable that go in during the merge window.
> > >>  It seems those are the ones that tend to bite us.
> > >
> > > Maintainers can always tag their patches to have me hold off until -rc2
> > > for that.
> > 
> > They can (not immediately sure how though?)
> 
> Some do:
> 	Cc: stable <stable@vger.kernel.org> # after -rc5 is out
> or
> 	Cc: stable <stable@vger.kernel.org> # wait a -rc cycle
> or
> 	Cc: stable <stable@vger.kernel.org> # wait a few weeks to bake

That's where I think that the default one (with no indication) should
be the higher delay. If the author has no clue about the emergency of
his patch, who else can guess for him ?

It's too optimistic to consider that some code authors will be
realist about the impacts of their code. We all create bugs and
regressions everywhere because we're sure about what we do, until
someone says "hey dude you broke this". So if we expect authors to
say "look, I managed to get this merged into mainline but I'm still
not sure about the risks", I suspect only a small fraction of the
patches will be tagged this way. But I may be wrong, after all it
already works well with -net.

Willy

Re: Proposed stable release changes

[Steven Rostedt]

First I want to say that I 100% support the idea of waiting at least one
-rc. Maybe even two.

On Wed, Aug 21, 2013 at 07:38:36AM +0200, Willy Tarreau wrote:
> > 	Cc: stable <stable@vger.kernel.org> # after -rc5 is out
> > or
> > 	Cc: stable <stable@vger.kernel.org> # wait a -rc cycle
> > or
> > 	Cc: stable <stable@vger.kernel.org> # wait a few weeks to bake
> 
> That's where I think that the default one (with no indication) should
> be the higher delay. If the author has no clue about the emergency of
> his patch, who else can guess for him ?
> 
> It's too optimistic to consider that some code authors will be
> realist about the impacts of their code. We all create bugs and
> regressions everywhere because we're sure about what we do, until
> someone says "hey dude you broke this". So if we expect authors to
> say "look, I managed to get this merged into mainline but I'm still
> not sure about the risks", I suspect only a small fraction of the
> patches will be tagged this way. But I may be wrong, after all it
> already works well with -net.

Really, most fixes are for regressions. If it isn't something that can
let non root crash the kernel, or have access that they shouldn't have,
then let it simmer in the kernel for a while. I don't see any reason to
rush out a regression fix if it just causes a device not to work
anymore. The user can go back to the old kernel for a week or two and
wait. Even with two weeks (or even a month) of waiting, we are still much
faster than Apple or Microsoft in getting regression fixes out.

If people are not rushing to update their kernel to stable every time a
new stable is released then why are we rushing to get them out?

Maybe people are rushing, but I don't update my main machines every
stable release because I can't always afford the down time it causes me
to do so.

-- Steve

Re: Proposed stable release changes

[Jochen Striepe]

	Hello,

just speaking as a user, not a developer.

On Wed, Aug 21, 2013 at 09:37:13AM -0400, Steven Rostedt wrote:
> First I want to say that I 100% support the idea of waiting at least one
> -rc. Maybe even two.

I think so, too. But ...

> Really, most fixes are for regressions.
[...]
> If people are not rushing to update their kernel to stable every time a
> new stable is released then why are we rushing to get them out?

... people are very different. Many just update here and then, but
there are those who do update on most stable releases. Those are the
ones you get feedback and testing from, and I think you should encourage
them to update as soon as a new -stable kernel is released. Getting
regression fixes fast sounds like a good encouragement especially for
people interested in the -stable series. Keeping the number of
regressions low is the whole point of -stable, right?

> Maybe people are rushing, but I don't update my main machines every
> stable release because I can't always afford the down time it causes me
> to do so.

On my server/router and on my desktop machine it's the same, those are
for daily work. But I have a netbook for playing around. :)


Have a nice day,
Jochen.

Re: Proposed stable release changes

[Steven Rostedt]

On Wed, 21 Aug 2013 19:23:27 +0200
Jochen Striepe <jochen@tolot.escape.de> wrote:

> ... people are very different. Many just update here and then, but
> there are those who do update on most stable releases. Those are the
> ones you get feedback and testing from, and I think you should encourage
> them to update as soon as a new -stable kernel is released. Getting
> regression fixes fast sounds like a good encouragement especially for
> people interested in the -stable series. Keeping the number of
> regressions low is the whole point of -stable, right?

Yes, but its even worse when we add a new regression to fix the old one.

Note, this will still encourage people to upgrade to stable, and the
delay should be even more encouragement. The point of the delay is to
catch regressions caused by fixes in mainline. We don't want that added
regression to trickle into stable like it has a few times.

As you say "Keeping the number of regressions low is the whole point of
-stable". Yes it is. By the delay, it should help stable from getting
as many regressions as it is today.

> 
> > Maybe people are rushing, but I don't update my main machines every
> > stable release because I can't always afford the down time it causes me
> > to do so.
> 
> On my server/router and on my desktop machine it's the same, those are
> for daily work. But I have a netbook for playing around. :)

Right, and after upgrading your server/router to a new stable, it will
suck that it hits a new regression and you have to update again. A
delay should help limit those occurrences even more.

For netbook that you can play with the -rc candidates and if you hit a
regression, report it, and go back to a more stable kernel.

The point I'm making, we should be more reluctant in pulling patches
into stable as quick as we are. A patch ideally should simmer in
linux-next for a bit, then go into mainline. It should also simmer
there for a bit before it goes into stable. Except for those very rare
patches that can cause the system to easily crash, let an exploit
happen, or corrupt data. Those do need to go into stable ASAP. But
luckily, those are also rare and far between.

-- Steve

Re: Proposed stable release changes

[Borislav Petkov]

On Wed, Aug 21, 2013 at 01:58:16PM -0400, Steven Rostedt wrote:
> The point I'm making, we should be more reluctant in pulling patches
> into stable as quick as we are. A patch ideally should simmer in
> linux-next for a bit, then go into mainline.

Oh, and it is really debatable if the sheer volume of -stable patches is
actually warranted - several people already raised the question whether
we should be more conservative with the stable tag. But you're probably
going to have this as one of the topics at KS...

Thanks.

Re: Proposed stable release changes

[Greg KH]

On Wed, Aug 21, 2013 at 10:07:03PM +0200, Borislav Petkov wrote:
> On Wed, Aug 21, 2013 at 01:58:16PM -0400, Steven Rostedt wrote:
> > The point I'm making, we should be more reluctant in pulling patches
> > into stable as quick as we are. A patch ideally should simmer in
> > linux-next for a bit, then go into mainline.
> 
> Oh, and it is really debatable if the sheer volume of -stable patches is
> actually warranted - several people already raised the question whether
> we should be more conservative with the stable tag. But you're probably
> going to have this as one of the topics at KS...

And I pushed back on that.  Which specific stable patch should _not_
have been included?

I am going to be pickier (and already have, as some maintainers have
found out), with what I accept, but so far, the number of patches I've
rejected can be counted on one hand, a very small percentage of the
overall number of stable patches.

thanks,

greg k-h

Re: Proposed stable release changes

[Borislav Petkov]

On Wed, Aug 21, 2013 at 01:16:00PM -0700, Greg KH wrote:
> And I pushed back on that. Which specific stable patch should _not_
> have been included?

Well, here's one for example:

https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/commit/?id=f0a56c480196a98479760862468cc95879df3de0
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717473#54

I decided not to tag it for stable, even though Ben wanted it, just
because it is the first bug report for this and it was caused by a
pretty unusual hardware configuration. It simply wasn't important enough
to need to add it to stable, IMO.

So basically the rules at the beginning of
Documentation/stable_kernel_rules.txt didn't really apply and that's why
I held off on it.

And I'm pretty sure I've seen similar minor issues like that simply
"automatically" tagged for stable - I just don't have more concrete
examples right now.

> I am going to be pickier (and already have, as some maintainers have
> found out), with what I accept, but so far, the number of patches I've
> rejected can be counted on one hand, a very small percentage of the
> overall number of stable patches.

Ok, fair enough. I mean, in the end of the day, it is less work for you
and for distro people. And more importantly, less unnecessary work. :-)

Thanks.

Re: Proposed stable release changes

[Steven Rostedt]

On Tue, Aug 20, 2013 at 08:41:23PM -0400, Josh Boyer wrote:
> 
> Let me phrase this as a question instead.  Is there something we can
> do to help catch the patches that get sucked into stable during the
> merge window and then wind up causing issues and reverted/fixed after
> things settle down in the -rc releases?

I'm curious. Of the patches that went into stable that caused problems,
how many were from the merge window?

Reason that I ask this, is because the patches I tag for stable that I
wait for a merge window for release, goes through the process of all my
patches that enter the merge window. It sits in linux-next for a while
and usually gets much more testing than a fix that may come later in the
-rc cycles.

Personally, I still let my stable patches that go into later -rc sit
in linux-next for a few days before pushing them to mainline. I may even
wait for the next -rc to push it just to make sure the patch wont cause
more issues. But I know others that just send the fix directly to
mainline without going through linux-next. Those, I would think, are the
most prone to cause issues in stable.

-- Steve

Re: Proposed stable release changes

[Willy Tarreau]

On Wed, Aug 21, 2013 at 09:42:48AM -0400, Steven Rostedt wrote:
> Personally, I still let my stable patches that go into later -rc sit
> in linux-next for a few days before pushing them to mainline. I may even
> wait for the next -rc to push it just to make sure the patch wont cause
> more issues. But I know others that just send the fix directly to
> mainline without going through linux-next. Those, I would think, are the
> most prone to cause issues in stable.

This is also what I suspected though I have no data to confirm or deny that.
If this happens to be the case, maybe then there should be some barrier such
as :
  - everything merged at -rc4 or before gets backported after the next -rc
  - everything from -rc5 and upper waits for next -rc1 unless tagged urgent ?

Willy

Re: Proposed stable release changes

[Steven Rostedt]

On Wed, 21 Aug 2013 16:17:25 +0200
Willy Tarreau <w@1wt.eu> wrote:

> This is also what I suspected though I have no data to confirm or deny that.
> If this happens to be the case, maybe then there should be some barrier such
> as :
>   - everything merged at -rc4 or before gets backported after the next -rc
>   - everything from -rc5 and upper waits for next -rc1 unless tagged urgent ?

No, I think a full -rc cycle would be good. All commits tagged for
stable that appeared in -rc4 (added from -rc3), must wait till -rc5
before it is added to stable. That way we have a week of full exposure
to find problems.

I guess the other question to ask is, how long does it take for a
problem to appear after hitting mainline? If a problem is found in -rc4
before -rc5 comes out, then this would be sufficient. But if the
problem from -rc4 isn't found till -rc6 then that tells us something
too.

We should be using pass data to determine these heuristics. But I don't
have that data, but Greg should.


-- Steve

Re: Proposed stable release changes

[Stefan Richter]

On Aug 21 Steven Rostedt wrote:
> I guess the other question to ask is, how long does it take for a
> problem to appear after hitting mainline? If a problem is found in -rc4
> before -rc5 comes out, then this would be sufficient. But if the
> problem from -rc4 isn't found till -rc6 then that tells us something
> too.

The estimate of one or two RC periods applies to hardware/ workloads which
everyone has or which wasn't supported by the previous release.  For less
widely used hardware/ workloads which has been supported for years already,
it's more like between 2.5 months ( = a mainline kernel release period)
and more than a year ( = two release periods of typical distributions).
-- 
Stefan Richter
-=====-===-= =--- ==---
http://arcgraph.de/sr/

Re: Proposed stable release changes

[Borislav Petkov]

On Tue, Aug 20, 2013 at 03:40:32PM -0700, Greg KH wrote:
> - I will wait for a -rc to come out with the patch in it before putting
>   it into a stable release, unless:

Question: what's the exact reasoning of that delay? To get more people
who install -rc kernels to smoke-test patches tagged for stable.

What happens if the bug gets detected *after* you've taken the patch? We
probably end up in the same situation.

I guess I'm questioning the usefullness of such a delay period...

Thanks.

Re: Proposed stable release changes

[Greg KH]

On Wed, Aug 21, 2013 at 03:48:52PM +0200, Borislav Petkov wrote:
> On Tue, Aug 20, 2013 at 03:40:32PM -0700, Greg KH wrote:
> > - I will wait for a -rc to come out with the patch in it before putting
> >   it into a stable release, unless:
> 
> Question: what's the exact reasoning of that delay? To get more people
> who install -rc kernels to smoke-test patches tagged for stable.

Yes.

> What happens if the bug gets detected *after* you've taken the patch? We
> probably end up in the same situation.

Yes we will, but this does give us a bit more testing, right?  That's
the point of the delay.

thanks,

greg k-h

Re: Proposed stable release changes

[Stephen Warren]

On 08/20/2013 04:40 PM, Greg KH wrote:
> Hi all,
> 
> Given that I had to just revert a patch in the recent stable releases
> that didn't get enough time to "bake" in Linus's tree (or in -next), I
> figured it was worth discussing some possible changes with how "fast" I
> pick up patches for stable releases.
> 
> So, how about this proposal:
> 
> - I will wait for a -rc to come out with the patch in it before putting
>   it into a stable release, unless: [...]

Presumably the idea is that much useful testing only happens on -rc
kernels rather than linux-next or arbitrary points in Linus' tree.

If so, then don't you want to wait for two -rc releases to come out that
include the patch?

When the first -rc that contains the patch is released, people will test
it then. This won't happen immediately, but might take a few days.

By the time the second -rc that contains the patch is released, that's
presumably enough time for people to have tested the first -rc, and
hence we then presume the patch doesn't cause too many issues.

That's still only an average of 1.5 weeks delay, with a min-max of 1-2,
ignoring the merge window and assuming bugfix patches go quickly
upstream from subsystem maintainers.

Re: Proposed stable release changes

[Linus Torvalds]

On Wed, Aug 21, 2013 at 11:20 AM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> On 08/20/2013 04:40 PM, Greg KH wrote:
>
> Presumably the idea is that much useful testing only happens on -rc
> kernels rather than linux-next or arbitrary points in Linus' tree.

Linux-next gets little to no testing outside of compiles.

And I don't think the -rc releases are all that important either. The
important part is to _wait_. Not "wait for an -rc". There are
reasonable number of developers and users who actually run git
kernels, just because they want to help. At rc points, you tend to get
a few more of those.

In contrast, when patches get moved from the development tree to
stable within a day or two, that patch has gotten basically _no_
testing in some cases (or rather, it's been tested to fix the thing it
was supposed to fix, but then there are surprising new problems that
it introduces that nobody even though about, and wasn't tested for).

So I don't think "is in an rc release" is the important thing. I think
"has been in the standard git tree for at least a week" is what we
should aim for.

Will it catch all cases? Hell no. We don't have *that* many people who
run git kernels, and even people who do don't tend to update daily
anyway. But at least this kind of embarrassing "We found a bug within
almost minutes of it hitting mainline" should not make it into stable.

              Linus

Re: Proposed stable release changes

[Borislav Petkov]

On Wed, Aug 21, 2013 at 11:36:05AM -0700, Linus Torvalds wrote:
> Will it catch all cases? Hell no. We don't have *that* many people who
> run git kernels, and even people who do don't tend to update daily
> anyway.

We don't want to run daily snapshots of your tree though, right? Only
-rcs because the daily states are kinda arbitrary and they can be broken
in various ways. Or are we at a point in time where we can amend that
rule?

For example, what I do currently is, I take your -rcX something and
merge tip/master ontop of it and this is running on my machines for that
week. Come next week, I rinse and repeat. Or does it make sense to do
that more than once a week?

Thanks.

Re: Proposed stable release changes

[Tony Luck]

On Wed, Aug 21, 2013 at 1:00 PM, Borislav Petkov <bp@alien8.de> wrote:
> We don't want to run daily snapshots of your tree though, right? Only
> -rcs because the daily states are kinda arbitrary and they can be broken
> in various ways. Or are we at a point in time where we can amend that
> rule?

If *nobody* runs daily snapshots - then problems just sit latent all week until
the -rc is released and people start testing. Doesn't sound optimal.

Running daily git snapshots can be "exciting" during the merge window. But
I rarely see problems running a random build after -rc1.  If you are still
running that ancient 3.11-rc6 released on Sunday - then you are missing out
on 28 commits worth of goodness since then :-)

-Tony

Re: Proposed stable release changes

[Borislav Petkov]

On Wed, Aug 21, 2013 at 01:54:01PM -0700, Tony Luck wrote:
> On Wed, Aug 21, 2013 at 1:00 PM, Borislav Petkov <bp@alien8.de> wrote:
> > We don't want to run daily snapshots of your tree though, right? Only
> > -rcs because the daily states are kinda arbitrary and they can be broken
> > in various ways. Or are we at a point in time where we can amend that
> > rule?
> 
> If *nobody* runs daily snapshots - then problems just sit latent all week until
> the -rc is released and people start testing. Doesn't sound optimal.
> 
> Running daily git snapshots can be "exciting" during the merge window. But
> I rarely see problems running a random build after -rc1.  If you are still
> running that ancient 3.11-rc6 released on Sunday - then you are missing out
> on 28 commits worth of goodness since then :-)

Damn, I'm such a l0z3r!

Re: Proposed stable release changes

[Geert Uytterhoeven]

On Wed, Aug 21, 2013 at 10:54 PM, Tony Luck <tony.luck@gmail.com> wrote:
> Running daily git snapshots can be "exciting" during the merge window. But
> I rarely see problems running a random build after -rc1.  If you are still

Indeed.

> running that ancient 3.11-rc6 released on Sunday - then you are missing out
> on 28 commits worth of goodness since then :-)

Yeah, that's why I almost started debugging the non-appearance of my
shiny new procfs entry. Still running 3.11-rc6 :-(
Then I remembered the proc_readdir() issues...

Perhaps -rc7 should have been released immediately after those fixes ;-)

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Proposed stable release changes

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 588 bytes --]

On Wed, 21 Aug 2013 11:36:05 -0700 Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> On Wed, Aug 21, 2013 at 11:20 AM, Stephen Warren <swarren@wwwdotorg.org> wrote:
> > On 08/20/2013 04:40 PM, Greg KH wrote:
> >
> > Presumably the idea is that much useful testing only happens on -rc
> > kernels rather than linux-next or arbitrary points in Linus' tree.
> 
> Linux-next gets little to no testing outside of compiles.

Besides which, regression fixes often get no exposure in linux-next anyway.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]