New (now current development process)

New (now current development process)

[Paolo Ciarrocchi]

Hi all,
I would like to write a short article about the new development
process discussed during last Linux Kernel Developers' Summit for my
local LUG.

Since I'm not able to find an accurate report of what has been
discussed during that meeting I try to summariza what is my
understanding of the current process:

The are two kind of releases, 2.6.x kerneles and 2.6.x.y

2.6.x.y are maintained by the "stable" team (stable at kernel dot
org), are released amost every week.

Here is the latest ( I hope ) announce I found in the archive
including an explanation of how stable development works:

Rules on what kind of patches are accepted, and what ones are not, into
the "-stable" tree:
 - It must be obviously correct and tested.
 - It can not bigger than 100 lines, with context.
 - It must fix only one thing.
 - It must fix a real bug that bothers people (not a, "This could be a
   problem..." type thing.)
 - It must fix a problem that causes a build error (but not for things
   marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
   security issue, or some "oh, that's not good" issue.  In short,
   something critical.
 - No "theoretical race condition" issues, unless an explanation of how
   the race can be exploited.
 - It can not contain any "trivial" fixes in it (spelling changes,
   whitespace cleanups, etc.)
 - It must be accepted by the relevant subsystem maintainer.
 - It must follow Documentation/SubmittingPatches rules.

Procedure for submitting patches to the -stable tree:
 - Send the patch, after verifying that it follows the above rules, to
   stable@kernel.org.
 - The sender will receive an ack when the patch has been accepted into
   the queue, or a nak if the patch is rejected.  This response might
   take a few days, according to the developer's schedules.
 - If accepted, the patch will be added to the -stable queue, for review
   by other developers.
 - Security patches should not be sent to this alias, but instead to the
   documented security@kernel.org.

Review cycle:
 - When the -stable maintainers decide for a review cycle, the patches
   will be sent to the review committee, and the maintainer of the
   affected area of the patch (unless the submitter is the maintainer of
   the area) and CC: to the linux-kernel mailing list.
 - The review committee has 48 hours in which to ack or nak the patch.
 - If the patch is rejected by a member of the committee, or linux-kernel
   members object to the patch by bringing up issues that the maintainer
   and members did not realize, the patch will be dropped from the
   queue.
 - At the end of the review cycle, the acked patches will be added to
   the latest -stable release, and a new -stable release will happen.
 - Security patches will be accepted into the -stable tree directly from
   the security kernel team, and not go through the normal review cycle.
   Contact the kernel security team for more details on this procedure.

Review committe:
 - This will be made up of a number of kernel developers who have
   volunteered for this task, and a few that haven't.

2.6.x kernel are maintained by Linus Torvalds and Andrew Morton (both
are from OSDL) and the development is as follow:
- As soon a new kernel is released a two weeks windows is open, during
this period of time maintainers can submit big diffs to Linus (usually
the patched sitted in -mm kernels for a few weeks). Preferred way to
submit big changes is using GIT ( more information about GIT at
http://git.or.cz/ and
http://www.kernel.org/pub/software/scm/git/docs/tutorial.html).
- After two weeks a -rc1 kernel is released and now is possible to
push only patches that do not include new functionalities)
- Aftwer two weeks -rc2 is released
- Process continues until the kernel is considered "ready", the
process should last three months ( 4 kernels per yeard should be
released).


I'm sure I'm missing lot of details so I would really appreciate if
you could support me in writing this article that hopefully will be
usefull outside my LUG as well :-)

Thanks in advance.

--
Paolo

Re: New (now current development process)

[Tony Luck]

On 10/29/05, Paolo Ciarrocchi <paolo.ciarrocchi@gmail.com> wrote:
> 2.6.x kernel are maintained by Linus Torvalds and Andrew Morton (both
> are from OSDL) and the development is as follow:
Linus maintains the 2.6.x kernel, and Andrew maintains the "-mm" tree
(which is used as a testing ground for anything non-trivial before it is
fed into Linus' tree).

> - As soon a new kernel is released a two weeks windows is open, during
> this period of time maintainers can submit big diffs to Linus (usually
> the patched sitted in -mm kernels for a few weeks). Preferred way to
> submit big changes is using GIT ( more information about GIT at
> http://git.or.cz/ and
> http://www.kernel.org/pub/software/scm/git/docs/tutorial.html).
> - After two weeks a -rc1 kernel is released and now is possible to
> push only patches that do not include new functionalities)
Initially Linus said that he would only accept e-mail patches after
rc1 ... but common sense prevailed and he later clarified to say that
git merges could still be used, but only to include bug fixes.

Also note that a whole new driver (or filesystem) might be accepted
after -rc1 because there is no risk of causing regressions with
such a change.

> - After two weeks -rc2 is released
There isn't really a hard schedule for when -rcN for N>1 are released.

> - Process continues until the kernel is considered "ready", the
> process should last three months ( 4 kernels per yeard should be
> released).
IIRC the goal was to make a release in around 8 weeks (which would
be closer to six per year).  But you have the important part, which is
that Linus doesn't make the release until it is "ready".  2.6.13 was
released on August 28th, and 2.6.14 on October 27th ... so we
appear to have hit the goal this time through.

-Tony

Re: New (now current development process)

[Russell King]

On Sat, Oct 29, 2005 at 11:57:30AM -0700, Tony Luck wrote:
> > - Process continues until the kernel is considered "ready", the
> > process should last three months ( 4 kernels per yeard should be
> > released).
> IIRC the goal was to make a release in around 8 weeks (which would
> be closer to six per year).  But you have the important part, which is
> that Linus doesn't make the release until it is "ready".  2.6.13 was
> released on August 28th, and 2.6.14 on October 27th ... so we
> appear to have hit the goal this time through.

However, three months is _far_ too long.  Look at what is happening
post-2.6.14?  There's loads of really huge changes going in which
were backed up over the last two months.

Continuing like this will just push the release of each kernel further
and further out as there's more stuff merged during the mayhem than
can possibly be properly reviewed and tested.

Shorter release cycles means that there's less mayhem, which in turn
means more time to review.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: New (now current development process)

[Linus Torvalds]

On Sat, 29 Oct 2005, Russell King wrote:

> On Sat, Oct 29, 2005 at 11:57:30AM -0700, Tony Luck wrote:
> > IIRC the goal was to make a release in around 8 weeks (which would
> > be closer to six per year).  But you have the important part, which is
> > that Linus doesn't make the release until it is "ready".  2.6.13 was
> > released on August 28th, and 2.6.14 on October 27th ... so we
> > appear to have hit the goal this time through.
> 
> However, three months is _far_ too long.  Look at what is happening
> post-2.6.14?  There's loads of really huge changes going in which
> were backed up over the last two months.

Yes. Three months would be too much. I considered even 2.6.14 to be 
delayed by at least one week. So 8 weeks is certainly better than it used 
to be, but I think 6 weeks should be the goal-post.

But to hit that, we'd might need to shrink the "merge window" from two 
weeks to just one, otherwise there's not enough time to calm down. With 
most of the real development happening during the previous calm-down 
stage, that might not be impossible: we'd only have one week for merges, 
but that isn't the week when development actually happens, so who cares?

Everything said, I think 2.6.13->14 worked well enough, even if it's hard 
to say how well a process works after one release. Considering that 2.6.13 
had the painful PCI changes (you may not have noticed too much, since they 
were x86 only) and there were some potentially painful SCSI changes in the 
.14 early merges, so it's not like 13->14 was an "easy" release - so the 
process certainly _seems_ to be workable.

So I'm planning on continuing with it unchanged for now. Two-week merge 
window until -rc1, and then another -rc kernel roughly every week until 
release. With the goal being 6 weeks, and 8 weeks being ok.

I don't think anybody has been really unhappy with this approach? Hmm?

			Linus

Re: New (now current development process)

[Akula2]

Hi Linus,

This is my first mail to the Kernel tree:-

> Everything said, I think 2.6.13->14 worked well enough, even if it's hard
> to say how well a process works after one release. Considering that 2.6.13
> had the painful PCI changes (you may not have noticed too much, since they
> were x86 only) and there were some potentially painful SCSI changes in the
> .14 early merges, so it's not like 13->14 was an "easy" release - so the
> process certainly _seems_ to be workable.

Will you please throw more light on the *painful* PCI & SCSI changees?


> I don't think anybody has been really unhappy with this approach? Hmm?

honestly, I did download 14 yesterday and made the kernel ready to
compile. today I've seen 14git1 patch! Am trying to understand why is
it this kind of kernel development model...

thanks,
Akula

Re: New (now current development process)

[Greg KH]

On Sun, Oct 30, 2005 at 02:14:39AM +0530, Akula2 wrote:
> Hi Linus,
> 
> This is my first mail to the Kernel tree:-
> 
> > Everything said, I think 2.6.13->14 worked well enough, even if it's hard
> > to say how well a process works after one release. Considering that 2.6.13
> > had the painful PCI changes (you may not have noticed too much, since they
> > were x86 only) and there were some potentially painful SCSI changes in the
> > .14 early merges, so it's not like 13->14 was an "easy" release - so the
> > process certainly _seems_ to be workable.
> 
> Will you please throw more light on the *painful* PCI & SCSI changees?

If it worked for you, it wasn't painful :)

But for PCI, we changed the way we did resource allocation for devices,
which caused a lot of odd problems on some machines.

thanks,

greg k-h

Re: New (now current development process)

[Andi Kleen]

Linus Torvalds <torvalds@osdl.org> writes:
> 
> But to hit that, we'd might need to shrink the "merge window" from two 
> weeks to just one, otherwise there's not enough time to calm down. With 

Please don't. Even the two weeks are too short IMHO, because it is
hard to digest so much code in such a short time and also it is not
always easy for maintainers to hit such short time windows for sending
patches.

> I don't think anybody has been really unhappy with this approach? Hmm?

The long freeze periods were nothing much happens are painful. It
would be better to have some more overlap of merging and stabilizing
(stable does that already kind of, but not enough)

-Andi

Re: New (now current development process)

[Russell King]

On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> Linus Torvalds <torvalds@osdl.org> writes:
> > I don't think anybody has been really unhappy with this approach? Hmm?
> 
> The long freeze periods were nothing much happens are painful. It
> would be better to have some more overlap of merging and stabilizing
> (stable does that already kind of, but not enough)

Violently agree.  I find the long freeze periods painful and very very
very boring, to the point of looking for other stuff to do (such as
cleaning up bits of the kernel and queuing mega-patches for the next
round of merging.)

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: New (now current development process)

[Andrew Morton]

Russell King <rmk+lkml@arm.linux.org.uk> wrote:
>
> On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > Linus Torvalds <torvalds@osdl.org> writes:
> > > I don't think anybody has been really unhappy with this approach? Hmm?
> > 
> > The long freeze periods were nothing much happens are painful. It
> > would be better to have some more overlap of merging and stabilizing
> > (stable does that already kind of, but not enough)
> 
> Violently agree.  I find the long freeze periods painful and very very
> very boring, to the point of looking for other stuff to do (such as
> cleaning up bits of the kernel and queuing mega-patches for the next
> round of merging.)

The freezes are for fixing bugs, especially recent regressions.  There's no
shortage of them, you know.

I you can think of a better way to get kernel developers off their butts
and actually fixing bugs, I'm all ears.

Re: New (now current development process)

[Russell King]

On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> >
> > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > Linus Torvalds <torvalds@osdl.org> writes:
> > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > > 
> > > The long freeze periods were nothing much happens are painful. It
> > > would be better to have some more overlap of merging and stabilizing
> > > (stable does that already kind of, but not enough)
> > 
> > Violently agree.  I find the long freeze periods painful and very very
> > very boring, to the point of looking for other stuff to do (such as
> > cleaning up bits of the kernel and queuing mega-patches for the next
> > round of merging.)
> 
> The freezes are for fixing bugs, especially recent regressions.

Given my stated low activity during the -rc periods, well, you draw
your conclusion from that.

> There's no shortage of them, you know.

Please let me know when there's something in my area regresses.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: New (now current development process)

[Andrew Morton]

Russell King <rmk+lkml@arm.linux.org.uk> wrote:
>
> On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> > Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> > >
> > > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > > Linus Torvalds <torvalds@osdl.org> writes:
> > > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > > > 
> > > > The long freeze periods were nothing much happens are painful. It
> > > > would be better to have some more overlap of merging and stabilizing
> > > > (stable does that already kind of, but not enough)
> > > 
> > > Violently agree.  I find the long freeze periods painful and very very
> > > very boring, to the point of looking for other stuff to do (such as
> > > cleaning up bits of the kernel and queuing mega-patches for the next
> > > round of merging.)
> > 
> > The freezes are for fixing bugs, especially recent regressions.
> 
> Given my stated low activity during the -rc periods, well, you draw
> your conclusion from that.
> 
> > There's no shortage of them, you know.
> 
> Please let me know when there's something in my area regresses.
> 

a) you're sitting around feeling very very very bored while

b) the kernel is in long freeze due to the lack of kernel developer
   attention to known bugs

The solution seems fairly obvious to me?

Re: New (now current development process)

[Russell King]

On Sun, Oct 30, 2005 at 02:31:03PM -0800, Andrew Morton wrote:
> Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> >
> > On Sun, Oct 30, 2005 at 11:12:41AM -0800, Andrew Morton wrote:
> > > Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> > > >
> > > > On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> > > > > Linus Torvalds <torvalds@osdl.org> writes:
> > > > > > I don't think anybody has been really unhappy with this approach? Hmm?
> > > > > 
> > > > > The long freeze periods were nothing much happens are painful. It
> > > > > would be better to have some more overlap of merging and stabilizing
> > > > > (stable does that already kind of, but not enough)
> > > > 
> > > > Violently agree.  I find the long freeze periods painful and very very
> > > > very boring, to the point of looking for other stuff to do (such as
> > > > cleaning up bits of the kernel and queuing mega-patches for the next
> > > > round of merging.)
> > > 
> > > The freezes are for fixing bugs, especially recent regressions.
> > 
> > Given my stated low activity during the -rc periods, well, you draw
> > your conclusion from that.
> > 
> > > There's no shortage of them, you know.
> > 
> > Please let me know when there's something in my area regresses.
> > 
> 
> a) you're sitting around feeling very very very bored while
> 
> b) the kernel is in long freeze due to the lack of kernel developer
>    attention to known bugs
> 
> The solution seems fairly obvious to me?

That's fine if you have the hardware to be able to debug these issues.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: New (now current development process)

[Andrew Morton]

Russell King <rmk+lkml@arm.linux.org.uk> wrote:
>
>  That's fine if you have the hardware to be able to debug these issues.

Most driver bugs cannot be reproduced by the developer.  Almost by
definition: if the patch had caused problems on the developer's machine, he
wouldn't have shipped it.

This is why we have this wonderful group of long-suffering people who
download and test development kernels for us, and who take the time to
report defects.

Yes, it's painful to get into a long-range debugging session, sending debug
patches, twelve-hour turnaround, etc.  But what alternative have we?

Re: New (now current development process)

[Russell King]

On Sun, Oct 30, 2005 at 02:55:33PM -0800, Andrew Morton wrote:
> Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> >
> >  That's fine if you have the hardware to be able to debug these issues.
> 
> Most driver bugs cannot be reproduced by the developer.  Almost by
> definition: if the patch had caused problems on the developer's machine, he
> wouldn't have shipped it.
> 
> This is why we have this wonderful group of long-suffering people who
> download and test development kernels for us, and who take the time to
> report defects.
> 
> Yes, it's painful to get into a long-range debugging session, sending debug
> patches, twelve-hour turnaround, etc.  But what alternative have we?

However, it helps if you have a grasp of technologies like ACPI,
IO-APICs, APICs, PCI IRQ routing for x86 problems.  Since I don't
work in the x86 field, I have _zero_ knowledge of such things
because they just don't apply when working on ARM.

That makes debugging x86 problems a nightmare.  Remember I gave
up with trying to sort out peoples PCMCIA problems?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: New (now current development process)

[Andi Kleen]

On Sunday 30 October 2005 20:12, Andrew Morton wrote:

> The freezes are for fixing bugs, especially recent regressions.  There's no
> shortage of them, you know.
>
> I you can think of a better way to get kernel developers off their butts
> and actually fixing bugs, I'm all ears.

The problem is that you usually cannot do proper bug fixing because
the release might be just around the corner, so you typically
chose the ugly workaround or revert, or just reject changes for bugs that a 
are too risky or the impact too low because there is not enough time to 
properly test anymore.

It might work better if we were told when the releases would actually
happen and you don't need to fear that this not quite tested everywhere
bugfix you're about to submit might make it into the gold kernel, breaking
the world for some subset of users.

-Andi

Re: New (now current development process)

[Russell King]

On Mon, Oct 31, 2005 at 01:48:56AM +0100, Andi Kleen wrote:
> On Sunday 30 October 2005 20:12, Andrew Morton wrote:
> 
> > The freezes are for fixing bugs, especially recent regressions.  There's no
> > shortage of them, you know.
> >
> > I you can think of a better way to get kernel developers off their butts
> > and actually fixing bugs, I'm all ears.
> 
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a 
> are too risky or the impact too low because there is not enough time to 
> properly test anymore.
> 
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.

Indeed - a prime example is the bootmem initialisation problem.  Had
I known on 1st October that the final release wouldn't have been for
a long time, I'd have applied that patch and you wouldn't have had
that problem with ARM relying on the bootmem initialisation order
clashing with your ia64 (or x86_64) swiotlb fix.

But stupid rmk - again I hasten to add - was suckered into this "-rc
is frozen" idea again and decided it wasn't appropriate to submit it.
In hind-sight, there would have been plenty of time to sort out any
issues.

Hell, we held up 2.6.14 for swiotlb _anyway_.

Time to start writing those lines...
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.
I must learn to ignore what Linus says about frozen releases.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: New (now current development process)

[Andrew Morton]

Russell King <rmk+lkml@arm.linux.org.uk> wrote:
>
> > It might work better if we were told when the releases would actually
>  > happen and you don't need to fear that this not quite tested everywhere
>  > bugfix you're about to submit might make it into the gold kernel, breaking
>  > the world for some subset of users.
> 
>  Indeed - a prime example is the bootmem initialisation problem.

No, I'd say that was an *exception*, not a "prime example".

I've filed away 322 unresolved bug reports here.  The great majority are
busted drivers on random hardware dating back as far as 2.6.11.  Many of
them are regressions.

There is nothing stopping anyone from working with the originators to get
these things fixed up at any time.

Why is it necessary for me to chase maintainers to get their bugs fixed?

Why are maintainers working on new features when they have unresolved bugs?

Why is it so often me who has to do the followup for un-responded-to bug
reports against subsystems which I don't know anything about?

(Those are rhetorical questions, btw).

Re: New (now current development process)

[Andi Kleen]

On Monday 31 October 2005 02:22, Andrew Morton wrote:

> There is nothing stopping anyone from working with the originators to get
> these things fixed up at any time.
>
> Why is it necessary for me to chase maintainers to get their bugs fixed?
>
> Why are maintainers working on new features when they have unresolved bugs?

Because zero bugs is just unrealistic and they would never get anything done
if that was the requirement? 

(especially considering that a lot of the bugs at least on x86-64 are 
hardware/firmware bugs of some sort, so often it is not really a linux
bug but just a missing ha^w^wwork^w^w^w^wfix for something else) 

I agree regressions are a problem and need to be addressed, but handling all 
non regressions on a non trivial platforms is just impossible IMHO...

Perhaps it would be nice to have better bug classification: e.g.
regression/new hardware/reported by more than one person etc.  I think
with some prioritization like that it would be much easier to keep the bugs
under control. 

Sometimes bugs are less important than others.
e.g. on x86-64 the bootmem issue was obscure at best, affecting
only a very small part of the user base. Sure the few people hit by it
will be annoyed, but trying to make everyone happy is impossible so
one has to try to just make the majority of users happy. So it was imho
ok to just revert the patch to fix ARM again and not breaking IA64 (I cannot 
assess how many users were on ARM/IA64 affected)  for the next release and 
try to sort it out later.

Regressions are important, everything else has to be prioritized based on the 
impact on the user base (and this doesn't necessarily mean the most noisy 
part of the userbase) 

Perhaps some people could volunteer to set some flags in bugzilla for obvious 
things, like regression or new hardware or missing basic information or for 
really old kernel and no report for a new one and that could be used to 
filter the queries better. Should be an relatively easy task.

-Andi

Re: New (now current development process)

[Zwane Mwaikambo]

On Mon, 31 Oct 2005, Andi Kleen wrote:

> Perhaps some people could volunteer to set some flags in bugzilla for obvious 
> things, like regression or new hardware or missing basic information or for 
> really old kernel and no report for a new one and that could be used to 
> filter the queries better. Should be an relatively easy task.

I don't think we need any special flags, we just need more people paying 
attention to it. It doesn't take that much time to go through pending bugs 
and trying to identify real ones, but the problem is that we need 
knowledgeable people to do this.

Re: New (now current development process)

[Andrew Morton]

Andi Kleen <ak@suse.de> wrote:
>
> On Monday 31 October 2005 02:22, Andrew Morton wrote:
> 
> > There is nothing stopping anyone from working with the originators to get
> > these things fixed up at any time.
> >
> > Why is it necessary for me to chase maintainers to get their bugs fixed?
> >
> > Why are maintainers working on new features when they have unresolved bugs?
> 
> Because zero bugs is just unrealistic and they would never get anything done
> if that was the requirement? 
> 
> (especially considering that a lot of the bugs at least on x86-64 are 
> hardware/firmware bugs of some sort, so often it is not really a linux
> bug but just a missing ha^w^wwork^w^w^w^wfix for something else) 
> 
> I agree regressions are a problem and need to be addressed, but handling all 
> non regressions on a non trivial platforms is just impossible IMHO...
> 
> Perhaps it would be nice to have better bug classification: e.g.
> regression/new hardware/reported by more than one person etc.  I think
> with some prioritization like that it would be much easier to keep the bugs
> under control. 

Well sure.  But who does that?  It should be the relevant maintainer, IMO. 
If that's the way in which he chooses to work.  Expecting some third party
to do this on a kenrel-wide basis won't fly.

> ...
> Sometimes bugs are less important than others.

I don't believe that what we're seeing is some prioritisation process.  The
problem is that some bugs are *harder* than others, and people are just
ducking things which cannot be locally reproduced.

Re: New (now current development process)

[Roman Zippel]

Hi,

On Mon, 31 Oct 2005, Andi Kleen wrote:

> I agree regressions are a problem and need to be addressed, but handling all 
> non regressions on a non trivial platforms is just impossible IMHO...

Here is a different kind of non trivial regression on a common platform:

$ ll -S /boot/ | grep ...
-rw-r--r--  1 root root 1518317 2005-11-01 00:38 vmlinuz-2.6.14
-rw-r--r--  1 root root 1506432 2005-08-30 00:36 vmlinuz-2.6.13
-rw-r--r--  1 root root 1451154 2004-12-26 16:54 vmlinuz-2.6.10
-rw-r--r--  1 root root 1432032 2005-06-26 03:50 vmlinuz-2.6.12
-rw-r--r--  1 root root 1413888 2004-06-21 18:33 vmlinuz-2.6.7
-rw-r--r--  1 root root 1394801 2004-05-17 22:14 vmlinuz-2.6.5
-rw-r--r--  1 root root 1390233 2004-05-18 01:54 vmlinuz-2.6.6
-rw-r--r--  1 root root 1315050 2003-09-12 22:10 vmlinuz-2.6.0-test5
-rw-r--r--  1 root root 1280189 2003-06-06 01:00 vmlinuz-2.5.70
-rw-r--r--  1 root root  918663 2002-11-11 22:42 vmlinuz-2.5.47
-rw-r--r--  1 root root  887758 2004-02-19 01:24 vmlinuz-2.4.25
-rw-r--r--  1 root root  883868 2003-12-20 21:02 vmlinuz-2.4.23
-rw-r--r--  1 root root  875858 2003-08-26 00:39 vmlinuz-2.4.22
-rw-r--r--  1 root root  859547 2003-02-06 22:35 vmlinuz-2.5.59
-rw-r--r--  1 root root  855092 2003-06-14 02:29 vmlinuz-2.4.21
-rw-r--r--  1 root root  703585 2002-12-19 11:58 vmlinuz-2.4.20
-rw-r--r--  1 root root  700359 2002-08-03 19:57 vmlinuz-2.4.19
-rw-r--r--  1 root root  690106 2001-01-07 23:38 vmlinuz-2.4.0
-rw-r--r--  1 root root  678080 2002-03-02 20:48 vmlinuz-2.4.18
-rw-r--r--  1 root root  641409 2001-02-23 02:53 vmlinuz-2.4.2
-rw-r--r--  1 root root  640950 2001-01-31 01:30 vmlinuz-2.4.1
-rw-r--r--  1 root root  635339 2001-12-22 03:16 vmlinuz-2.4.17
-rw-r--r--  1 root root  632647 2001-12-04 00:37 vmlinuz-2.4.16
-rw-r--r--  1 root root  629787 2001-11-07 22:00 vmlinuz-2.4.14
-rw-r--r--  1 root root  629240 2001-08-11 18:16 vmlinuz-2.4.8
-rw-r--r--  1 root root  628556 2001-07-21 17:06 vmlinuz-2.4.7
-rw-r--r--  1 root root  628386 2001-09-24 00:02 vmlinuz-2.4.10
-rw-r--r--  1 root root  627597 2001-10-20 15:14 vmlinuz-2.4.12
-rw-r--r--  1 root root  626634 2001-07-11 20:22 vmlinuz-2.4.6
-rw-r--r--  1 root root  623325 2001-05-28 01:57 vmlinuz-2.4.5
-rw-r--r--  1 root root  622619 2001-05-01 16:20 vmlinuz-2.4.4
-rw-r--r--  1 root root  611493 2001-04-02 20:44 vmlinuz-2.4.3
-rw-r--r--  1 root root  497523 2001-03-30 18:56 vmlinuz-2.2.19
-rw-r--r--  1 root root  495553 2001-03-25 17:24 vmlinuz-2.2.18

I'm a bit scared of this development and I'm afraid that merging all the 
new features at high speed has become more important than code quality...

bye, Roman

Re: New (now current development process)

[Andrew Morton]

Roman Zippel <zippel@linux-m68k.org> wrote:
>
> Here is a different kind of non trivial regression on a common platform:
> 
> $ ll -S /boot/ | grep ...
> -rw-r--r--  1 root root 1518317 2005-11-01 00:38 vmlinuz-2.6.14
> -rw-r--r--  1 root root 1506432 2005-08-30 00:36 vmlinuz-2.6.13
> -rw-r--r--  1 root root 1451154 2004-12-26 16:54 vmlinuz-2.6.10
> -rw-r--r--  1 root root 1432032 2005-06-26 03:50 vmlinuz-2.6.12
> -rw-r--r--  1 root root 1413888 2004-06-21 18:33 vmlinuz-2.6.7
> -rw-r--r--  1 root root 1394801 2004-05-17 22:14 vmlinuz-2.6.5
> -rw-r--r--  1 root root 1390233 2004-05-18 01:54 vmlinuz-2.6.6
> -rw-r--r--  1 root root 1315050 2003-09-12 22:10 vmlinuz-2.6.0-test5
> -rw-r--r--  1 root root 1280189 2003-06-06 01:00 vmlinuz-2.5.70
> -rw-r--r--  1 root root  918663 2002-11-11 22:42 vmlinuz-2.5.47
> -rw-r--r--  1 root root  887758 2004-02-19 01:24 vmlinuz-2.4.25
> -rw-r--r--  1 root root  883868 2003-12-20 21:02 vmlinuz-2.4.23

What does size(1) say?

Are you sure these kernels are feature-equivalent?

Re: New (now current development process)

[Linus Torvalds]

On Mon, 31 Oct 2005, Andrew Morton wrote:
> 
> Are you sure these kernels are feature-equivalent?

They may not be feature-equivalent in reality, but it's hard to generate 
something that has the features (or lack there-of) of old kernels these 
days. Which is problematic.

But some of it is likely also compilers. gcc does insane padding in many 
cases these days. 

And a lot of it is us just being bloated. Argh.

		Linus

Re: New (now current development process)

[Andrew Morton]

Linus Torvalds <torvalds@osdl.org> wrote:
>
> 
> 
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> > 
> > Are you sure these kernels are feature-equivalent?
> 
> They may not be feature-equivalent in reality, but it's hard to generate 
> something that has the features (or lack there-of) of old kernels these 
> days. Which is problematic.

Probably.

> But some of it is likely also compilers. gcc does insane padding in many 
> cases these days. 

2.6.14 `make allnoconfig':

gcc-2.95.4:

	bix:/usr/src/25> size vmlinux 
	   text    data     bss     dec     hex filename
	 665502  152379   55120  873001   d5229 vmlinux

gcc version 4.1.0 20050513 (experimental):

	bix:/usr/src/25> size vmlinux
	   text    data     bss     dec     hex filename
	 761415  151851   55280  968546   ec762 vmlinux

(There's a new reason for retaining gcc-2.95.x support)

(gcc-4.x can probably be tuned up with appropriate `-malign' options)

> And a lot of it is us just being bloated. Argh.

2.5.71, gcc-2.95.4:

	bix:/usr/src/aa/linux-2.5.71> size vmlinux 
	   text    data     bss     dec     hex filename
	 501892   54163   40420  596475   919fb vmlinux

yes, it got bigger.   .data went through the roof - maybe inlined debug stuff?

2.6.8.1, gcc-2.95.4:

	bix:/usr/src/aa/linux-2.6.8.1> size vmlinux
	   text    data     bss     dec     hex filename
	 605032  153817   58176  817025   c7781 vmlinux


It happened somewhere between 2.5.71 and 2.6.8.

2.4.x doesn't have allnoconfig, so no numbers for that.

Re: New (now current development process)

[Grant Coady]

Andrew Morton wrote:
> 
> 2.4.x doesn't have allnoconfig, so no numbers for that.

-rw-r--r--  1 root root    7501 2005-11-01 10:57 config-2.4.31-none
-rw-r--r--  1 root root  249263 2005-11-01 10:57 bzImage-2.4.31-none
-rw-r--r--  1 root root   71548 2005-11-01 10:57 System.map-2.4.31-none

These things snuck in (everything on make menuconfig cleared):

grant@sempro:~/linux/linux-2.4.31-none$ grep = .config
CONFIG_X86=y
CONFIG_UID16=y
CONFIG_MPENTIUMIII=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_L1_CACHE_SHIFT=5
CONFIG_X86_HAS_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_PGE=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_F00F_WORKS_OK=y
CONFIG_NOHIGHMEM=y
CONFIG_X86_TSC=y
CONFIG_RAMFS=y
CONFIG_MSDOS_PARTITION=y
CONFIG_LOG_BUF_SHIFT=0
grant@sempro:~/linux/linux-2.4.31-none$

Cheers,
Grant.

Re: New (now current development process)

[Adrian Bunk]

On Mon, Oct 31, 2005 at 04:34:08PM -0800, Andrew Morton wrote:
> Linus Torvalds <torvalds@osdl.org> wrote:
> >
> > 
> > 
> > On Mon, 31 Oct 2005, Andrew Morton wrote:
> > > 
> > > Are you sure these kernels are feature-equivalent?
> > 
> > They may not be feature-equivalent in reality, but it's hard to generate 
> > something that has the features (or lack there-of) of old kernels these 
> > days. Which is problematic.
> 
> Probably.
> 
> > But some of it is likely also compilers. gcc does insane padding in many 
> > cases these days. 
> 
> 2.6.14 `make allnoconfig':
> 
> gcc-2.95.4:
> 
> 	bix:/usr/src/25> size vmlinux 
> 	   text    data     bss     dec     hex filename
> 	 665502  152379   55120  873001   d5229 vmlinux
> 
> gcc version 4.1.0 20050513 (experimental):
> 
> 	bix:/usr/src/25> size vmlinux
> 	   text    data     bss     dec     hex filename
> 	 761415  151851   55280  968546   ec762 vmlinux

That's not a fair comparison.

If you do not tell gcc to optimize for size you can't expect it to 
produce small code.

The following is with allnoconfig + CONFIG_EMBEDDED=y + intelligent 
setting of the options shown by "make oldconfig" after setting 
CONFIG_EMBEDDED=y (the third kernel is built without 
-fno-unit-at-a-time):

   text    data     bss     dec     hex filename
 522545   77694   31256  631495   9a2c7 vmlinux-2.95
 436878   76594   32248  545720   853b8 vmlinux-4.0
 429420   76370   32184  537974   83576 vmlinux-4.0-unit-at-a-time


> (There's a new reason for retaining gcc-2.95.x support)
>...

There's a new reason for removing gcc-2.95.x support.  ;-)

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: New (now current development process)

[Nix]

On 1 Nov 2005, Andrew Morton stated:
> gcc-2.95.4:
> 
> 	bix:/usr/src/25> size vmlinux 
> 	   text    data     bss     dec     hex filename
> 	 665502  152379   55120  873001   d5229 vmlinux
> 
> gcc version 4.1.0 20050513 (experimental):
> 
> 	bix:/usr/src/25> size vmlinux
> 	   text    data     bss     dec     hex filename
> 	 761415  151851   55280  968546   ec762 vmlinux
> 
> (There's a new reason for retaining gcc-2.95.x support)

What if you build with -Os? That tends to hold alignments down quite a
bit.

(I'll try it this evening...)

-- 
`"Gun-wielding recluse gunned down by local police" isn't the epitaph
 I want. I am hoping for "Witnesses reported the sound up to two hundred
 kilometers away" or "Last body part finally located".' --- James Nicoll

Re: New (now current development process)

[Bill Davidsen]

Andrew Morton wrote:
> Linus Torvalds <torvalds@osdl.org> wrote:
> 
>>
>>
>>On Mon, 31 Oct 2005, Andrew Morton wrote:
>>
>>>Are you sure these kernels are feature-equivalent?
>>
>>They may not be feature-equivalent in reality, but it's hard to generate 
>>something that has the features (or lack there-of) of old kernels these 
>>days. Which is problematic.
> 
> 
> Probably.
> 
> 
>>But some of it is likely also compilers. gcc does insane padding in many 
>>cases these days. 
> 
> 
> 2.6.14 `make allnoconfig':
> 
> gcc-2.95.4:
> 
> 	bix:/usr/src/25> size vmlinux 
> 	   text    data     bss     dec     hex filename
> 	 665502  152379   55120  873001   d5229 vmlinux
> 
> gcc version 4.1.0 20050513 (experimental):
> 
> 	bix:/usr/src/25> size vmlinux
> 	   text    data     bss     dec     hex filename
> 	 761415  151851   55280  968546   ec762 vmlinux

Was this with -Os or ???
-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: New (now current development process)

[Roland Dreier]

> And a lot of it is us just being bloated. Argh.

Perhaps it's time to dust this patch off:


Reduce kernel text by moving many uses of GFP_KERNEL out of line, by
creating __kmem_cache_alloc_gfp_kernel(), __kmalloc_gfp_kernel() and
__kzalloc_gfp_kernel() wrapper functions.  Then kmem_cache_alloc(),
kmalloc() and kzalloc() can be inline functions that just call their
_gfp_kernel version if their flags parameter is a compile-time
constant and is GFP_KERNEL, and call their ordinary version otherwise.

On an x86_64 allnoconfig kernel, I see the following sizes:

	   text	   data	    bss	    dec	    hex	filename
	 795683	 197356	 116384	1109423	 10edaf	vmlinux-before
	 794894	 197420	 116384	1108698	 10eada	vmlinux-after

for a net savings of 789 bytes of text.  More realistic configs will
save even more.  With my usual config, the patched kernel boots and
runs fine.

Signed-off-by: Roland Dreier <rolandd@cisco.com>

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 09b9aa6..618331d 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -61,7 +61,8 @@ extern kmem_cache_t *kmem_cache_create(c
 				       void (*)(void *, kmem_cache_t *, unsigned long));
 extern int kmem_cache_destroy(kmem_cache_t *);
 extern int kmem_cache_shrink(kmem_cache_t *);
-extern void *kmem_cache_alloc(kmem_cache_t *, gfp_t);
+extern void *__kmem_cache_alloc(kmem_cache_t *, gfp_t);
+extern void *__kmem_cache_alloc_gfp_kernel(kmem_cache_t *);
 extern void kmem_cache_free(kmem_cache_t *, void *);
 extern unsigned int kmem_cache_size(kmem_cache_t *);
 extern const char *kmem_cache_name(kmem_cache_t *);
@@ -75,6 +76,15 @@ struct cache_sizes {
 };
 extern struct cache_sizes malloc_sizes[];
 extern void *__kmalloc(size_t, gfp_t);
+extern void *__kmalloc_gfp_kernel(size_t);
+
+static inline void *kmem_cache_alloc(kmem_cache_t *cache, gfp_t flags)
+{
+	if (__builtin_constant_p(flags) && flags == GFP_KERNEL)
+		return __kmem_cache_alloc_gfp_kernel(cache);
+
+	return __kmem_cache_alloc(cache, flags);
+}
 
 static inline void *kmalloc(size_t size, gfp_t flags)
 {
@@ -96,10 +106,22 @@ found:
 			malloc_sizes[i].cs_dmacachep :
 			malloc_sizes[i].cs_cachep, flags);
 	}
+
+	if (__builtin_constant_p(flags) && flags == GFP_KERNEL)
+		return __kmalloc_gfp_kernel(size);
 	return __kmalloc(size, flags);
 }
 
-extern void *kzalloc(size_t, gfp_t);
+extern void *__kzalloc(size_t, gfp_t);
+extern void *__kzalloc_gfp_kernel(size_t);
+
+static inline void *kzalloc(size_t size, gfp_t flags)
+{
+	if (__builtin_constant_p(flags) && flags == GFP_KERNEL)
+		return __kzalloc_gfp_kernel(size);
+
+	return __kzalloc(size, flags);
+}
 
 /**
  * kcalloc - allocate memory for an array. The memory is set to zero.
diff --git a/mm/slab.c b/mm/slab.c
index 22bfb0b..2971e1c 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -2788,11 +2788,17 @@ static inline void __cache_free(kmem_cac
  * Allocate an object from this cache.  The flags are only relevant
  * if the cache has no available objects.
  */
-void *kmem_cache_alloc(kmem_cache_t *cachep, gfp_t flags)
+void *__kmem_cache_alloc(kmem_cache_t *cachep, gfp_t flags)
 {
 	return __cache_alloc(cachep, flags);
 }
-EXPORT_SYMBOL(kmem_cache_alloc);
+EXPORT_SYMBOL(__kmem_cache_alloc);
+
+void *__kmem_cache_alloc_gfp_kernel(kmem_cache_t *cachep)
+{
+	return __cache_alloc(cachep, GFP_KERNEL);
+}
+EXPORT_SYMBOL(__kmem_cache_alloc_gfp_kernel);
 
 /**
  * kmem_ptr_validate - check if an untrusted pointer might
@@ -2925,6 +2931,12 @@ void *__kmalloc(size_t size, gfp_t flags
 }
 EXPORT_SYMBOL(__kmalloc);
 
+void *__kmalloc_gfp_kernel(size_t size)
+{
+	return __kmalloc(size, GFP_KERNEL);
+}
+EXPORT_SYMBOL(__kmalloc_gfp_kernel);
+
 #ifdef CONFIG_SMP
 /**
  * __alloc_percpu - allocate one copy of the object for every present
@@ -2998,14 +3010,20 @@ EXPORT_SYMBOL(kmem_cache_free);
  * @size: how many bytes of memory are required.
  * @flags: the type of memory to allocate.
  */
-void *kzalloc(size_t size, gfp_t flags)
+void *__kzalloc(size_t size, gfp_t flags)
 {
 	void *ret = kmalloc(size, flags);
 	if (ret)
 		memset(ret, 0, size);
 	return ret;
 }
-EXPORT_SYMBOL(kzalloc);
+EXPORT_SYMBOL(__kzalloc);
+
+void *__kzalloc_gfp_kernel(size_t size)
+{
+	return __kzalloc(size, GFP_KERNEL);
+}
+EXPORT_SYMBOL(__kzalloc_gfp_kernel);
 
 /**
  * kfree - free previously allocated memory

Re: New (now current development process)

[Linus Torvalds]

On Tue, 1 Nov 2005, Roland Dreier wrote:
> 
> On an x86_64 allnoconfig kernel, I see the following sizes:
> 
> 	   text	   data	    bss	    dec	    hex	filename
> 	 795683	 197356	 116384	1109423	 10edaf	vmlinux-before
> 	 794894	 197420	 116384	1108698	 10eada	vmlinux-after

I really don't see the point of shaving less than a kB with ugly calling 
convention magic, when switching to -Os can save us much more, and when 
the networking code is several hundred kB.

If we start doing size optimizations, we need to think big.

		Linus

Re: New (now current development process)

[Roland Dreier]

    Linus> I really don't see the point of shaving less than a kB with
    Linus> ugly calling convention magic, when switching to -Os can
    Linus> save us much more, and when the networking code is several
    Linus> hundred kB.

Fair enough, although to be fair the savings are still there even with
-Os enabled.  With CONFIG_CC_OPTIMIZE_FOR_SIZE=y:

   text	   data	    bss	    dec	    hex	filename
 682564	 197172	 116384	 996120	  f3318	vmlinux-before
 681824	 197244	 116384	 995452	  f307c	vmlinux-after

And with a realistic config rather than allnoconfig, you could easily
save as much as 3 or even 4 KB of text!

BTW, allnoconfig has CONFIG_NET=n, so we have to find someone else to
blame here.

    Linus> If we start doing size optimizations, we need to think big.

No pun intended I'm sure...

Anyway, it would be great to find ways to make big improvements.  But
I think the most realistic way to shrink the kernel is the same way it
grows in the first place -- one small piece at a time.

 - R.

Re: New (now current development process)

[Linus Torvalds]

On Tue, 1 Nov 2005, Roland Dreier wrote:
> 
> Anyway, it would be great to find ways to make big improvements.  But
> I think the most realistic way to shrink the kernel is the same way it
> grows in the first place -- one small piece at a time.

No, I think that's a lost cause.

It doesn't grow by 700 bytes once in a while. It grows by much more, and 
much more often. And we can't fight it that way, that's just not going to 
work. Maybe have something that tracks individual object file sizes and 
shames people into not growing them..

		Linus

Re: New (now current development process)

[Roland Dreier]

    Linus> No, I think that's a lost cause.

    Linus> It doesn't grow by 700 bytes once in a while. It grows by
    Linus> much more, and much more often. And we can't fight it that
    Linus> way, that's just not going to work. Maybe have something
    Linus> that tracks individual object file sizes and shames people
    Linus> into not growing them..

I think we're actually agreeing.  My kmalloc/kzalloc patch is a cute
hack but magic tricks like that aren't going to shrink the kernel by
very much and it's probably not worth merging complications like that.

The way to a smaller kernel is for a lot of people to do a lot of hard
work and look at where we can trim fat.

For your last suggestion, maybe someone can automate running Andi's
bloat-o-meter?  I think the hard part is maintaining comparable configs.

 - R.

Re: New (now current development process)

[Linus Torvalds]

On Tue, 1 Nov 2005, Roland Dreier wrote:
> 
> I think we're actually agreeing.  My kmalloc/kzalloc patch is a cute
> hack but magic tricks like that aren't going to shrink the kernel by
> very much and it's probably not worth merging complications like that.

Right. I don't like having too many ways of doing the same thing - it just 
confuses things and makes different people have different "coding styles" 
and makes things less maintainable (I think the perl philosophy is 
broken).

> The way to a smaller kernel is for a lot of people to do a lot of hard
> work and look at where we can trim fat.

Yes. On the other hand, I do believe that bloat is a fact of life. 
Eventually somebody younger and leaver will come along and displace us. 
It's how things should be. We can (and should try to, of course) only 
delay the inevitable ;)

The fact is, we do do more, and we're more complex. Out VM is a _lot_ more 
complex, and our VFS layer has grown a lot due to all the support it has 
for situations that simply weren't an issue before. And even when not 
used, that support is there.

> For your last suggestion, maybe someone can automate running Andi's
> bloat-o-meter?  I think the hard part is maintaining comparable configs.

Yes. And we should probably make -Os the default. Apparently Fedora 
already does that by just forcibly hacking the Kconfig files.

With modern CPU's, instructions are almost "free". The real cost is in 
cache misses, and that tends to be doubly true of system software that 
tends to have a lot more cache misses than "normal" programs (because 
people try hard to batch up system calls like write etc, so by the time 
the kernel is called, the L1 cache is mostly flushed already - possibly 
the L2 too. And interrupts may be in the "fast path", but they'd sure as 
hell better not happen so often that they stay cached very well etc etc).

So -Os probably performs better in real life, and likely only performs 
worse on micro-benchmarks. Sadly, micro-benchmarks are often very 
instructive in many other ways.

			Linus

Re: New (now current development process)

[Dave Jones]

On Wed, Nov 02, 2005 at 07:54:04AM -0800, Linus Torvalds wrote:

 > > For your last suggestion, maybe someone can automate running Andi's
 > > bloat-o-meter?  I think the hard part is maintaining comparable configs.
 > 
 > Yes. And we should probably make -Os the default. Apparently Fedora 
 > already does that by just forcibly hacking the Kconfig files.

(excuse any typos, this wireless connection is god-awful)
We do. We rip out the dependancyon CONFIG_EMBEDDED, and build
with OPTIMISE_FOR_SIZE set. At least we usually do.
Once every so often, we hit something which throws a spanner
in the works, like the "x86-64 doesn't boot any more" problem
that was fixed by the patch that Alexandre posted earlier
this week.

Most of the time now, when we hit bugs with -Os, it seems to be due
to broken asm constraints in the kernel rather than actual
gcc bugs, but of course, they also occur from time to time,
whereas the same code works just fine with -O2.
I think part of th reason for this is exactly because it
doesn't get a great deal of testing.

		Dave

Re: New (now current development process)

[Adrian Bunk]

On Wed, Nov 02, 2005 at 12:48:52PM -0500, Dave Jones wrote:
> On Wed, Nov 02, 2005 at 07:54:04AM -0800, Linus Torvalds wrote:
> 
>  > > For your last suggestion, maybe someone can automate running Andi's
>  > > bloat-o-meter?  I think the hard part is maintaining comparable configs.
>  > 
>  > Yes. And we should probably make -Os the default. Apparently Fedora 
>  > already does that by just forcibly hacking the Kconfig files.
> 
> (excuse any typos, this wireless connection is god-awful)
> We do. We rip out the dependancyon CONFIG_EMBEDDED, and build
> with OPTIMISE_FOR_SIZE set. At least we usually do.
> Once every so often, we hit something which throws a spanner
> in the works, like the "x86-64 doesn't boot any more" problem
> that was fixed by the patch that Alexandre posted earlier
> this week.
> 
> Most of the time now, when we hit bugs with -Os, it seems to be due
> to broken asm constraints in the kernel rather than actual
> gcc bugs, but of course, they also occur from time to time,
> whereas the same code works just fine with -O2.
> I think part of th reason for this is exactly because it
> doesn't get a great deal of testing.

Is the usage of -Os in Fedora based on actual measurements?

> 		Dave

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: New (now current development process)

[David Lang]

On Wed, 2 Nov 2005, Dave Jones wrote:

> On Wed, Nov 02, 2005 at 07:54:04AM -0800, Linus Torvalds wrote:
>
> > > For your last suggestion, maybe someone can automate running Andi's
> > > bloat-o-meter?  I think the hard part is maintaining comparable configs.
> >
> > Yes. And we should probably make -Os the default. Apparently Fedora
> > already does that by just forcibly hacking the Kconfig files.
>
> (excuse any typos, this wireless connection is god-awful)
> We do. We rip out the dependancyon CONFIG_EMBEDDED, and build
> with OPTIMISE_FOR_SIZE set. At least we usually do.
> Once every so often, we hit something which throws a spanner
> in the works, like the "x86-64 doesn't boot any more" problem
> that was fixed by the patch that Alexandre posted earlier
> this week.
>
> Most of the time now, when we hit bugs with -Os, it seems to be due
> to broken asm constraints in the kernel rather than actual
> gcc bugs, but of course, they also occur from time to time,
> whereas the same code works just fine with -O2.
> I think part of th reason for this is exactly because it
> doesn't get a great deal of testing.

I used to compile with Os on all my kernels, but when the alignment 
settings got added to the embedded section a few kernels ago and I saw 
that they all default to 0 (no alignment) it scared me off, becouse I know 
that unaligned accesses can be extremely expensive, and I wasn't 
comfortable that I had the right info to set them for the different CPU's 
that I have.

I would really like to see Os move out of the embedded section so that it 
can be tested (and used) without having to worry about all those other 
settings. The other option would be to have all those other settings get 
good defaults (with an option to reset them to the defaults when you 
change the CPU type).

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: New (now current development process)

[Sam Ravnborg]

> I used to compile with Os on all my kernels, but when the alignment 
> settings got added to the embedded section a few kernels ago and I saw 
> that they all default to 0 (no alignment) it scared me off

Alignment settings in the EMBEDDED menu are ignored if set to 0, in
other words setting alignment to 0 in Kconfig will fall back to
compilers default values.

Thats also documented in the help for the config options.

	Sam

Re: New (now current development process)

[Andi Kleen]

On Wednesday 02 November 2005 23:31, Sam Ravnborg wrote:
> > I used to compile with Os on all my kernels, but when the alignment
> > settings got added to the embedded section a few kernels ago and I saw
> > that they all default to 0 (no alignment) it scared me off
>
> Alignment settings in the EMBEDDED menu are ignored if set to 0, in
> other words setting alignment to 0 in Kconfig will fall back to
> compilers default values.
>
> Thats also documented in the help for the config options.

In my opinion the alignment settings are completely useless
(far too low level for a user configuration) and should be just 
removed. The per cpu or -Os defaults are totally adequate.

-Andi

Re: New (now current development process)

[Rob Landley]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

On Wednesday 02 November 2005 00:05, Linus Torvalds wrote:
> On Tue, 1 Nov 2005, Roland Dreier wrote:
> > Anyway, it would be great to find ways to make big improvements.  But
> > I think the most realistic way to shrink the kernel is the same way it
> > grows in the first place -- one small piece at a time.
>
> No, I think that's a lost cause.
>
> It doesn't grow by 700 bytes once in a while. It grows by much more, and
> much more often. And we can't fight it that way, that's just not going to
> work. Maybe have something that tracks individual object file sizes and
> shames people into not growing them..

nm --size-sort vmlinux

Old busybox size optimization trick.  Running it on 2.6.14 "allnoconfig" 
attached.  Lots and lots of small symbols (sizes are in hex), with no really 
big low-hanging fruit jumping out.

Rob

[-- Attachment #2: sizesort.txt.bz2 --]
[-- Type: application/x-bzip2, Size: 31840 bytes --]

Re: New (now current development process)

[Tim Bird]

Linus Torvalds wrote:
> 
> On Tue, 1 Nov 2005, Roland Dreier wrote:
> 
>>Anyway, it would be great to find ways to make big improvements.  But
>>I think the most realistic way to shrink the kernel is the same way it
>>grows in the first place -- one small piece at a time.
> 
> 
> No, I think that's a lost cause.
> 
> It doesn't grow by 700 bytes once in a while. It grows by much more, and 
> much more often. And we can't fight it that way, that's just not going to 
> work. Maybe have something that tracks individual object file sizes and 
> shames people into not growing them..

As mentioned at the kernel summit, we're working on it
at the CE Linux Forum.  These things take time to set up,
but we have code already for something that tests
the size impact of individual kernel configs, and we're
working on a test to track individual function sizes for
each new kernel (using Andi Kleen's bloat-o-meter).

We're still in process of setting up the test lab,
but we have a number of hardware boards already,
and we're hoping to be publishing size regression data
for the kernel on a regular basis, starting in about
April of next year.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: New (now current development process)

[Andi Kleen]

On Friday 04 November 2005 23:08, Tim Bird wrote:
> starting in about April of next year.

Any particular reason it's taking so long? Would it perhaps be possible
to do a subset (e.g. only a single architecture and less configs) sooner?
If the problem is writing the necessary scripts, I'm sure some people
(e.g. M. Bligh or Al Viro) have already written some that do
at least the automated building and would be happy to
share them.

-Andi

Re: New (now current development process)

[Tim Bird]

Andi Kleen wrote:
> On Friday 04 November 2005 23:08, Tim Bird wrote:
>
>>starting in about April of next year.
>
> Any particular reason it's taking so long? Would it perhaps be possible
> to do a subset (e.g. only a single architecture and less configs) sooner?
> If the problem is writing the necessary scripts, I'm sure some people
> (e.g. M. Bligh or Al Viro) have already written some that do
> at least the automated building and would be happy to
> share them.

The main reason it's taking so long is that it's
a background task for everyone working on it.
(Basically, me and one guy at NEC. ;-)
We've gotten sidetracked with lab setup issues
(signing a lease for office space, processing
paperwork for board donations, etc. etc.)
But I can run some tests now on a few different
platforms just on my desk.

I had hoped to have at least some preliminary
results by the end of November, with a bigger
call for help inside the forum to finish up
something big and professional-looking in April.
However, maybe I can cobble together some
things, and let people (both in and out of
the forum) look at the preliminary
results sooner.

I'll see what I can do by the end of November.

Thanks for the encouragement.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: New (now current development process)

[Andreas Kleen]

Am Di 01.11.2005 01:34 schrieb Andrew Morton <akpm@osdl.org>:

> It happened somewhere between 2.5.71 and 2.6.8.
>

bloat-o-meter might tell who is to blame. Just run it
with the two vmlinuxes and it will display the differences in symbol
sizes
 
ftp://ftp.firstfloor.org/pub/ak/bloat-o-meter
 
-Andi

Re: New (now current development process)

[Russell King]

On Mon, Oct 31, 2005 at 04:13:22PM -0800, Linus Torvalds wrote:
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> > Are you sure these kernels are feature-equivalent?
> 
> They may not be feature-equivalent in reality, but it's hard to generate 
> something that has the features (or lack there-of) of old kernels these 
> days. Which is problematic.
> 
> But some of it is likely also compilers. gcc does insane padding in many 
> cases these days. 
> 
> And a lot of it is us just being bloated. Argh.

Which is one of the reasons I've started working on fixing up the
platform device/driver stuff to conform to the "usual" method,
with the view to killing off _all_ the function pointers in
struct device_driver.

Most bus types wrap struct device_driver, and then provide their own
function pointers which pass their bus-type specific device structure.
This does two things: 1. it centralises the conversion from struct
device to struct whatever_device, and 2. improves typechecking.

However, once the use of the function pointers in struct device_driver
have been eliminated, we can be sure of reclaiming at least 20 bytes
per device driver, maybe more if GCC does insane padding.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: New (now current development process)

[Rob Landley]

On Monday 31 October 2005 18:13, Linus Torvalds wrote:
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> > Are you sure these kernels are feature-equivalent?
>
> They may not be feature-equivalent in reality, but it's hard to generate
> something that has the features (or lack there-of) of old kernels these
> days. Which is problematic.
>
> But some of it is likely also compilers. gcc does insane padding in many
> cases these days.
>
> And a lot of it is us just being bloated. Argh.
>
>   Linus

Matt Mackall!  Tiny tree!  Yay rah cool!

http://selenic.com/tiny/2.6.14-tiny1-broken-out.tar.bz2

Rob

P.S. There's a reason I'm trying to make a real working development system 
based on busybox and uclibc.  I think things like live CDs should be using 
that, not the GNU packages.

There seems to be a periodic trend, where ever few years open source programs 
get feature-laden enough that somebody forks off (or starts over) a version 
that has the sole virtue of being smaller and simpler.  From glibc->uClibc, 
gnome/kde->xfce, OpenSSH->dropbear, gnu->busybox...  Of course mozilla had to 
do this twice (Galleon, then Firefox) to get something remotely reasonable, 
but oh well.

(And it'd be really NICE if tcc became a reasonable replacement for gcc.  
Guess what the bloated memory-thrashing load that selectively triggers the 
OOM killer (when swappiness=0 but not when swappiness=60) I reported earlier 
is?  Building gcc 4.0.2, genattrtab and compiling the resulting 
insn-attrab.c.  It won't run in "only" 128 megs of ram at the best of 
times...)

Rob

Re: New (now current development process)

[Adrian Bunk]

On Mon, Oct 31, 2005 at 04:13:22PM -0800, Linus Torvalds wrote:
> 
> 
> On Mon, 31 Oct 2005, Andrew Morton wrote:
> > 
> > Are you sure these kernels are feature-equivalent?
> 
> They may not be feature-equivalent in reality, but it's hard to generate 
> something that has the features (or lack there-of) of old kernels these 
> days. Which is problematic.
>...

There's also the question how to define feature-equivalent.

I want my computer to power off after "halt".

There's a significant difference between how much kernel code was 
required for this feature for my old computer and what is required for 
my new computer (in kernel 2.6.14):

   text    data     bss     dec     hex filename
  12721    2124     104   14949    3a65 arch/i386/kernel/apm.o
 136789    5724    4788  147301   23f65 drivers/acpi/built-in.o

> 		Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: New (now current development process)

[Roman Zippel]

Hi,

On Mon, 31 Oct 2005, Andrew Morton wrote:

> Roman Zippel <zippel@linux-m68k.org> wrote:
> >
> > Here is a different kind of non trivial regression on a common platform:
> > 
> > $ ll -S /boot/ | grep ...
> > -rw-r--r--  1 root root 1518317 2005-11-01 00:38 vmlinuz-2.6.14
> > -rw-r--r--  1 root root 1506432 2005-08-30 00:36 vmlinuz-2.6.13
> > -rw-r--r--  1 root root 1451154 2004-12-26 16:54 vmlinuz-2.6.10
> > -rw-r--r--  1 root root 1432032 2005-06-26 03:50 vmlinuz-2.6.12
> > -rw-r--r--  1 root root 1413888 2004-06-21 18:33 vmlinuz-2.6.7
> > -rw-r--r--  1 root root 1394801 2004-05-17 22:14 vmlinuz-2.6.5
> > -rw-r--r--  1 root root 1390233 2004-05-18 01:54 vmlinuz-2.6.6
> > -rw-r--r--  1 root root 1315050 2003-09-12 22:10 vmlinuz-2.6.0-test5
> > -rw-r--r--  1 root root 1280189 2003-06-06 01:00 vmlinuz-2.5.70
> > -rw-r--r--  1 root root  918663 2002-11-11 22:42 vmlinuz-2.5.47
> > -rw-r--r--  1 root root  887758 2004-02-19 01:24 vmlinuz-2.4.25
> > -rw-r--r--  1 root root  883868 2003-12-20 21:02 vmlinuz-2.4.23
> 
> What does size(1) say?

This is already stripped and compressed, so it's not directly available.

> Are you sure these kernels are feature-equivalent?

Pretty much, on this machine I generally only include what I need, so only 
a few drivers were added, I even have KALLSYMS disabled.

bye, Roman

Re: New (now current development process)

[Jesse Barnes]

On Monday, October 31, 2005 4:17 pm, Roman Zippel wrote:
> > Are you sure these kernels are feature-equivalent?
>
> Pretty much, on this machine I generally only include what I need, so
> only a few drivers were added, I even have KALLSYMS disabled.

Is it just one top level subsystem that's increasing in size faster than 
the others?  Last time I broke it down, networking (net/built-in.o) was 
the biggest by far, and it does seem to add features at a fast rate (not 
that I'm complaining!).  On FC devel with the FC kernel config:

-rw-rw-r--  1 jbarnes jbarnes 555088 Oct 31 16:33 net/built-in.o 
(stripped)

Jesse

Re: New (now current development process)

[Andrew Morton]

Andi Kleen <ak@suse.de> wrote:
>
> On Sunday 30 October 2005 20:12, Andrew Morton wrote:
> 
> > The freezes are for fixing bugs, especially recent regressions.  There's no
> > shortage of them, you know.
> >
> > I you can think of a better way to get kernel developers off their butts
> > and actually fixing bugs, I'm all ears.
> 
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a 
> are too risky or the impact too low because there is not enough time to 
> properly test anymore.

There is absolutely nothing preventing people from working on bugs at any
time!  It's not as if a bugfix can ever come too early.

> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.

Nobody knows when a kernel will be released, because it's released
according to perceived bug status, not according to a preconceived
timeline.

I just don't buy what you're saying.  Unless the kernel is at -rc3 or -rc4,
we *know* the release is weeks away - it's always been thus.

Re: New (now current development process)

[Rob Landley]

On Sunday 30 October 2005 18:48, Andi Kleen wrote:
> The problem is that you usually cannot do proper bug fixing because
> the release might be just around the corner, so you typically
> chose the ugly workaround or revert, or just reject changes for bugs that a
> are too risky or the impact too low because there is not enough time to
> properly test anymore.
>
> It might work better if we were told when the releases would actually
> happen and you don't need to fear that this not quite tested everywhere
> bugfix you're about to submit might make it into the gold kernel, breaking
> the world for some subset of users.

Hence the -mm tree, which takes stuff that may still need to be debugged.  
Except that it has this nasty habit of taking stuff which still needs to be 
debugged from people _other_than_you_, which screws you up.  You seem to want 
a tree where the only stuff likely to break is your stuff, which is another 
popular option: maintaining your own developer tree.  Getting people to _use_ 
such a tree takes a bit of work, but that's not news to anybody.

Think about what you're asking for here.  Imagine that other people _also_ get 
what you're asking for, at the same time.  Is it still what you want?

Right now patches go from developer tree, to -mm tree, to -linus tree, with a 
larger audience each time.  The _reason_ linus's tree has a larger audience 
is exactly _because_ the patches in it have had more testing so it's less 
likely to break.  And the releases have a way larger audience than Linus's 
-rc releases, and the distro kernels have a larger audience than that...

> -Andi

Rob

Re: New (now current development process)

[Andrew Morton]

Rob Landley <rob@landley.net> wrote:
>
> You seem to want 
>  a tree where the only stuff likely to break is your stuff

That's what I was thinking ;)

The simple fact is that we have more developers doing more stuff faster
than they used to.  All within a coupled system which has a lot of
interactions.

End result: yes, we do all need to spend more time looking at other
people's code and less time looking at our own.  That's just life in a
large project.

I'm very careful to make sure that relevant developers are copied on
patches which go into -mm.  In fact there's significantly better review
opportunity on patches which go developer->mm->Linus than there are on
patches which go developer->maintainer-git->Linus.

But the cc'ed people just _have_ to take time out to read the dang patch! 
They almost always have multiple weeks in which to do this.  But if they
just delete the thing while they work on their own stuff, well...

Re: New (now current development process)

[Rogério Brito]

On Oct 30 2005, Andrew Morton wrote:
> But the cc'ed people just _have_ to take time out to read the dang
> patch!  They almost always have multiple weeks in which to do this.
> But if they just delete the thing while they work on their own stuff,
> well...

Indeed. And there are the guinea pigs/lusers like me that are willing to
test and do some roundtrips of reports to get the kernel working better
(with the little that I can help). :-)

Now that I have backups working as I wanted, I surely intend to give the
-mm tree more tries.

Oh, and I would really appreciate if some options had at least some
description when I do a make menuconfig. Would identifying these options
or asking for clarification of others be of any interest?

I do think that they would help me when I try to compile the kernel for
some of my friends' computers.


Thanks, Rogério.

-- 
Rogério Brito : rbrito@ime.usp.br : http://www.ime.usp.br/~rbrito
Homepage of the algorithms package : http://algorithms.berlios.de
Homepage on freshmeat:  http://freshmeat.net/projects/algorithms/

Re: New (now current development process)

[Andrew Morton]

Rogério Brito <rbrito@ime.usp.br> wrote:
>
>  Oh, and I would really appreciate if some options had at least some
>  description when I do a make menuconfig. Would identifying these options
>  or asking for clarification of others be of any interest?

Yes please - people often seem to omit the help text.

Better still is to send a patch which adds the help text, of course.  If
it's a bit wrong, people will help sort that out.

Re: New (now current development process)

[Martin J. Bligh]

> That's what I was thinking ;)
> 
> The simple fact is that we have more developers doing more stuff faster
> than they used to.  All within a coupled system which has a lot of
> interactions.
> 
> End result: yes, we do all need to spend more time looking at other
> people's code and less time looking at our own.  That's just life in a
> large project.
> 
> I'm very careful to make sure that relevant developers are copied on
> patches which go into -mm.  In fact there's significantly better review
> opportunity on patches which go developer->mm->Linus than there are on
> patches which go developer->maintainer-git->Linus.

Moreover, it's fairly easy to test stuff that's all in one place, in a
consistent format, with a simple linear stack of patches to sort through 
to find culprits.

Plus you have a great tendency of dropping stuff like a stone when it's
broken, which helps a lot. Having some basic pre-mainline-merge testing
keeps the quality of mainline way up.

It'd help more if people focused more on testing their own shit before
submitting it than complaining about -mm. If it's the same people breaking 
the tree all the time, I'm sure we can find a recycled set of stocks 
somewhere.

M.

Re: New (now current development process)

[Theodore Ts'o]

On Sun, Oct 30, 2005 at 12:29:28AM +0200, Andi Kleen wrote:
> Please don't. Even the two weeks are too short IMHO, because it is
> hard to digest so much code in such a short time and also it is not
> always easy for maintainers to hit such short time windows for sending
> patches.
> 
> > I don't think anybody has been really unhappy with this approach? Hmm?
> 
> The long freeze periods were nothing much happens are painful. It
> would be better to have some more overlap of merging and stabilizing
> (stable does that already kind of, but not enough)

I thought Andrew was accepting patches targeted at 2.6.n+1 into the
-mm tree during the freeze periods, yes?  If so, why would it be a
case of "nothing much happens"?  Nothing much might be happening in
Linus's git tree, but that doesn't that they can't be happening in
Andrew's -mm patchsets....

						- Ted

Re: New (now current development process)

[Andi Kleen]

On Sunday 30 October 2005 22:32, Theodore Ts'o wrote:

> I thought Andrew was accepting patches targeted at 2.6.n+1 into the
> -mm tree during the freeze periods, yes?  If so, why would it be a
> case of "nothing much happens"?  Nothing much might be happening in
> Linus's git tree, but that doesn't that they can't be happening in
> Andrew's -mm patchsets....

The problem is that -mm* contains typically so many more or less
broken changes that any extensive work on there is futile 
because you never know whose bugs you're debugging
(and if the patch that is broken will even make it anywhere) 

In short mainline is frozen too long and -mm* is too unstable.

-Andi

Re: New (now current development process)

[Al Viro]

On Mon, Oct 31, 2005 at 01:45:43AM +0100, Andi Kleen wrote:
> The problem is that -mm* contains typically so many more or less
> broken changes that any extensive work on there is futile 
> because you never know whose bugs you're debugging
> (and if the patch that is broken will even make it anywhere) 
> 
> In short mainline is frozen too long and -mm* is too unstable.

Besides, -mm is changing so fscking fast that it doesn't build on a lot
of configs most of the time.  And trying to keep track of it and at least
deal with build breakage at real time is, IME, hopeless.

Re: New (now current development process)

[Paul Jackson]

Al, responding to Andi,
> > because you never know whose bugs you're debugging
> > (and if the patch that is broken will even make it anywhere) 
> > 
> > In short mainline is frozen too long and -mm* is too unstable.
> 
> Besides, -mm is changing so fscking fast that it doesn't build on a lot
> of configs most of the time.

I think you are exagerating.

It builds on most configs most of the time in my experience.  If I
haven't tried a crosstool rebuild of the several defconfig arch's in a
week, I might expect one of the less popular archs to drop out, usually
for something so easy even I can figure some sort of fix or workaround.

It's tough to get stable kernels if the only kernels people want to
test on are stable kernels.  Linux is benefiting immensely from its
rapid evolution in various directions.

Granted - build and boot tested versions of *-mm, just a couple days
behind Andrew's patch set releases, would save people of dealing with
some of the simply stupid breakage.

Is there any corporate Linux supporter able to fund that?  Clearing
off the simple breakage could help attract the more serious and
diverse testing of a wider group of users with various specialized
interests.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: New (now current development process)

[Al Viro]

On Sun, Oct 30, 2005 at 07:14:02PM -0800, Paul Jackson wrote:
> I think you are exagerating.
> 
> It builds on most configs most of the time in my experience.  If I
> haven't tried a crosstool rebuild of the several defconfig arch's in a
> week, I might expect one of the less popular archs to drop out, usually
> for something so easy even I can figure some sort of fix or workaround.

Try allmodconfig for a change...  I'm doing that for mainline on a regular
basis and even that turns into considerable amount of time.  I have tried
that for -mm and had to give up.

Re: New (now current development process)

[Paul Jackson]

Al Viro wrote:
> Try allmodconfig for a change... 

;).  That's ok.  What I do keeps me busy enough.

Still, it's too bad one of Linux's corporate friends can't find a way
to do this sort of thing, providing rapid turn around on build-blessed
*-mm versions.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: New (now current development process)

[Andrew Morton]

Al Viro <viro@ftp.linux.org.uk> wrote:
>
> On Sun, Oct 30, 2005 at 07:14:02PM -0800, Paul Jackson wrote:
> > I think you are exagerating.
> > 
> > It builds on most configs most of the time in my experience.  If I
> > haven't tried a crosstool rebuild of the several defconfig arch's in a
> > week, I might expect one of the less popular archs to drop out, usually
> > for something so easy even I can figure some sort of fix or workaround.
> 
> Try allmodconfig for a change...  I'm doing that for mainline on a regular
> basis and even that turns into considerable amount of time.  I have tried
> that for -mm and had to give up.

fud.  Every -mm release is built with allmodconfig on x86 and on x86_64. 
It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64,
arm and ia64.  It's booted on x86, x86_64, ppc64 and ia64.  Every release.

Yes, some tty drivers have been busted in recent releases, as described in
the release notices.  That's why the tty patches are at the end of the -mm
series: so I can back them off for the allmodconfig builds.

I usually do allnoconfig and allyesconfig, too.

Re: New (now current development process)

[Al Viro]

On Sun, Oct 30, 2005 at 11:22:34PM -0800, Andrew Morton wrote:
> Al Viro <viro@ftp.linux.org.uk> wrote:
> >
> > On Sun, Oct 30, 2005 at 07:14:02PM -0800, Paul Jackson wrote:
> > > I think you are exagerating.
> > > 
> > > It builds on most configs most of the time in my experience.  If I
> > > haven't tried a crosstool rebuild of the several defconfig arch's in a
> > > week, I might expect one of the less popular archs to drop out, usually
> > > for something so easy even I can figure some sort of fix or workaround.
> > 
> > Try allmodconfig for a change...  I'm doing that for mainline on a regular
> > basis and even that turns into considerable amount of time.  I have tried
> > that for -mm and had to give up.
> 
> fud.  Every -mm release is built with allmodconfig on x86 and on x86_64. 
> It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64,
> arm and ia64.  It's booted on x86, x86_64, ppc64 and ia64.  Every release.

What fud?  I stand by my claim - I have tried to do the same thing for
-mm and found that I didn't have guts for that; too much work.  For mainline
I do cross-builds for allmodconfig on a *lot* more targets than what you've
mentioned and generally it stays within ~150-200Kb of patches, about half
of that being a fix for 8390 mess.

_IF_ somebody wants to do that for -mm, yell and you are more than welcome
to all infrastructure, except for the cycles on build box I'm using.
Incidentally, it is a box at work - my energy bill is high enough as it
is, without adding an 8-way 3GHz iamd64 to it...

The last time I've attempted that for -mm was this summer; right now mainline
is quite enough work, TYVM...  Especially since I use the same tree as a
staging point for annotations and watch for build regressions both for gcc
and sparse.

Re: New (now current development process)

[Paul Jackson]

Al Viro wrote:
> I do cross-builds for allmodconfig on a *lot* more targets than what you've
> mentioned

Apples to oranges, and half full versus half empty.

If Al is testing combinations that Andrew doesn't regularly test,
then Al might expect to find problems that Andrew didn't find.

Non-specific complaints that make it all sound too hard are
not particularly constructive.  But then Al was trying for the
"particularly constructive" achievement award of the day ;).

I find that the defconfig crosstool builds of *-mm more or less always
work for the popular configurations.  That meets my needs just fine.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: New (now current development process)

[Martin J. Bligh]

>> fud.  Every -mm release is built with allmodconfig on x86 and on x86_64. 
>> It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64,
>> arm and ia64.  It's booted on x86, x86_64, ppc64 and ia64.  Every release.
> 
> What fud?  I stand by my claim - I have tried to do the same thing for
> -mm and found that I didn't have guts for that; too much work.  For mainline
> I do cross-builds for allmodconfig on a *lot* more targets than what you've
> mentioned and generally it stays within ~150-200Kb of patches, about half
> of that being a fix for 8390 mess.
> 
> _IF_ somebody wants to do that for -mm, yell and you are more than welcome
> to all infrastructure, except for the cycles on build box I'm using.
> Incidentally, it is a box at work - my energy bill is high enough as it
> is, without adding an 8-way 3GHz iamd64 to it...

I'll do that if you want. I have a big lab full of largish boxes with 
serious aircon, and IBM can afford the power bill. I'm assuming this 
is a farm of cross-compilers that'll run on x86 (or x86_64)?

M.

Re: New (now current development process)

[Martin J. Bligh]

--Al Viro <viro@ftp.linux.org.uk> wrote (on Monday, October 31, 2005 00:18:10 +0000):

> On Mon, Oct 31, 2005 at 01:45:43AM +0100, Andi Kleen wrote:
>> The problem is that -mm* contains typically so many more or less
>> broken changes that any extensive work on there is futile 
>> because you never know whose bugs you're debugging
>> (and if the patch that is broken will even make it anywhere) 
>> 
>> In short mainline is frozen too long and -mm* is too unstable.
> 
> Besides, -mm is changing so fscking fast that it doesn't build on a lot
> of configs most of the time.  And trying to keep track of it and at least
> deal with build breakage at real time is, IME, hopeless.

To be fair, it's been a lot better the last month than it has the 2 before
that, when it was really pretty scary.

If it helps, we can try to build a wider range of configs to test. And I 
can try to hook Andrew up with something he can do a quick compile-farm 
test on before release (though I think he has his own battery of stuff).

M.

Re: New (now current development process)

[Rob Landley]

On Sunday 30 October 2005 18:45, Andi Kleen wrote:
> The problem is that -mm* contains typically so many more or less
> broken changes that any extensive work on there is futile
> because you never know whose bugs you're debugging
> (and if the patch that is broken will even make it anywhere)
>
> In short mainline is frozen too long and -mm* is too unstable.

Are you implying that if mainline wasn't frozen so much, it would still be 
more stable than -mm?

Rob

Re: New (now current development process)

[Andreas Kleen]

Am Mo 31.10.2005 05:52 schrieb Rob Landley <rob@landley.net>:

> On Sunday 30 October 2005 18:45, Andi Kleen wrote:
> > The problem is that -mm* contains typically so many more or less
> > broken changes that any extensive work on there is futile
> > because you never know whose bugs you're debugging
> > (and if the patch that is broken will even make it anywhere)
> >
> > In short mainline is frozen too long and -mm* is too unstable.
>
> Are you implying that if mainline wasn't frozen so much, it would
> still be
> more stable than -mm?

Yes. Historically 2.<odd> kernels were like this.
 
-Andi
 

Re: New (now current development process)

[Tony Luck]

On 10/29/05, Linus Torvalds <torvalds@osdl.org> wrote:
> But to hit that, we'd might need to shrink the "merge window" from two
> weeks to just one, otherwise there's not enough time to calm down.

The problem with just one week to merge would be if other stuff
happened that ate up the merge period (e.g. this coming week I'm
travelling and will spend all day Tuesday and Wednesday in meetings).

It would be easier to manage if I could be sure which week was going
to be the merge window ... but I know that it is hard to predict when
a release will happen.  Overall I'd be much happier sticking with a
two week window.

-Tony

Re: New (now current development process)

[Willy Tarreau]

On Sat, Oct 29, 2005 at 01:28:23PM -0700, Linus Torvalds wrote:
 
> So I'm planning on continuing with it unchanged for now. Two-week merge 
> window until -rc1, and then another -rc kernel roughly every week until 
> release. With the goal being 6 weeks, and 8 weeks being ok.
> 
> I don't think anybody has been really unhappy with this approach? Hmm?

I believe it was a good experience. However, I still find it sad that
there are changes between the latest -rc and the final version. This
time, it seems it did not cause trouble, but in the past, it happened
several times, because any valid fix can have side effects.

I think that if you could announce what you intend to release with a
message like "I will make this -final tomorrow", there would be some
time to test builds in various configs, and check that no obvious bug
has been introduced.

Regards,
Willy

Re: New (now current development process)

[Eric Sandall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 31 Oct 2005, Willy Tarreau wrote:
> On Sat, Oct 29, 2005 at 01:28:23PM -0700, Linus Torvalds wrote:
>
>> So I'm planning on continuing with it unchanged for now. Two-week merge
>> window until -rc1, and then another -rc kernel roughly every week until
>> release. With the goal being 6 weeks, and 8 weeks being ok.
>>
>> I don't think anybody has been really unhappy with this approach? Hmm?
>
> I believe it was a good experience. However, I still find it sad that
> there are changes between the latest -rc and the final version. This
> time, it seems it did not cause trouble, but in the past, it happened
> several times, because any valid fix can have side effects.
>
> I think that if you could announce what you intend to release with a
> message like "I will make this -final tomorrow", there would be some
> time to test builds in various configs, and check that no obvious bug
> has been introduced.

A -final should never be changed from the last -rc. That defeats the
purpose of having -rc releases (rc == 'release candidate' ;)). If you
do make changes to the last -rc, then you need to release another -rc
with those changes. If all's good, then you can release that /last/
- -rc with /no/ changes as the One and True Release.

- -sandalle

- --
Eric Sandall                     |  Source Mage GNU/Linux Developer
eric@sandall.us                  |  http://www.sourcemage.org/
http://eric.sandall.us/          |  SysAdmin @ Inst. Shock Physics @ WSU
http://counter.li.org/  #196285  |  http://www.shock.wsu.edu/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDbt4KHXt9dKjv3WERAiIXAKClGeeapTkGJ7gFWymHISb99JGw7QCgrPag
lArvgGMUprksA28CX+9I2Bc=
=8xAI
-----END PGP SIGNATURE-----

Re: New (now current development process)

[Krzysztof Halasa]

Eric Sandall <eric@sandall.us> writes:

> A -final should never be changed from the last -rc. That defeats the
> purpose of having -rc releases (rc == 'release candidate' ;)).

This logic is flawed. RCs are for performing tests. If you don't want
further tests (for example, tests on previous RC completed and you're
quite sure new changes introduce no new bugs) you don't need further
RCs.
-- 
Krzysztof Halasa

Re: New (now current development process)

[Christopher Friesen]

Krzysztof Halasa wrote:
> Eric Sandall <eric@sandall.us> writes:

>>A -final should never be changed from the last -rc. That defeats the
>>purpose of having -rc releases (rc == 'release candidate' ;)).
> 
> 
> This logic is flawed. RCs are for performing tests. If you don't want
> further tests (for example, tests on previous RC completed and you're
> quite sure new changes introduce no new bugs) you don't need further
> RCs.

How do you ever know that new change introduced no new bugs?  Maybe 
there was a latent race condition that is activated by timing 
differences caused by the new code.  Maybe it shifts the spacing of the 
code just enough to get hit by a pre-existing trampler.  Unless you test 
it, you *can't* know.

The safe bet is to simply rename the final -rc with no further changes.

Chris

Re: New (now current development process)

[Linus Torvalds]

On Mon, 7 Nov 2005, Christopher Friesen wrote:
> 
> The safe bet is to simply rename the final -rc with no further changes.

That's not actually a very safe bet at all.

Why? Because only a small percentage of people actually test -rc kernels 
in the first place. 

So if you release the last -rc as the standard kernel, you (a) get no real 
coverage advantage anyway and (b) you just wasted a week in order to try 
to get some coverage that you aren't getting.

The current kernel development model is to merge stuff early, which 
hopefully motivates the people who _do_ test -rc kernels to actually test 
-rc1, since they know that that's the one that has _all_ the really 
relevant goodies.

And most of those that do test -rc1 will never see any problems at all. 
Those that do, are now more likely to test the rest of the -rcs, hopefully 
until their problem is resolved. And those that don't test -rc releases 
(because they simply don't upgrade very often) will _never_ test an -rc 
release, whether it's the first one or the last one.

So what we have to fight this problem is the stable kernel series. We 
release the final 2.6.x kernel with as much testing as we can, but it's 
just an undeniable fact that a lot of people will try that kernel only 
after the release, and often it might be weeks after the release. Doing 
-rc kernels didn't do anything for those cases.

But when they find a problem (or somebody who _did_ test an -rc kernel, 
but didn't notice a problem until much later), we try to have a process to 
get those issues fixed. 

So repeat after me: "Most people never test -rc kernels". 

			Linus

Re: New (now current development process)

[Linus Torvalds]

On Mon, 7 Nov 2005, Linus Torvalds wrote:
> 
> So repeat after me: "Most people never test -rc kernels". 

Btw, the ones that _do_ test -rc kernels usually don't test all of them. 
The current model is set up in a way where there is _one_ special -rc 
kernel that we should try to get people to test: the first one.

That hopefully encourages people to try an -rc kernel who might otherwise 
decide that there's too many -rc kernels to bother with. If they know that 
all of the real development happened before -rc1, they also are thus aware 
that it doesn't really matter which -rc kernel they test, so just testing 
_one_ is very good indeed.

The first -rc kernel is also special in another way: it's the one we 
"wait" for. It's the one that happens after two weeks, and has a deadline. 
The others happen more frequently, and are really objectively less 
important than the first one.

(In contrast, some other projects try to make the _last_ -rc be the 
important one. That's totally the wrong way around, because if there are 
more people testing the last one, the testing happens at _exactly_ the 
wrong point in time from a "let's fix the problems" standpoint)

So the call to people who can be bothered to test at all: if you 
only test one -rc kernel, please test the first one. That way we get a 
heads-up on problems earlier.

(And if you like testing -rc kernels, please test all of them. Or even the 
nightly snapshots. Or track the git tree several times a day. The more, 
the merrier, but if you only want to boot one kernel a month, make it be 
the -rc1 kernel).

			Linus

Re: New (now current development process)

[Willy Tarreau]

Hi Linus,

you made valid points, you know your public very well :-)
However, I have an objection on one point below.

On Mon, Nov 07, 2005 at 09:28:35AM -0800, Linus Torvalds wrote:
> On Mon, 7 Nov 2005, Linus Torvalds wrote:
> > 
> > So repeat after me: "Most people never test -rc kernels". 
> 
> Btw, the ones that _do_ test -rc kernels usually don't test all of them. 

That's true at least for me, except that I *try* to test *ALL* versions which
are announced as the most likely future -final. When Marcelo tells us that
-rc2 will be -final if nobody complains, I really test it. I build it on
several archs and try to catch stupid bugs because I hate stupid bugs in
final releases, they pollute bug reports (worst ones being build errors).

However, when he announces -preX (X>1) or when you annonce any -rc which
is not likely to become -final, I just check the changelog, and build it
if I both see something which applies to my setup, and I have nothing else
to do.

When I ask you to turn -rc into -final, it's not just to avoid adding
new bugs (eventhough the most quickly fixed bugs may be the most dangerous),
but it's mainly to add credibility to the call so that many people who would
be hesitant will try it.

> The current model is set up in a way where there is _one_ special -rc 
> kernel that we should try to get people to test: the first one.
>
> That hopefully encourages people to try an -rc kernel who might otherwise 
> decide that there's too many -rc kernels to bother with. If they know that 
> all of the real development happened before -rc1, they also are thus aware 
> that it doesn't really matter which -rc kernel they test, so just testing 
> _one_ is very good indeed.

Agreed too. That's the same reason I asked Marcelo if he would agree to
merge all the uncertain recent mcast fixes early in 2.4.33-pre1.

> The first -rc kernel is also special in another way: it's the one we 
> "wait" for. It's the one that happens after two weeks, and has a deadline. 
> The others happen more frequently, and are really objectively less 
> important than the first one.
> 
> (In contrast, some other projects try to make the _last_ -rc be the 
> important one. That's totally the wrong way around, because if there are 
> more people testing the last one, the testing happens at _exactly_ the 
> wrong point in time from a "let's fix the problems" standpoint)

Are there really projects managed like this ? I hope I don't use anything
from them !!!

> So the call to people who can be bothered to test at all: if you 
> only test one -rc kernel, please test the first one. That way we get a 
> heads-up on problems earlier.

Agreed here too from the developper standpoint. However, your final release
are references for all trees. It's better if the common ancestor between
many trees at least builds and does not propagate one horrible bug discovered
10 minutes after release. The least differences there are in code base between
tree, the most relevant the bug reports are. And yes, I know that there is
-stable for this, but most people don't know about it and will simply build
the 2.6.15 announced on slashdot then be the 100th to complain that 2.6.15
does not build because of this or that. (I hope it will not happen :-))

> (And if you like testing -rc kernels, please test all of them. Or even the 
> nightly snapshots. Or track the git tree several times a day. The more, 
> the merrier, but if you only want to boot one kernel a month, make it be 
> the -rc1 kernel).

I consider two rcs as important :
  - the first one, to test bugs in features
  - the last one to increase the signal/noise ration in bug reports.

Then it makes sense to test them both. However, I agree that if only one can
be tested, the first one is most interesting to us.

> 			Linus

Regards,
Willy

Re: New (now current development process)

[Krzysztof Halasa]

"Christopher Friesen" <cfriesen@nortel.com> writes:

> How do you ever know that new change introduced no new bugs?

Changing comments doesn't change generated code or your tools are
screwed. Trivial changes - you can be reasonably sure, too. Note
it's different from not having bugs at all.

>  Maybe
> there was a latent race condition that is activated by timing
> differences caused by the new code.

Then the bug already existed, you aren't adding any.

> The safe bet is to simply rename the final -rc with no further changes.

Safe? You can't be really safe here. Bad luck and version string
change (compile time or -rc* removal) will trigger disaster.
-- 
Krzysztof Halasa

Re: New (now current development process)

[Jesper Juhl]

On 10/29/05, Paolo Ciarrocchi <paolo.ciarrocchi@gmail.com> wrote:
> Hi all,
> I would like to write a short article about the new development
> process discussed during last Linux Kernel Developers' Summit for my
> local LUG.
>
> Since I'm not able to find an accurate report of what has been
> discussed during that meeting I try to summariza what is my
> understanding of the current process:
>
> The are two kind of releases, 2.6.x kerneles and 2.6.x.y
>
I recently wrote a document on the different kernel trees and how to
apply patches for them. In that document I also give a description of
the various trees.
Perhaps that document will be useful to you. You can find it in a
recent kernel source tree as Documentation/applying-patches.txt or you
can read it online via lrx at this URL :
http://sosdg.org/~coywolf/lxr/source/Documentation/applying-patches.txt

Good luck on your article. Once you are done with it I'd love to read
it and I guess a lot of other people would too, so please supply an
URL to it at that time :-)

--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html