2.5.19 - What's up with the kernel build?

2.5.19 - What's up with the kernel build?

[Kai Germaschewski]

Okay, as people probably already noticed, I touched quite a lot of
Makefiles lately. Contrary to what the patches (which touched many
files) might imply, not much did actually change, in particular not
much visible outside of the kernel build process.

There is still quite a bit left to do (in particular improving
dependency generation and modversions handling), but I think it makes
sense to explain what happened so far.

There's also some points (marked with >>>) where I'd like to get
feedback on how things should be handled in the future.


For users (i.e. people who compile kernels)
-------------------------------------------
Not much changed at all. Do what you always did, it should still work
the same.

o vmlinux/(b)zImage etc. and modules can now be built in the same
  pass.
  For testing this feature, please use

      make BUILD_MODULES=1 vmlinuz/(b)zImage/whatever,

  i.e. add the "BUILD_MODULES=1" to the command line you normally use,
  and you save the additional "make modules" run.

>>> What do you guys think how this should be handled in the future? -
>>> my suggestion would be to make this behavior the default, and, in
>>> case someone needs it, add the option to say "BUILD_MODULES=0".
  
o you can ask to build a single target, like 

      make some/dir/foo.[iso]

  and it'll give you preprocessed/assembler/object file output, provided
  that some/dir/foo.c or some/dir/foo.S exists. It should get all cases
  correct (well, except for arch/i386/kernel/{head.o,init_task.o}, which
  will cause an entire recursive run), with adding the right flags,
  compiling as modular etc.

o when your source tree changed only little or not at all, performance
  should be quite a bit improved. If you do "make BUILD_MODULES=1 
  vmlinux", the number of "make" invocations should be down by about a 
  factor of three compared to a previous "make vmlinux modules". Also, 
  vmlinux will not be relinked if nothing actually changed. (If lots of 
  files changed, the time make consumes will be down in the noise, the 
  build time will be dominated by gcc).

>>> It's possible with only small changes to provide a quiet mode now,
>>> which would not print the entire command lines but only
>>>
>>>	  Descending into drivers/isdn/kcapi
>>>	  Compiling kcapi.o
>>>	  Compiling capiutil.o
>>>	  Linking kernelcapi.o
>>>	  ...
>>>
>>> Is that considered useful?

For developers:
---------------

If you had clean Makefiles for your subsystem/driver/whatever before,
again, not much changed.

o "list-multi" + associated link rules are gone for good. It's been
  like that for a couple of months now, so that's probably no news
  for you.

o Descending into and linking subdirs was often handled with constructs
  like the following:

       subdir-$(CONFIG_E100) += e100
       ifeq ($(CONFIG_E100),y)
         obj-y += e100/e100.o
       endif 

  This can now be expressed in a much more compact way as

       obj-$(CONFIG_E100) += e100/

  which basically means the exact same thing:
  - if CONFIG_E100=y, descend into the subdir and link the result
  - if CONFIG_E100=m, descend into the subdir and build the modules there
  - otherwise, ignore that subdir

  Only requirement is that the O_TARGET in the subdir (that's what getting
  linked in, "e100.o" in the example above) is named "built-in.o". To
  simplify things even further, just leaving out the "O_TARGET := ..."
  line does the right thing, it uses "built-in.o" as default. Anyway,
  the conversion is mostly done now, so you don't have to actually deal
  with it.

o Once we make "module and built-in in one pass" default, the

       mod-subdirs := ...

  lines can go, too. Nobody understood what they're there for anyway,
  right? ;-)

For arch maintainers:
---------------------

o In the arch specific Makefiles, you needed to build not only 
  O_TARGET (or L_TARGET) out of $(obj-y) and the modules 
  in $(obj-m), but also additional objects, e.g. for i386: 
  "head.o" and "init_task.o"

  This can (and should) be done cleanly now, by adding such targets to
  the variable EXTRA_TARGETS instead of playing games with overriding
  Rules.make's "first_rule:"

  The conversion is basically done, so you don't really need to
  bother.

o Rules.make provides standard rules for assembling files now,

  So use them - again, the conversion is done already, just don't undo
  it.



BTW, yeah, I know, Documentation/kbuild/makefiles.txt could use an
update. Will do that...


--Kai

Re: 2.5.19 - What's up with the kernel build?

[Paul P Komkoff Jr]

Replying to Kai Germaschewski:
> There is still quite a bit left to do (in particular improving
> dependency generation and modversions handling), but I think it makes
> sense to explain what happened so far.
> 
> There's also some points (marked with >>>) where I'd like to get
> feedback on how things should be handled in the future.

May I ask you just one questin?
Have you read (yet) kbuild25 ?

Ahh, no, another one.
Is this a "signs" of kbuild25 being thrown away like cml2 ?
If yes, then I am very unhappy person now (c) ac

Played with kbuild25 today I migrated 2.4.19-pre8-ac5-s2 to it. Now make -f
Makefile-2.5 is a preferred way to make here. New system is much cleaner,
and don't need that mess of makedep and listmultis. But hey, people you
should already know that whilst you trying to install a couple of new nails
to rotten construct to help it stay for another couple of time intervals ...

:(((


-- 
Paul P 'Stingray' Komkoff 'Greatest' Jr /// (icq)23200764 /// (http)stingr.net
  When you're invisible, the only one really watching you is you (my keychain)

Re: 2.5.19 - What's up with the kernel build?

[Jeff Garzik]

Paul P Komkoff Jr wrote:

>Replying to Kai Germaschewski:
>  
>
>>There is still quite a bit left to do (in particular improving
>>dependency generation and modversions handling), but I think it makes
>>sense to explain what happened so far.
>>
>>There's also some points (marked with >>>) where I'd like to get
>>feedback on how things should be handled in the future.
>>    
>>
>
>May I ask you just one questin?
>Have you read (yet) kbuild25 ?
>
>Ahh, no, another one.
>Is this a "signs" of kbuild25 being thrown away like cml2 ?
>If yes, then I am very unhappy person now (c) ac
>
>Played with kbuild25 today I migrated 2.4.19-pre8-ac5-s2 to it. Now make -f
>Makefile-2.5 is a preferred way to make here. New system is much cleaner,
>and don't need that mess of makedep and listmultis. But hey, people you
>should already know that whilst you trying to install a couple of new nails
>to rotten construct to help it stay for another couple of time intervals ...
>  
>


Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
he wanted an evolution to a new build system... not an unreasonable 
request to at least consider.  Despite Keith's quality of code (again -- 
I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
very "take it or leave it dammit".  Not the best way to win friends and 
influence people.

If Keith is indeed leaving it, I'm hoping someone will maintain it, or 
work with Kai to integrate it into 2.5.x.

    Jeff

Re: 2.5.19 - What's up with the kernel build?

[Nicolas Pitre]

On Wed, 29 May 2002, Jeff Garzik wrote:

> Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
> he wanted an evolution to a new build system... not an unreasonable 
> request to at least consider.  Despite Keith's quality of code (again -- 
> I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
> very "take it or leave it dammit".  Not the best way to win friends and 
> influence people.
> 
> If Keith is indeed leaving it, I'm hoping someone will maintain it, or 
> work with Kai to integrate it into 2.5.x.

When I suggested to Keith he push kbuild25 the way Linus likes, he (Keith) 
considered that was a "stupid comment" and that he'd ignore stupid comments.

So it looks like someone else will have to volunteer to split kbuild25 into
multiple small patches and feed them "piecemeal" to Linus before we ever see
it into the kernel tree.


Nicolas

Re: 2.5.19 - What's up with the kernel build?

[Thunder from the hill]

Hi,

On Wed, 29 May 2002, Nicolas Pitre wrote:
> So it looks like someone else will have to volunteer to split kbuild25 into
> multiple small patches and feed them "piecemeal" to Linus before we ever see
> it into the kernel tree.

Well, I would, but... wasn't Kai on that already?

Regards,
Thunder
-- 
Was it a black who passed along in the sand?
Was it a white who left his footprints?
Was it an african? An indian?
Sand says, 'twas human.

Re: 2.5.19 - What's up with the kernel build?

[Nicolas Pitre]

On Wed, 29 May 2002, Thunder from the hill wrote:

> Hi,
> 
> On Wed, 29 May 2002, Nicolas Pitre wrote:
> > So it looks like someone else will have to volunteer to split kbuild25 into
> > multiple small patches and feed them "piecemeal" to Linus before we ever see
> > it into the kernel tree.
> 
> Well, I would, but... wasn't Kai on that already?

No.

Kai is only banging on the current build system to make it somewhat more
palatable.  While this is certainly useful today, those efforts would
probably produce a better system in the long run if they were directed at
kbuild25 instead.

Or maybe people just don't care enough about the build system for kbuild25
to be worth it...


Nicolas

Re: 2.5.19 - What's up with the kernel build?

[Daniel Phillips]

On Thursday 30 May 2002 01:18, Nicolas Pitre wrote:
> On Wed, 29 May 2002, Thunder from the hill wrote:
> > 
> > On Wed, 29 May 2002, Nicolas Pitre wrote:
> > > So it looks like someone else will have to volunteer to split kbuild25 into
> > > multiple small patches and feed them "piecemeal" to Linus before we ever see
> > > it into the kernel tree.
> > 
> > Well, I would, but... wasn't Kai on that already?
> 
> No.
> 
> Kai is only banging on the current build system to make it somewhat more
> palatable.  While this is certainly useful today, those efforts would
> probably produce a better system in the long run if they were directed at
> kbuild25 instead.
> 
> Or maybe people just don't care enough about the build system for kbuild25
> to be worth it...

Omigod, don't even think that.  Kbuild 2.5 is faster and better than the
current kbuild.  I, for one, am waiting - impatiently - for the thing to get
in the tree.  The current build system is slow and unreliable, and don't even
think of trying to build two different architectures in the same source tree
at the same time.

-- 
Daniel

Re: 2.5.19 - What's up with the kernel build?

[Tomas Szepe]

> > Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
> > he wanted an evolution to a new build system... not an unreasonable 
> > request to at least consider.  Despite Keith's quality of code (again -- 
> > I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
> > very "take it or leave it dammit".  Not the best way to win friends and 
> > influence people.
> > 
> > If Keith is indeed leaving it, I'm hoping someone will maintain it, or 
> > work with Kai to integrate it into 2.5.x.
> 
> When I suggested to Keith he push kbuild25 the way Linus likes, he (Keith) 
> considered that was a "stupid comment" and that he'd ignore stupid comments.

What remains to be answered is, how does one split a system of a myriad of
build rule files into a reasonable amount of small patches.

Of course, you could have hundreds of patches each consisting of a single
Makefile.in, but how would that make the reviewing/integrating easier? In
the end you'd end up reading the same input, only you'd complement it by
frequently pressing your favorite show-me-the-next-mail key.

The solution here is not to create "artificial splitting," but rather spare
the good ol' system for the time being and have kbuild25 coexist with it until
all build issues are resolved and everything works acceptably -- and that's
what has been offered. 2.5 is certain to span a long enough period for such
treatment, and Keith will be so kind as to keep updating his work.

Alright I really didn't want to get involved in another kbuild thread but
couldn't help it. Sorry.


T.

Re: 2.5.19 - What's up with the kernel build?

[Daniel Phillips]

On Thursday 30 May 2002 01:06, Tomas Szepe wrote:
> > > Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
> > > he wanted an evolution to a new build system... not an unreasonable 
> > > request to at least consider.  Despite Keith's quality of code (again -- 
> > > I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
> > > very "take it or leave it dammit".  Not the best way to win friends and 
> > > influence people.

OK that's true, but think: how much work has Keith put into this?  How much
did he or his employer get paid?  And how many times has he been told to go
off and fix something, as a prelude to getting the thing in?  The last time
it was the first-time build speed.  He went away and came back with a *huge*
improvement, even more than what people were demanding.  You'd think that
would be enough.

Keith has to do *two* full time jobs as long as the patch isn't merged:
developing the patch itself and tracking the whole 2.5 tree as it (rapidly)
evolves.

> > > If Keith is indeed leaving it, I'm hoping someone will maintain it, or 
> > > work with Kai to integrate it into 2.5.x.
> > 
> > When I suggested to Keith he push kbuild25 the way Linus likes, he (Keith) 
> > considered that was a "stupid comment" and that he'd ignore stupid comments.

It is of course always regrettable when one is so rash as to call a stupid
comment a stupid comment ;-)

> What remains to be answered is, how does one split a system of a myriad of
> build rule files into a reasonable amount of small patches.
> 
> Of course, you could have hundreds of patches each consisting of a single
> Makefile.in, but how would that make the reviewing/integrating easier? In
> the end you'd end up reading the same input, only you'd complement it by
> frequently pressing your favorite show-me-the-next-mail key.

I thought BitKeeper was supposed to be able to deal with precisely this sort
of merge problem.  In this case, splitting the thing up just seems
unnatural, and a dubious use of time.

-- 
Daniel

Re: 2.5.19 - What's up with the kernel build?

[Nicolas Pitre]

On Thu, 30 May 2002, Daniel Phillips wrote:

> On Thursday 30 May 2002 01:18, Nicolas Pitre wrote:
> 
> > Or maybe people just don't care enough about the build system for kbuild25
> > to be worth it...
> 
> Omigod, don't even think that.  Kbuild 2.5 is faster and better than the
> current kbuild.  I, for one, am waiting - impatiently - for the thing to get
> in the tree.  The current build system is slow and unreliable, and don't even
> think of trying to build two different architectures in the same source tree
> at the same time.

I totally agree with you.

Then let me rephrase it that way: maybe not enough people care enough about
the build system for kbuild25 to be worth it...

> > > > Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
> > > > he wanted an evolution to a new build system... not an unreasonable 
> > > > request to at least consider.  Despite Keith's quality of code (again -- 
> > > > I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
> > > > very "take it or leave it dammit".  Not the best way to win friends and 
> > > > influence people.
> 
> OK that's true, but think: how much work has Keith put into this?  How much
> did he or his employer get paid?  And how many times has he been told to go
> off and fix something, as a prelude to getting the thing in?  The last time
> it was the first-time build speed.  He went away and came back with a *huge*
> improvement, even more than what people were demanding.  You'd think that
> would be enough.

But is it really enough?

Linus himself once said: "Especially as I don't find the existign system so
broken." He's not alone according to the amount (or lack) of public
complains with regards to the current system.

> > Of course, you could have hundreds of patches each consisting of a single
> > Makefile.in, but how would that make the reviewing/integrating easier? In
> > the end you'd end up reading the same input, only you'd complement it by
> > frequently pressing your favorite show-me-the-next-mail key.
> 
> I thought BitKeeper was supposed to be able to deal with precisely this sort
> of merge problem.  In this case, splitting the thing up just seems
> unnatural, and a dubious use of time.

Indeed, I agree.

But until there is enough people committed to kb25 to the point of pushing
for its wider adoption, either by releasing traditional kernel patches (like
the -dj, -aa, -ac or whatever) actually carrying kb25 so to convince Linus
and others, or producing "piecemeal" patches so it can be merged by Linus
with less resistance, well nothing will hapen.

If people aren't interested enough and/or willing to comply with Linus'
requirements this will be a sad dead end, regardless the amount of effort
Keith put into this.


Nicolas

Re: 2.5.19 - What's up with the kernel build?

[Daniel Phillips]

On Thursday 30 May 2002 02:17, Nicolas Pitre wrote:
> On Thu, 30 May 2002, Daniel Phillips wrote:
> > > > > Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
> > > > > he wanted an evolution to a new build system... not an unreasonable 
> > > > > request to at least consider.  Despite Keith's quality of code (again -- 
> > > > > I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
> > > > > very "take it or leave it dammit".  Not the best way to win friends and 
> > > > > influence people.
> > 
> > OK that's true, but think: how much work has Keith put into this?  How much
> > did he or his employer get paid?  And how many times has he been told to go
> > off and fix something, as a prelude to getting the thing in?  The last time
> > it was the first-time build speed.  He went away and came back with a *huge*
> > improvement, even more than what people were demanding.  You'd think that
> > would be enough.
> 
> But is it really enough?
> 
> Linus himself once said: "Especially as I don't find the existign system so
> broken." He's not alone according to the amount (or lack) of public
> complains with regards to the current system.

I think a lot of us were not complaining because we thought it was a done deal.
Now what's necessary, do we have to form cheerleading teams and start lobbying
to get what we thought was coming anyway?  It's really not a pleasant feeling
to be put in that position.  Nobody relishes the role of outspoken advocate of
this or that, at least I do not.  I'd *much* rather be hacking.

> > > Of course, you could have hundreds of patches each consisting of a single
> > > Makefile.in, but how would that make the reviewing/integrating easier? In
> > > the end you'd end up reading the same input, only you'd complement it by
> > > frequently pressing your favorite show-me-the-next-mail key.
> > 
> > I thought BitKeeper was supposed to be able to deal with precisely this sort
> > of merge problem.  In this case, splitting the thing up just seems
> > unnatural, and a dubious use of time.
> 
> Indeed, I agree.
> 
> But until there is enough people committed to kb25 to the point of pushing
> for its wider adoption, either by releasing traditional kernel patches (like
> the -dj, -aa, -ac or whatever) actually carrying kb25 so to convince Linus
> and others, or producing "piecemeal" patches so it can be merged by Linus
> with less resistance, well nothing will hapen.

You've got it backwards.  The function of the maintainers you mentioned is
to *resist* change, not promote it.  In this case, change comes from the top,
and as long as Linus doesn't realize that there really is a lot of demand
for this change, nothing is going to happen.

> If people aren't interested enough and/or willing to comply with Linus'
> requirements this will be a sad dead end, regardless the amount of effort
> Keith put into this.

Now wait, it seems to me that only Keith is being asked to comply with
requirements, and they're tough requirements.  The burden is just not
being shared equally.

-- 
Daniel

Re: 2.5.19 - What's up with the kernel build?

[Jeff Garzik]

One thing I would like to see is building where builddir != srcdir.

I don't particularly care if this is optional or made a requirement, but 
it's darned useful...

Re: 2.5.19 - What's up with the kernel build?

[Robert Love]

On Wed, 2002-05-29 at 17:42, Jeff Garzik wrote:
> One thing I would like to see is building where builddir != srcdir.
> 
> I don't particularly care if this is optional or made a requirement, but 
> it's darned useful...

I second that - it is incredibly useful.

Next on my list would be fixing dependency issues.

	Robert Love

Re: 2.5.19 - What's up with the kernel build?

[Rusty Russell]

On Wed, 29 May 2002 20:42:53 -0400
Jeff Garzik <jgarzik@mandrakesoft.com> wrote:

> One thing I would like to see is building where builddir != srcdir.

Hmm, I always do:
	cp -al linux-2.5.20 working-2.5.20-hotplug
	cd working-2.5.20-hotplug...

But this only works if you have an editor which breaks hardlinks,
Rusty.
-- 
   there are those who do and those who hang on and you don't see too
   many doers quoting their contemporaries.  -- Larry McVoy

Re: 2.5.19 - What's up with the kernel build?

[Nicolas Pitre]

On Thu, 30 May 2002, Daniel Phillips wrote:

> On Thursday 30 May 2002 02:17, Nicolas Pitre wrote:
> 
> > Linus himself once said: "Especially as I don't find the existign system so
> > broken." He's not alone according to the amount (or lack) of public
> > complains with regards to the current system.
> 
> I think a lot of us were not complaining because we thought it was a done
> deal.

Well it's obviously not.

> Now what's necessary, do we have to form cheerleading teams and start lobbying
> to get what we thought was coming anyway?  

Probably.  At least that often was necessary for most large features in the 
past... especially when those features weren't necessarily _interesting_ to 
Linus.

> It's really not a pleasant feeling to be put in that position.  Nobody
> relishes the role of outspoken advocate of this or that, at least I do
> not.  I'd *much* rather be hacking.

Sure.  But what if only Keith alone publicly praises the virtues of kb25?  
And if Linus on the other end doesn't care much?  And if nobody else is 
showing any interest?

> You've got it backwards.  The function of the maintainers you mentioned is
> to *resist* change, not promote it.  In this case, change comes from the top,
> and as long as Linus doesn't realize that there really is a lot of demand
> for this change, nothing is going to happen.

Well that's exactly what I'm saying is happening. Interest seem pretty low 
unfortunately.  Otherwise more people would stop hacking for a minute and 
advocate a little for the inclusion of kb25.

> > If people aren't interested enough and/or willing to comply with Linus'
> > requirements this will be a sad dead end, regardless the amount of effort
> > Keith put into this.
> 
> Now wait, it seems to me that only Keith is being asked to comply with
> requirements, and they're tough requirements.  The burden is just not
> being shared equally.

Well given the current lack of interest (or let's say enthusiasm to be more 
correct) for kb25, why would Linus be bothered?  Since most people are 
indifferent at this point the burden is likely to remain in Keith's hands.

If that's not the case then let's go people and speak up, advocate for
patches in secondary trees, send your appreciation to Linus and lkml, but
don't stay still.  The preemptive kernel patch just to name that example 
got much more visibility, adoption and promotion than what kb25 had up to 
now.

Otherwise we'll continue fixing the current build system and forget about
kbuild25.


Nicolas

Re: 2.5.19 - What's up with the kernel build?

[Daniel Phillips]

On Thursday 30 May 2002 04:23, you wrote:
> On Thu, 30 May 2002, Daniel Phillips wrote:
> If that's not the case then let's go people and speak up, advocate for
> patches in secondary trees, send your appreciation to Linus and lkml, but
> don't stay still.  The preemptive kernel patch just to name that example 
> got much more visibility, adoption and promotion than what kb25 had up to 
> now.

Sure, and I had a large hand in that.  But I'll tell you this, I'm *tired*
of uphill battles.  It's a major use or waste of my time, even when
successful, and I do not like making a pest of myself.  I'm sure this
particular battle is going to be successful, but I'm not going to fight
it.  I've said my piece and I have other things to do.

-- 
Daniel

Re: 2.5.19 - What's up with the kernel build?

[Kai Germaschewski]

On Wed, 29 May 2002, Jeff Garzik wrote:

> Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
> he wanted an evolution to a new build system... not an unreasonable 
> request to at least consider.  Despite Keith's quality of code (again -- 
> I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
> very "take it or leave it dammit".  Not the best way to win friends and 
> influence people.
> 
> If Keith is indeed leaving it, I'm hoping someone will maintain it, or 
> work with Kai to integrate it into 2.5.x.

Oh well, it really wasn't my intention to start the good old kbuild-2.5
thread at all.

Anyway, I believe kbuild-2.5 has lots of useful ideas and I'll go pick 
pieces - from kbuild-2.5, from dancing-makefiles, from stuff I've done 
myself and work on improving the current build system. But I believe in 
make, and don't think I'll move away from it.

One thing these patches show is that gradual improvement is actually
possible, so far the kbuild process has gained quite some features with a
lot of small patches - and some bigger ones, but these are only trivial
cleanups.

Of course it happened that I introduced some bugs in the process, but the
fact that fixes were posted to linux-kernel by the next morning shows that
it's obviously possible for other people to grasp what's going on and fix
bugs. Rules.make is some 400 lines currently, that's quite a difference to 
kbuild-2.5 core's 30000 lines of code.

Anyway, fortunately it's not up to me to decide what happens. From my 
perspective the plan is to go on with this gradual improvement, in
particular 
o fix dependencies / modversions (that includes "make dep" going away)
o allow for separate objdir (this one is actually easy for 95% of the
  compiled files which use standard rules, and lots of work for the 
  remaining 5%. So it'll take time to remove the 5% special cases, after
  that things are pretty easy)

I believe that Keith will keep maintaining his kbuild-2.5 (if he isn't
willing to put work into it, that's definitely a reason not to merge it,
after all he's the only one who can fix bugs in it for all I can see).  
At the end of the day we will see how the two systems compare and, who
knows, maybe my work will even have simplified getting kbuild-2.5 into the
kernel.

Unfortunately Keith, for all I can see, has an all-or-nothing attitude. 
There's definitely things which could be broken out of kbuild-2.5,
like e.g.

o standardize asm-offsets.h generation
o make the build target a config option (I'm not actually sure if that
  is a good idea at all, using different targets for make seems a 
  sensible concept to me, but anyway)

Splitting things would at least allow for discussing whether these (and
other) individual features are considered worthwhile or not, as opposed to
getting the full bag or nothing.

Also, getting back to the 95%, if you compare kbuild-2.5's Makefile.in's
and the Makefiles in 2.5.19, you will see that 95% of what's in there has
a one-to-one relationship between the two Makefiles. I can't teach "make"  
to read the Makefile.in's which use an entirely new syntax, but since
kbuild-2.5 uses its owner parser, it could be adapted to understand the
old syntax (which expresses the exact same thing, so I don't really
understand why Keith didn't use the old format in the first place). That
would allow to drastically cut down the 20.000 lines Makefile.in diff and
would also simplify maintaining kbuild-2.5.

So that's my point of view.

--Kai
  

Re: 2.5.19 - What's up with the kernel build?

[Daniel Phillips]

On Thursday 30 May 2002 04:00, Kai Germaschewski wrote:
> On Wed, 29 May 2002, Jeff Garzik wrote:
> > Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
> > he wanted an evolution to a new build system... not an unreasonable 
> > request to at least consider.  Despite Keith's quality of code (again -- 
> > I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
> > very "take it or leave it dammit".  Not the best way to win friends and 
> > influence people.
> > 
> > If Keith is indeed leaving it, I'm hoping someone will maintain it, or 
> > work with Kai to integrate it into 2.5.x.
> 
> Oh well, it really wasn't my intention to start the good old kbuild-2.5
> thread at all.
> 
> Anyway, I believe kbuild-2.5 has lots of useful ideas and I'll go pick 
> pieces - from kbuild-2.5, from dancing-makefiles, from stuff I've done 
> myself and work on improving the current build system. But I believe in 
> make, and don't think I'll move away from it.

I wish you would just join the kbuild team and work with them instead of
against them.  And what makes you think that it doesn't use make?

-- 
Daniel

Re: 2.5.19 - What's up with the kernel build?

[Jeff Garzik]

Kai Germaschewski wrote:

>On Wed, 29 May 2002, Jeff Garzik wrote:
>
>  
>
>>Well, I really like Keith's kbuild25 too, but Linus said (at least once) 
>>he wanted an evolution to a new build system... not an unreasonable 
>>request to at least consider.  Despite Keith's quality of code (again -- 
>>I like kbuild25), his 3 patch submissions seemed a lot like ultimatums, 
>>very "take it or leave it dammit".  Not the best way to win friends and 
>>influence people.
>>
>>If Keith is indeed leaving it, I'm hoping someone will maintain it, or 
>>work with Kai to integrate it into 2.5.x.
>>    
>>
>
>Oh well, it really wasn't my intention to start the good old kbuild-2.5
>thread at all.
>
>Anyway, I believe kbuild-2.5 has lots of useful ideas and I'll go pick 
>pieces - from kbuild-2.5, from dancing-makefiles, from stuff I've done 
>myself and work on improving the current build system. But I believe in 
>make, and don't think I'll move away from it.
>
>One thing these patches show is that gradual improvement is actually
>possible, so far the kbuild process has gained quite some features with a
>lot of small patches - and some bigger ones, but these are only trivial
>cleanups.
>
>Of course it happened that I introduced some bugs in the process, but the
>fact that fixes were posted to linux-kernel by the next morning shows that
>it's obviously possible for other people to grasp what's going on and fix
>bugs. Rules.make is some 400 lines currently, that's quite a difference to 
>kbuild-2.5 core's 30000 lines of code.
>
>Anyway, fortunately it's not up to me to decide what happens. From my 
>perspective the plan is to go on with this gradual improvement, in
>particular 
>o fix dependencies / modversions (that includes "make dep" going away)
>o allow for separate objdir (this one is actually easy for 95% of the
>  compiled files which use standard rules, and lots of work for the 
>  remaining 5%. So it'll take time to remove the 5% special cases, after
>  that things are pretty easy)
>  
>

A small request to add to the list:

Current 2.4.x kernels build (at least on x86) with
     -nostdinc -I /usr/lib/gcc-lib/i586-mandrake-linux-gnu/3.0.4/include
added to CFLAGS...  IMOit is a good idea in general to build all kernel 
code this way.  (note that userland programs created during build should 
not use this rule, of course)

Re: 2.5.19 - What's up with the kernel build?

[Erik Andersen]

On Thu May 30, 2002 at 04:45:12AM -0400, Jeff Garzik wrote:
> A small request to add to the list:
> 
> Current 2.4.x kernels build (at least on x86) with
>     -nostdinc -I /usr/lib/gcc-lib/i586-mandrake-linux-gnu/3.0.4/include

Shockingly, not everyone uses mandrake's gcc 3.0.4...  ;-)

GCCINCDIR:= ${shell $(CC) -print-search-dirs | sed -ne "s/install: \(.*\)/\1include/gp"}
CFLAGS+=-nostdinc -I $(GCCINCDIR)

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: 2.5.19 - What's up with the kernel build?

[Russell King]

On Thu, May 30, 2002 at 02:54:13AM -0600, Erik Andersen wrote:
> On Thu May 30, 2002 at 04:45:12AM -0400, Jeff Garzik wrote:
> > A small request to add to the list:
> > 
> > Current 2.4.x kernels build (at least on x86) with
> >     -nostdinc -I /usr/lib/gcc-lib/i586-mandrake-linux-gnu/3.0.4/include
> 
> Shockingly, not everyone uses mandrake's gcc 3.0.4...  ;-)
> 
> GCCINCDIR:= ${shell $(CC) -print-search-dirs | sed -ne "s/install: \(.*\)/\1include/gp"}
> CFLAGS+=-nostdinc -I $(GCCINCDIR)

There is a nicer way of achieving the same thing:

CFLAGS	+= -nostdinc -iwithprefix include

`-iwithprefix DIR'
     Add a directory to the second include path.  The directory's name
     is made by concatenating PREFIX and DIR, where PREFIX was
     specified previously with `-iprefix'.  If you have not specified a
     prefix yet, the directory containing the installed passes of the
     compiler is used as the default.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: 2.5.19 - What's up with the kernel build?

[Erik Andersen]

On Thu May 30, 2002 at 10:01:44AM +0100, Russell King wrote:
> > Shockingly, not everyone uses mandrake's gcc 3.0.4...  ;-)
> > 
> > GCCINCDIR:= ${shell $(CC) -print-search-dirs | sed -ne "s/install: \(.*\)/\1include/gp"}
> > CFLAGS+=-nostdinc -I $(GCCINCDIR)
> 
> There is a nicer way of achieving the same thing:
> 
> CFLAGS	+= -nostdinc -iwithprefix include
> 
> `-iwithprefix DIR'
>      Add a directory to the second include path.  The directory's name

Cool, I didn't know about this one.  thanks,

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: 2.5.19 - What's up with the kernel build?

[Kai Germaschewski]

On Thu, 30 May 2002, Jeff Garzik wrote:

> A small request to add to the list:
> 
> Current 2.4.x kernels build (at least on x86) with
>      -nostdinc -I /usr/lib/gcc-lib/i586-mandrake-linux-gnu/3.0.4/include
> added to CFLAGS...  IMOit is a good idea in general to build all kernel 
> code this way.  (note that userland programs created during build should 
> not use this rule, of course)

Yep, sure make sense. - It was on my list of things to look at anyway, I 
didn't realize it's in 2.4 already, though - so that saves some testing ;)

--Kai

Re: 2.5.19 - What's up with the kernel build?

[Martin Dalecki]

Kai Germaschewski wrote:
> 
> Okay, as people probably already noticed, I touched quite a lot of
  bla... bla ... bla...
> 

And all the things explained above just belong in to Documentation
or the top level README :-).

And plese note that no matter how much I wellcome the
clenups you did - you broke the usage of make -s!
And this is quite painfull for spotting warinings during
edit/compile cycle... Would you please dare to take a bit
care of this. I would be really glad if you could.

Re: 2.5.19 - What's up with the kernel build?

[Tom Rini]

On Wed, May 29, 2002 at 03:22:52PM -0500, Kai Germaschewski wrote:

> >>> It's possible with only small changes to provide a quiet mode now,
> >>> which would not print the entire command lines but only
> >>>
> >>>	  Descending into drivers/isdn/kcapi
> >>>	  Compiling kcapi.o
> >>>	  Compiling capiutil.o
> >>>	  Linking kernelcapi.o
> >>>	  ...
> >>>
> >>> Is that considered useful?

I don't think so.  If you're on a slow connection or something, redirect
stdout to a log and watch stderr.  If you just want something prettier,
and this is easy, I don't think this is a bad thing.  I don't think it
should be the default tho either. :)

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: 2.5.19 - What's up with the kernel build?

[Martin Dalecki]

Tom Rini wrote:
> On Wed, May 29, 2002 at 03:22:52PM -0500, Kai Germaschewski wrote:
> 
> 
>>>>>It's possible with only small changes to provide a quiet mode now,
>>>>>which would not print the entire command lines but only
>>>>>
>>>>>	  Descending into drivers/isdn/kcapi
>>>>>	  Compiling kcapi.o
>>>>>	  Compiling capiutil.o
>>>>>	  Linking kernelcapi.o
>>>>>	  ...
>>>>>
>>>>>Is that considered useful?
>>>>
> 
> I don't think so.  If you're on a slow connection or something, redirect
> stdout to a log and watch stderr.  If you just want something prettier,
> and this is easy, I don't think this is a bad thing.  I don't think it
> should be the default tho either. :)

Or in clear words - it's redundant bullshit if you don't care and
it's preventing you from seeing the "trueth" if you care.
Please just make make -s work agin and I would be happy. OK?

Re: 2.5.19 - What's up with the kernel build?

[Kai Germaschewski]

On Thu, 30 May 2002, Martin Dalecki wrote:

> Or in clear words - it's redundant bullshit if you don't care and
> it's preventing you from seeing the "trueth" if you care.
> Please just make make -s work agin and I would be happy. OK?

Have you tried make >/dev/null? But yes, I'll make it work again, just for 
you ;)

--Kai

Kernel compile quiet mode

[Kevin O'Connor]

On Wed, May 29, 2002 at 03:22:52PM -0500, Kai Germaschewski wrote:
[...]
> >>> It's possible with only small changes to provide a quiet mode now,
> >>> which would not print the entire command lines but only
> >>>
> >>>	  Descending into drivers/isdn/kcapi
> >>>	  Compiling kcapi.o
> >>>	  Compiling capiutil.o
> >>>	  Linking kernelcapi.o
> >>>	  ...
> >>>
> >>> Is that considered useful?

Hi Kai,

I think it would be useful, but in a slightly different way.  The
information printed during make can be informative, but too much of the
output is just redundant.  I'd prefer a change something like the
following:

--- ../gold-2.5/Rules.make      Fri May 31 22:10:04 2002
+++ ./Rules.make        Fri May 31 22:56:33 2002
@@ -399,5 +399,5 @@
 if_changed = $(if $(strip $? \
                          $(filter-out $($(1)),$(cmd_$(@F)))\
                          $(filter-out $(cmd_$(@F)),$($(1)))),\
-              @echo '$($(1))' && $($(1)) && echo 'cmd_$@ := $($(1))' > $(@D)/.$(@F).cmd)
+              @echo '$(subst $(CFLAGS),$$CFLAGS,$(strip $($(1))))' && $($(1)) && echo 'cmd_$@ := $($(1))' > $(@D)/.$(@F).cmd)
 

which causes output to look like:

gcc $CFLAGS -DKBUILD_BASENAME=raw -c -o raw.o raw.c
gcc $CFLAGS -DKBUILD_BASENAME=pty -DEXPORT_SYMTAB -c -o pty.o pty.c
gcc $CFLAGS -DKBUILD_BASENAME=misc -DEXPORT_SYMTAB -c -o misc.o misc.c
gcc $CFLAGS -DKBUILD_BASENAME=random -DEXPORT_SYMTAB -c -o random.o random.c
gcc $CFLAGS -DKBUILD_BASENAME=vt -c -o vt.o vt.c
gcc $CFLAGS -DKBUILD_BASENAME=vc_screen -c -o vc_screen.o vc_screen.c

IMO, this has all the advantages of the "quiet" output, but none of the
information loss.  (The only thing suppressed is CFLAGS which doesn't
change.)

-Kevin

-- 
 ------------------------------------------------------------------------
 | Kevin O'Connor                     "BTW, IMHO we need a FAQ for      |
 | kevin@koconnor.net                  'IMHO', 'FAQ', 'BTW', etc. !"    |
 ------------------------------------------------------------------------

Re: Kernel compile quiet mode

[Kai Germaschewski]

[Answered a follow-up mail and didn't realize that l-k was CC'ed in
 the first one, so here's my reply again.]

>From kai-germaschewski@uiowa.edu Sat Jun  1 02:30:31 2002
Date: Sat, 1 Jun 2002 02:27:48 -0500 (CDT)
From: Kai Germaschewski <kai-germaschewski@uiowa.edu>
To: Kevin O'Connor <kevin@koconnor.net>
Subject: Re: Kernel compile quiet mode

On Fri, 31 May 2002, Kevin O'Connor wrote:

> > gcc $CFLAGS -DKBUILD_BASENAME=raw -c -o raw.o raw.c
> > gcc $CFLAGS -DKBUILD_BASENAME=pty -DEXPORT_SYMTAB -c -o pty.o pty.c
> > gcc $CFLAGS -DKBUILD_BASENAME=misc -DEXPORT_SYMTAB -c -o misc.o misc.c
> > gcc $CFLAGS -DKBUILD_BASENAME=random -DEXPORT_SYMTAB -c -o random.o random.c
> > gcc $CFLAGS -DKBUILD_BASENAME=vt -c -o vt.o vt.c
> > gcc $CFLAGS -DKBUILD_BASENAME=vc_screen -c -o vc_screen.o vc_screen.c

Not bad, I suppose different people prefer different approaches, anyway. 
There is, however, still redundant output there, the "-o vt.o" and 
"KBUILD_BASENAME=vt" should go away as well too, IMO.

Anyway, I'll be out of town for a couple of days, if you want to see what
I'm up to, take a look at linux-isdn.bkbits.net, linux-2.5.make and
linux-2.5.make-new - What I'm doing for the quiet output is pretty
flexible, it can be easily adapted to things like the above.

I'd surely appreciate if people wanted to test things, if you don't have
bitkeeper but want to give it a shot, I can mail a patch.

(For the visually most prominent change, add "quiet" to your make command
line, or set KBUILD_QUIET=1 in the environment)

--Kai

Re: Kernel compile quiet mode

[Kevin O'Connor]

On Sat, Jun 01, 2002 at 02:34:25AM -0500, Kai Germaschewski wrote:
> > > gcc $CFLAGS -DKBUILD_BASENAME=raw -c -o raw.o raw.c
> > > gcc $CFLAGS -DKBUILD_BASENAME=pty -DEXPORT_SYMTAB -c -o pty.o pty.c
> > > gcc $CFLAGS -DKBUILD_BASENAME=misc -DEXPORT_SYMTAB -c -o misc.o misc.c
> > > gcc $CFLAGS -DKBUILD_BASENAME=random -DEXPORT_SYMTAB -c -o random.o random.c
> > > gcc $CFLAGS -DKBUILD_BASENAME=vt -c -o vt.o vt.c
> > > gcc $CFLAGS -DKBUILD_BASENAME=vc_screen -c -o vc_screen.o vc_screen.c
> 
> Not bad, I suppose different people prefer different approaches, anyway. 
> There is, however, still redundant output there, the "-o vt.o" and 
> "KBUILD_BASENAME=vt" should go away as well too, IMO.

I agree about the "-o vt.o" part.  I'm not so sure about KBUILD_BASENAME -
it's a little redundant, but it is sent to the compiler and is different on
each compile.  Anyway, fixing the '-o' redundancy is simple - just don't
send it to the compiler in the first place (the compiler will always do it
anyway).

I've attached another example patch to the end of this message.  Output
looks like:

gcc $CFLAGS -fno-omit-frame-pointer -DKBUILD_BASENAME=sched -c sched.c
gcc $CFLAGS -DKBUILD_BASENAME=dma -c dma.c
gcc $CFLAGS -DKBUILD_BASENAME=fork -c fork.c
gcc $CFLAGS -DKBUILD_BASENAME=exec_domain -DEXPORT_SYMTAB -c exec_domain.c
gcc $CFLAGS -DKBUILD_BASENAME=panic -c panic.c

> 
> [Answered a follow-up mail and didn't realize that l-k was CC'ed in
>  the first one, so here's my reply again.]

Sorry about that.  This one is also CC'ed to lkml.

Thanks,
-Kevin



--- ../gold-2.5/Rules.make      Fri May 31 22:10:04 2002
+++ ./Rules.make        Sat Jun  1 10:57:18 2002
@@ -127,7 +127,7 @@
 %.i: %.c FORCE
        $(call if_changed,cmd_cc_i_c)
 
-cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $<
+cmd_cc_o_c = $(CC) $(c_flags) -c $<
 
 %.o: %.c FORCE
        $(call if_changed,cmd_cc_o_c)
@@ -396,8 +396,9 @@
 
 # function to only execute the passed command if necessary
 
+echo_cmd = echo '$(strip $(subst $(CFLAGS),$$CFLAGS,$(subst $(CFLAGS_MODULE),$$CFLAGS_MODULE,$($(1)))))'
+
 if_changed = $(if $(strip $? \
                          $(filter-out $($(1)),$(cmd_$(@F)))\
                          $(filter-out $(cmd_$(@F)),$($(1)))),\
-              @echo '$($(1))' && $($(1)) && echo 'cmd_$@ := $($(1))' > $(@D)/.$(@F).cmd)
-
+              @$(call echo_cmd,$(1)) && $($(1)) && echo 'cmd_$@ := $($(1))' > $(@D)/.$(@F).cmd)



-- 
 ------------------------------------------------------------------------
 | Kevin O'Connor                     "BTW, IMHO we need a FAQ for      |
 | kevin@koconnor.net                  'IMHO', 'FAQ', 'BTW', etc. !"    |
 ------------------------------------------------------------------------