State of the new config & build system

State of the new config & build system

[Eric S. Raymond]

Linus (and Marcelo): I understand that right at the moment you have
higher priorities than merging in the new build system.  Keith Owens
and I agree with those priorities, so please consider the following to
be information rather than pressure for action.

Keith's kbuild-2.5 and my CML2 both appear to be shaking down quite
nicely.

In the last eight weeks the level of beta testing we're getting from
lkml regulars has risen dramatically, as has the amount of work being
put in on the codebases by people other than Keith and myself (just
last night I checked in an entire new X-based front end contributed by
a hacker from Korea).  

Despite the increased attention, the criticality level of incoming bug
reports has held steady or decreased, to the point that we're
basically both just doing normal maintainance and polishing the chrome
now.  I haven't seen a really serious bug in CML2 since I resumed
active work on it in early November, and Keith's stuff is stable
enough that he's now adding features like kernel-image type selection
that were obviously way down his to-do list.

Just as importantly, the kernel development community now seems to be
actively preparing for the build-system cutover, as opposed to just
passively waiting for it.  Some are doing their cutover in *advance*
of the main tree; the kinds of kbuild bug reports I see on the list
indicate that Keith's kbuild is already in production use, and in the
last week I've gotten requests from SGI's XFS group and the ELKS
project for help with switching to CML2.

In sum, we're ready now -- but that's been true since at latest early
November.  What's new in the last couple weeks is that the developer
community appears to be coming up to speed on our technology
effectively enough to be ready as well.

We can help plan and execute the cutover any time you're ready.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

When all government ...in little as in great things... shall be drawn to
Washington as the center of all power; it will render powerless the checks
provided of one government on another, and will become as venal and oppressive
as the government from which we separated."	-- Thomas Jefferson, 1821

Re: State of the new config & build system

[Dave Jones]

On Thu, 27 Dec 2001, Eric S. Raymond wrote:

> ..., and Keith's stuff is stable
> enough that he's now adding features like kernel-image type selection
> that were obviously way down his to-do list.

How far down the list was "make it not take twice as long
to build the kernel as kbuild 2.4" ? Keith mentioned O(n^2)
effects due to each compile operation needing to reload
the dependancies etc.

Given how early your both pushing to get these into the tree(s),
and given how many kernels are going to be built between now
and 2.6.0, slowing down development for _every_ kernel developer
doesn't strike me as a bright move. Maybe keep them both in the
tree until this issue is worked out ? That way those who want to
play with kbuild can do so, and those who build a few dozen
kernels a day don't have to twiddle thumbs.

Don't get me wrong, I'm not knocking CML2 or kbuild2.5,
I'm just interested in some of timescale for getting wrinkles
like this out.

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: State of the new config & build system

[Eric S. Raymond]

Dave Jones <davej@suse.de>:
>                                     Maybe keep them both in the
> tree until this issue is worked out ? That way those who want to
> play with kbuild can do so, and those who build a few dozen
> kernels a day don't have to twiddle thumbs.

That is such an unutterably horrible concept that the very tentacles
of Cthulhu himself must twitch in dread at the thought.  The last thing
anyone sane wants to do is have to maintain two parallel build systems
at the same time.

-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

You know why there's a Second Amendment?  In case the government fails to
follow the first one.
         -- Rush Limbaugh, in a moment of unaccustomed profundity 17 Aug 1993

Re: State of the new config & build system

[Larry McVoy]

On Thu, Dec 27, 2001 at 07:57:38PM -0500, Eric S. Raymond wrote:
> Dave Jones <davej@suse.de>:
> >                                     Maybe keep them both in the
> > tree until this issue is worked out ? That way those who want to
> > play with kbuild can do so, and those who build a few dozen
> > kernels a day don't have to twiddle thumbs.
> 
> That is such an unutterably horrible concept that the very tentacles
> of Cthulhu himself must twitch in dread at the thought.  The last thing
> anyone sane wants to do is have to maintain two parallel build systems
> at the same time.

Then it does seem reasonable to ask that the new one is at least as fast
as the old one.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Dave Jones]

On Thu, 27 Dec 2001, Eric S. Raymond wrote:

> That is such an unutterably horrible concept that the very tentacles
> of Cthulhu himself must twitch in dread at the thought.  The last thing
> anyone sane wants to do is have to maintain two parallel build systems
> at the same time.

Funny, I could have sworn I read this was Keith's intention at least
for a few pre's. Maybe I misinterpreted his intentions.

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: State of the new config & build system

[Keith Owens]

On Thu, 27 Dec 2001 17:15:45 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>[talking about kbuild 2.5 speed]
>Then it does seem reasonable to ask that the new one is at least as fast
>as the old one.

kbuild 2.4 is fast but inaccurate, kbuild 2.5 is slower but accurate.
Pick one.

I am sure that I can speed up kbuild 2.5 with a rewrite of the core
code but I am staying on stable code to send to Linus.

Re: State of the new config & build system

[Larry McVoy]

On Fri, Dec 28, 2001 at 12:35:50PM +1100, Keith Owens wrote:
> On Thu, 27 Dec 2001 17:15:45 -0800, 
> Larry McVoy <lm@bitmover.com> wrote:
> >[talking about kbuild 2.5 speed]
> >Then it does seem reasonable to ask that the new one is at least as fast
> >as the old one.
> 
> kbuild 2.4 is fast but inaccurate, kbuild 2.5 is slower but accurate.
> Pick one.
> 
> I am sure that I can speed up kbuild 2.5 with a rewrite of the core
> code but I am staying on stable code to send to Linus.

A couple of questions:

a) will 2.5 be as fast as the current system?  Faster?
b) what's the eta on 2.5?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 02:22:01 +0100 (CET), 
Dave Jones <davej@suse.de> wrote:
>On Thu, 27 Dec 2001, Eric S. Raymond wrote:
>
>> That is such an unutterably horrible concept that the very tentacles
>> of Cthulhu himself must twitch in dread at the thought.  The last thing
>> anyone sane wants to do is have to maintain two parallel build systems
>> at the same time.
>
>Funny, I could have sworn I read this was Keith's intention at least
>for a few pre's. Maybe I misinterpreted his intentions.

Only long enough to prove that kbuild 2.5 built kernels that worked.
And to give unconverted architectures a kernel that had both old and
new kbuild in it for their conversion.  Once kbuild 2.5 has proved it
works, kbuild 2.4 is coming out.

Re: State of the new config & build system

[Keith Owens]

On Thu, 27 Dec 2001 17:37:39 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>A couple of questions:
>
>a) will 2.5 be as fast as the current system?  Faster?

At the moment kbuild 2.5 ranges from 10% faster on small builds to 100%
slower on a full kernel build.  But that is using slow core code which
I know I can rewrite to make it significantly faster.

>b) what's the eta on 2.5?

kbuild 2.5 is ready now.  I am not even going to start on the core
rewrite until Linus takes the existing kbuild 2.5 code.  The existing
code works and is stable, doing a complete core rewrite just before
includeing in the kernel strikes me as stupid.

Re: State of the new config & build system

[Larry McVoy]

On Fri, Dec 28, 2001 at 12:41:48PM +1100, Keith Owens wrote:
> On Thu, 27 Dec 2001 17:37:39 -0800, 
> Larry McVoy <lm@bitmover.com> wrote:
> >A couple of questions:
> >
> >a) will 2.5 be as fast as the current system?  Faster?
> 
> At the moment kbuild 2.5 ranges from 10% faster on small builds to 100%
> slower on a full kernel build.  

I don't understand why it would be slower.  Maybe I'm clueless but I thought
you were moving more towards a single makefile system, kind of like what
BSD had about 15 years ago, you went to /sys/MYMACHINE and typed make and
it did the build completely in that directory.  You did different configs
by running a configure tool that made /sys/MYMACHINE /sys/YOURMACHINE, etc.

If this is the general approach, shouldn't this be a lot faster than the
current approach?  The current approach stats stuff many times.  Linux is
really good at making stats cheap, but nothing is as good as not doing it
twice.

Am I completely misunderstanding what kbuild is all about?  My apologies
if so...
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Keith Owens]

On Thu, 27 Dec 2001 17:47:23 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Fri, Dec 28, 2001 at 12:41:48PM +1100, Keith Owens wrote:
>> On Thu, 27 Dec 2001 17:37:39 -0800, 
>> Larry McVoy <lm@bitmover.com> wrote:
>> >A couple of questions:
>> >
>> >a) will 2.5 be as fast as the current system?  Faster?
>> 
>> At the moment kbuild 2.5 ranges from 10% faster on small builds to 100%
>> slower on a full kernel build.  
>
>I don't understand why it would be slower.  Maybe I'm clueless but I thought
>you were moving more towards a single makefile system

It uses a single generated Makefile, that is not the problem.  The slow
code is extracting the dependencies.

Unlike the broken make dep, kbuild 2.5 extracts accurate dependencies
by using the -MD option of cpp and post processing the cpp list.  The
post processing code is slow because the current design requires every
compile to read a complete list of all the files, giving O(n^2)
effects.  Mark 2 of the core code will use a shared database with
concurrent update so post processing is limited to looking up just the
required files, instead of reading the complete list every time.

Re: State of the new config & build system

[Larry McVoy]

> Unlike the broken make dep, kbuild 2.5 extracts accurate dependencies
> by using the -MD option of cpp and post processing the cpp list.  The
> post processing code is slow because the current design requires every
> compile to read a complete list of all the files, giving O(n^2)
> effects.  Mark 2 of the core code will use a shared database with
> concurrent update so post processing is limited to looking up just the
> required files, instead of reading the complete list every time.

Ah, OK, I get it.  Hey, would it help to have a dbm interface compat 
library which uses mmap instead of building the db in brk() space?
We've got a small, fast one that you can have under any license you
like, GPL, LGPL, whatever.  We use it all over the BK code.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Legacy Fishtank]

On Fri, Dec 28, 2001 at 01:54:42AM +0100, Dave Jones wrote:
> How far down the list was "make it not take twice as long
> to build the kernel as kbuild 2.4" ? Keith mentioned O(n^2)
> effects due to each compile operation needing to reload
> the dependancies etc.

Each compile needs to reload deps???

Ug.  IMHO if you are doing to shake up the entire build system, you
should Do It Right(tm) and build a -complete- dependency graph -once-.

	Jeff

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 04:26:48 -0500, 
Legacy Fishtank <garzik@havoc.gtf.org> wrote:
>On Fri, Dec 28, 2001 at 01:54:42AM +0100, Dave Jones wrote:
>> How far down the list was "make it not take twice as long
>> to build the kernel as kbuild 2.4" ? Keith mentioned O(n^2)
>> effects due to each compile operation needing to reload
>> the dependancies etc.
>
>Each compile needs to reload deps???
>
>Ug.  IMHO if you are doing to shake up the entire build system, you
>should Do It Right(tm) and build a -complete- dependency graph -once-.

We have one complete dependency graph for the explicit dependencies.
What is slow is extracting the implicit dependencies after an object
has been compiled, i.e. the files that it includes.  Actually
extracting the implicit dependencies is fast, converting them to
standard names is fast, what is slow is _reading_ the big list that
maps from absolute names to standardized names.

I need the big list in order to remove absolute names in the dependency
trees.  kbuild 2.4 forces a complete recompile if you rename a tree,
including if you build on one system then try to install via NFS on a
second system.  kbuild 2.5 can cope with trees being renamed and trees
having different names on local and NFS mounted systems.  That
flexibility comes at a cost.

"All" I need to do is have one server process that reads the big list
once and the other client processes talk to the server.  Much less data
involved means faster conversion from absolute to standardized names.

Re: State of the new config & build system

[Alan Cox]

> Ah, OK, I get it.  Hey, would it help to have a dbm interface compat 
> library which uses mmap instead of building the db in brk() space?

mmap for db file seems to be slower. For basic db hash usage and raw speed
nothing seems to touch tdb (Tridge's db hack). Its also portable code which
is important since the tool has to be built on the compiling host.

Personally I've always considered make dep good enough. Its trying to solve
the extra .5% that probably can be solved by careful use of make clean when
CML realises a critical rule changed (SMP etc)

Alan

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 14:14:37 +0000 (GMT), 
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> Ah, OK, I get it.  Hey, would it help to have a dbm interface compat 
>> library which uses mmap instead of building the db in brk() space?
>
>mmap for db file seems to be slower. For basic db hash usage and raw speed
>nothing seems to touch tdb (Tridge's db hack). Its also portable code which
>is important since the tool has to be built on the compiling host.

lm sent me the bk mdbm code but I will look at tdb as well.  Four
acronyms in one sentance, I must be a phb :).

>Personally I've always considered make dep good enough. Its trying to solve
>the extra .5% that probably can be solved by careful use of make clean when
>CML realises a critical rule changed (SMP etc)

http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2
Especially makefile-2.5_make_dep.html, 9 reasons why make dep is broken
as designed.  Some are fixable in the current system, others are
inherently unfixable.  I skipped that page when I did my presentation
at the 2.5 developer's conference.

Re: State of the new config & build system

[Alan Cox]

> At the moment kbuild 2.5 ranges from 10% faster on small builds to 100%
> slower on a full kernel build.  But that is using slow core code which

> kbuild 2.5 is ready now.  I am not even going to start on the core

"Its 100% slower so its ready"

I must be missing something here. If its 100% slower its not ready.

Alan

Re: State of the new config & build system

[Alan Cox]

> including if you build on one system then try to install via NFS on a
> second system.  kbuild 2.5 can cope with trees being renamed and trees
> having different names on local and NFS mounted systems.  That
> flexibility comes at a cost.

So you've halved performance rather than documented that you have to mount
the tree in the space place on every NFS export ? I'm obviously still
missing something here.

Alan

Re: State of the new config & build system

[Christer Weinigel]

In article <4481.1009549017@ocs3.intra.ocs.com.au> you write:
>Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>>Personally I've always considered make dep good enough. Its trying to solve
>>the extra .5% that probably can be solved by careful use of make clean when
>>CML realises a critical rule changed (SMP etc)
>
>http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2
>Especially makefile-2.5_make_dep.html, 9 reasons why make dep is broken
>as designed.  Some are fixable in the current system, others are
>inherently unfixable.  I skipped that page when I did my presentation
>at the 2.5 developer's conference.

* make dep is only run once

Personally, I don't see this as a big problem.  Just tell people to 
always run "make oldconfig && make dep" after patching the kernel 
and if they can't read, too bad.  I usually compile a kernel a lot 
more often than I add include files.  Since I'm quite impatient
I often do "make SUBDIRS=drivers/mtd/maps modules" just so that the
compile will go faster, so having to do dependency checking each time
I want to compile feels like an unfortunate tradeoff to me.

* The generated dependencies are absolute

That dependencies are absolute is also not a thing that has bothered me 
too much, it's always possible to run "make dep" after moving a tree, 
on the other hand, I don't use NFS a lot anymore, so I can see it being 
a problem in other environments.

    /Christer
-- 
"Just how much can I get away with and still go to heaven?"

Re: State of the new config & build system

[Alan Cox]

> * make dep is only run once
> 
> Personally, I don't see this as a big problem.  Just tell people to 

I run make dep whenever I change config. Guess what - one cmp and I can
automate that as part of make. If the .config doesnt match the
.configwhendep then its time to make dep again

> That dependencies are absolute is also not a thing that has bothered me 
> too much, it's always possible to run "make dep" after moving a tree, 
> on the other hand, I don't use NFS a lot anymore, so I can see it being 
> a problem in other environments.

sed works too, as do symlinks

Re: State of the new config & build system

[Larry McVoy]

On Fri, Dec 28, 2001 at 02:14:37PM +0000, Alan Cox wrote:
> > Ah, OK, I get it.  Hey, would it help to have a dbm interface compat 
> > library which uses mmap instead of building the db in brk() space?
> 
> mmap for db file seems to be slower. 

I'll need to see some numbers to back up that statement, please.  If you
look at the graphs produced by LMbench, they tell you exactly what
you need to know.  It's true that for very small files, 8K and under,
using read() to access them is faster than using mmap, due to the extra
work of setting up and tearing down the mapping.  To quantify this, a
4KB open/read/close is 500MB/sec, but an open/mmap/access/unmap/close
is 425MB/sec.  By the time we hit 16K, mmap wins by 15% and just gets
better from there.

And that all assumes you are doing large reads, which in db code you 
are not.  So mmap will look better even on the small files if you 
are doing little DB style accesses.

> For basic db hash usage and raw speed
> nothing seems to touch tdb (Tridge's db hack). 

Taking nothing away from Tridge, I like Tridge, I'd like to see numbers.
I'm sure that Tridge's stuff is great, but we were very motivated to 
go well beyond the normal effort when we wrote this code.

A multithreaded version of the code that I sent to Keith was doing 455,000
lookups/second on an 8way 200Mhz R4400 SGI box in 1996.  Each lookup
was locked.  If you assume perfect scaling (it was) and you assume the
locks took 0 time (they didn't), that's 1.75 usecs for each lookup.
On a machine with horrible memory latency and a large dataset.

We designed the MDBM code to be scalable (its 64bit clean), portable
(runs on 20+ platforms today), multiplatform (metadata is stored in
network byte order on disk), and fast (we knew exactly what the 
instruction and data cache footprint was for hot cache, and we made
sure that you did at most 2 disk accesses, 1 was typical, to get at
any item in a cold cache).

SGI uses this code for their name server, every process mmaps the 
name server cache.  

We use this code all over BitKeeper.

> Its also portable code which
> is important since the tool has to be built on the compiling host.

The code works on Windows, MacOS X, and basically all Unix platforms.


Yeah, yeah, I pounding my chest and I'm sorry, but I get beat up all the
time that the BK license doesn't let you reuse code and this code is part
of BK that we broke out and licensed under the GPL.  The point being
that if there is reusable code in BK, we're willing to let people use
it under whatever license they want.  It would be nice if that actually
happened after all the yelling and screaming.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Linus Torvalds]

[ Btw, Jeff, any reason why you changed your name to "Legacy Fishtank"? It
  took a few mails before I noticed that it also said "garzik" in the
  fine print;]

One thing that this big flame-war has brought up is that different people
like different things. There may be a simpler solution to this: have the
core dependency files generated from some other file format.

My pet peeve is "centralized knowledge". I absolutely detested the first
versions of cml2 for having a single config file, and quite frankly I
don't think Eric has even _yet_ separated things out enough - why does the
main "rules.cml" file have architecture-specific info, for example?

That's a big step backwards as far as I'm concerned - we didn't use to
have those stupid global files, and each architecture could do it's own
config rules. Eric never got the point that to me, modularity is _the_
most important thing for maintenance.

Something I also asked for the config system at least a year ago was to
have Configure.help split up. Never happened. It's still one large ugly
file. Driver or architecture maintainers still can't just change _their_
small fragment, they have to touch a global file that they don't "own".

So if somebody really wants to help this, make scripts that generate
config files AND Configure.help files from a distributed set. And once you
do that, you could even imagine creating the old-style config files
(without the automatic checking and losing some information) from the
information.

		Linus

Re: State of the new config & build system

[Alan Cox]

> that if there is reusable code in BK, we're willing to let people use
> it under whatever license they want.  It would be nice if that actually
> happened after all the yelling and screaming.

mdbm is one I've not seen. The timings I've done are with db2/db3/tdb when
I was playing with a fast UDP server that had to do a db lookup per packet.

Alan

Re: State of the new config & build system

[Alan Cox]

> So if somebody really wants to help this, make scripts that generate
> config files AND Configure.help files from a distributed set. And once you
> do that, you could even imagine creating the old-style config files

Something like:

	find $TOPDIR -name "*.cf" -exec cat {} \; > Configure.help 

or changing the tools to look for 

	Documentation/Configure/CONFIG_SMALL_BANANA

??

Alan

Re: State of the new config & build system

[Riley Williams]

Hi Linus.

> [ Btw, Jeff, any reason why you changed your name to "Legacy
> Fishtank"? It took a few mails before I noticed that it also said
> "garzik" in the fine print;]

I had wondered where he'd gone to - Jeff was one of the few from who I
read every email, and it's been a while since I saw anything from him.

> One thing that this big flame-war has brought up is that different
> people like different things. There may be a simpler solution to
> this: have the core dependency files generated from some other file
> format.
> 
> My pet peeve is "centralized knowledge". I absolutely detested the
> first versions of cml2 for having a single config file, and quite
> frankly I don't think Eric has even _yet_ separated things out
> enough - why does the main "rules.cml" file have
> architecture-specific info, for example?
> 
> That's a big step backwards as far as I'm concerned - we didn't use
> to have those stupid global files, and each architecture could do
> it's own config rules. Eric never got the point that to me,
> modularity is _the_ most important thing for maintenance.
> 
> Something I also asked for the config system at least a year ago was
> to have Configure.help split up. Never happened. It's still one
> large ugly file. Driver or architecture maintainers still can't just
> change _their_ small fragment, they have to touch a global file that
> they don't "own".
> 
> So if somebody really wants to help this, make scripts that generate
> config files AND Configure.help files from a distributed set. And
> once you do that, you could even imagine creating the old-style
> config files (without the automatic checking and losing some
> information) from the information.

I offered to go through Configure.help and split it up a while back, and
I was drowned in several dozen emails saying that such was a BAD THING,
with absolutely zilch saying otherwise. I'm more than willing to have a
go at splitting it up into separate files by directory, but before I do,
I would need to know how you wished to deal with symbols that are
referenced all over the source tree rather than just in a single
directory.

Another thing I'd like to do is to introduce a "boilerplate" mechanism
whereby help text that's repeated in multiple options gets stored just
once and dragged in when it's needed. I've made a start on doing that
with the current Configure.help file and the `make config` and `make
menuconfig` commands, and have patches available against the 2.0.39,
2.2.20 and 2.4.17 trees to provide the base implementation for `make
config` but gather that such is pointless as those commands will soon be
extracted from the Linux kernel.

Best wishes from Riley.

Re: State of the new config & build system

[Eric S. Raymond]

Linus Torvalds <torvalds@transmeta.com>:
> My pet peeve is "centralized knowledge". I absolutely detested the first
> versions of cml2 for having a single config file, and quite frankly I
> don't think Eric has even _yet_ separated things out enough - why does the
> main "rules.cml" file have architecture-specific info, for example?

I'm not certain what you're objecting to, and I want to understand it.
There are rules that use architecture symbols to suppress things like
bus types.  I presume that's not a problem for you, but tell me if it is.

My best guess is that you're objecting to the archihacks and kernelhacking
menus, or the architecture-dependent derivations down around line 330.

In general what's going on here is actually the beginnings of an attempt to 
replace architecture-dependent questions with architecture-*independent*
questions.  It looks kind of ugly right now because it's too early in
the game to mess with the config-symbol namespace -- but, for example, I
want to merge the MATH_EMULATION and MATHEMU symbols eventually.  And 
there ought to be a generic set of toggles for kernel-debugging that
present to the user as cross-platform capabilities rather than platform-
specific switches.  

In those two menus I've gathered together architecture-specific
symbols that I think ought to merge into cross-platform capabilities.
But I know there is other cruft in there for historical reasons.  Since
you've brought up the point, I'll do a cleanup pass on these and see
how much I can exile to the arch/*/rules.cml files.

There isn't really any help for the ceoss-platform derivations.  There
are exactly four of these. I've worked hard at holding them to a
minimum:

derive HAVE_DEC_LOCK from (SMP and (ALPHA or X86_CMPXCHG)) or SPARC or PPC
derive HIGHMEM from HIGHMEM4G or HIGHMEM64G or SPARC
derive MAC_HID from (ALL_PPC and INPUT!=n) or (MAC and INPUT_ADBHID)
derive PC_KEYB from ARM_PC_KEYB or MIPS_PC_KEYB

If you notice that each right-hand part includes port symbols from at
least two different architectures, I think it will be clearer why these
are necessary. 

CML1's way of doing this had the problem that it was hard to know by
inspection of the rulebase under what circumstances a given symbol was
actually turned on.  This is why CML2 has a rule that each symbol is
derived (or occurs in a menu) exactly once.  With some work I could
relax this restriction, but I don't want to -- it's a major factor in
keeping the rulebase's complexity down in the range that a human brain
can mentally model.

> That's a big step backwards as far as I'm concerned - we didn't use to
> have those stupid global files, and each architecture could do it's own
> config rules. Eric never got the point that to me, modularity is _the_
> most important thing for maintenance.

Oh, no, I got that all right.  What I have been trying to do is trade
off correctly between modularity (which helps maintenance) and the
advantages to the configurator *users* of having a global capability
namespace, single-apex menu structure, and the symbols-to-prompts
mapping in one file.  These choices weren't made at random.

You don't readily see their advantages because you have a
nose-to-the-code, maintainer perspective (quite properly so, in most
cases).  But in designing the configuration system, simplifying life
for *users* is just as important, if not more so.  Sometimes this
implies not going as far in the direction you favor direction as you
might like (monolithic Configure.help is an example).

> Something I also asked for the config system at least a year ago was to
> have Configure.help split up. Never happened. It's still one large ugly
> file. Driver or architecture maintainers still can't just change _their_
> small fragment, they have to touch a global file that they don't "own".

Yes, there are two reasons for this.

The contingent, historical reason is that I wanted to get
Configure.help in good shape before thinking about dispersing it.
That work is now done (though you haven't kept up to date with it).

The design reason is that having a single file with all the symbol-to-prompt
mappings in it is really helpful when you want to localize the rulebase for
another language.  I'm still leaning towards keeping symbols.cml together
just to make it easier for people to do and distribute translations of it.

I think this is an issue that is rising in importance.  I have no problem
with assuming that kernel hackers are English-literate, but it's no longer
an assumption we should make about people *building* kernels.  I want
to encourage CML2 and question-string localizations for French.  And
German.  And Thai.  And Ethiopian.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

If I were to select a jack-booted group of fascists who are 
perhaps as large a danger to American society as I could pick today,
I would pick BATF [the Bureau of Alcohol, Tobacco, and Firearms].
        -- U.S. Representative John Dingell, 1980

Re: State of the new config & build system

[Larry McVoy]

On Fri, Dec 28, 2001 at 08:42:44PM +1100, Keith Owens wrote:
> "All" I need to do is have one server process that reads the big list
> once and the other client processes talk to the server.  Much less data
> involved means faster conversion from absolute to standardized names.

Actually, if you use the mdbm code, you can have a server process which
reads the data, stashes it in the db, touchs ./i_am_done, and exits.
"client" processes do a 

	while (!exists("i_am_done")) usleep(100000);
	m = mdbm_open("db", O_RDONLY, 0, 0);
	val = mdbm_fetch_str(m, "key");
	etc.

No sockets, no back and forth, runs at mmap speed.

If you tell me what the data looks like that needs to be in the DB, i.e.,
how to get it, I'll code you up the "server" side.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Alexander Viro]

On Fri, 28 Dec 2001, Eric S. Raymond wrote:

> The design reason is that having a single file with all the symbol-to-prompt
> mappings in it is really helpful when you want to localize the rulebase for
> another language.  I'm still leaning towards keeping symbols.cml together
> just to make it easier for people to do and distribute translations of it.
> 
> I think this is an issue that is rising in importance.  I have no problem
> with assuming that kernel hackers are English-literate, but it's no longer
> an assumption we should make about people *building* kernels.  I want
> to encourage CML2 and question-string localizations for French.  And
> German.  And Thai.  And Ethiopian.

You are nuts.  OK, you've got these translations.  Now what?  $FOO adds
a new option.  Should he, by any chance, supply all relevant translations
in the same patch?  No?  Good, so we are going to have them permanently
out of sync.  Better yet, option changes its meaning.  Now we have
English variant with new semantics and Thai one with the old.  Happy,
happy, joy, joy...

And that's aside of the real problem with "internationalization" - quality
of translations _sucks_.  Always.  Yes, USAnian to English is easy.  But
that's it.  I've tried to use LANG=ru_RU.koi8-r.  It had lasted a couple
of days.  You end up reconstructing the English original and translating
it to Russian - and boy, does that process annoy...

Frankly, I find it very amusing that advocates of i18n efforts tend to
be either British or USAnians.  Folks, get real - your languages are
too close to show where the problems are.  I can see how doing that
gives you a warm fuzzy feeling, but could you please listen to those
of us who have to deal with the resulting mess for real?

Re: State of the new config & build system

[Richard Gooch]

Larry McVoy writes:
> On Fri, Dec 28, 2001 at 08:42:44PM +1100, Keith Owens wrote:
> > "All" I need to do is have one server process that reads the big list
> > once and the other client processes talk to the server.  Much less data
> > involved means faster conversion from absolute to standardized names.
> 
> Actually, if you use the mdbm code, you can have a server process which
> reads the data, stashes it in the db, touchs ./i_am_done, and exits.
> "client" processes do a 
> 
> 	while (!exists("i_am_done")) usleep(100000);
> 	m = mdbm_open("db", O_RDONLY, 0, 0);
> 	val = mdbm_fetch_str(m, "key");
> 	etc.
> 
> No sockets, no back and forth, runs at mmap speed.

That sounds like a better approach. I got a bit nervous when Keith
talked about a "server process". Made me think I'm going to have to
install some daemon, or I'm going to have a pile of background
processes being left behind (no matter how careful you are, you always
end up with some "leakage" of stale processes).

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: State of the new config & build system

[Eric S. Raymond]

Alexander Viro <viro@math.psu.edu>:
> > I think this is an issue that is rising in importance.  I have no problem
> > with assuming that kernel hackers are English-literate, but it's no longer
> > an assumption we should make about people *building* kernels.  I want
> > to encourage CML2 and question-string localizations for French.  And
> > German.  And Thai.  And Ethiopian.
> 
> You are nuts.  OK, you've got these translations.  Now what?  $FOO adds
> a new option.  Should he, by any chance, supply all relevant translations
> in the same patch?  No?

No.  The usual way to handle this, of course, is to fall back on the English
where you don't have translations.  Imperfect, but liveable.

>                       Good, so we are going to have them permanently
> out of sync.  Better yet, option changes its meaning.  Now we have
> English variant with new semantics and Thai one with the old.  Happy,
> happy, joy, joy...

Which is why there are organized translation groups that do periodic
translation updates for software that has registered with them.  This
doesn't eliminate the problem, but it can keep it within manageable bounds
that make having localizations better than not.  I deal with this regularly
with respect to fetchmail.

Anyway, options change semantics only very rarely in the kernel rulebase.
Trust me on this as I've been maintaining the CML2 rulebase for 18 months;
I have a better handle on the frequency of these events than *anyone* else.
You are worrying about a non-problem in this case.

> And that's aside of the real problem with "internationalization" - quality
> of translations _sucks_.  Always.

No, not always.  I read French, Italian, and Spanish; I can puzzle out
technical prose in a couple of other languages.  I can read
fetchmail's .po files and *see* that they don't suck.  

> Frankly, I find it very amusing that advocates of i18n efforts tend to
> be either British or USAnians.

That's not my experience.  I've had technical problems with GNU
gettext (unrelated to quality of translation) severe enough that I've
come very close to dropping localization support twice.  The people
who plead with me not to drop it have been non-Anglophones.

It may be that the reason our experiences have been different is because we 
focus on different target languages.  But I think my experience is an
existence proof that there *is* demand for localization and that meeting
it can have useful results.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

I do not find in orthodox Christianity one redeeming feature.
	-- Thomas Jefferson

Re: State of the new config & build system

[Legacy Fishtank]

On Fri, Dec 28, 2001 at 10:02:01AM -0800, Linus Torvalds wrote:
> [ Btw, Jeff, any reason why you changed your name to "Legacy Fishtank"? It
>   took a few mails before I noticed that it also said "garzik" in the
>   fine print;]

Away-from-home account and a long story :)

	Jeff

Re: State of the new config & build system

[Legacy Fishtank]

On Fri, Dec 28, 2001 at 10:02:01AM -0800, Linus Torvalds wrote:
> Something I also asked for the config system at least a year ago was to
> have Configure.help split up. Never happened. It's still one large ugly
> file. Driver or architecture maintainers still can't just change _their_
> small fragment, they have to touch a global file that they don't "own".
> 
> So if somebody really wants to help this, make scripts that generate
> config files AND Configure.help files from a distributed set. And once you
> do that, you could even imagine creating the old-style config files
> (without the automatic checking and losing some information) from the
> information.

For single-file drivers, I like Becker's (correct credit?) system...
about 10 lines of metadata is embedded in a C comment, and it includes
the Config.in and Configure.help info.

	Jeff

Re: State of the new config & build system

[Eric S. Raymond]

Legacy Fishtank <garzik@havoc.gtf.org>:
> For single-file drivers, I like Becker's (correct credit?) system...
> about 10 lines of metadata is embedded in a C comment, and it includes
> the Config.in and Configure.help info.

I proposed implementing something like this about a year ago (to
replace the nasty centralized knowledge in the MAINTAINERS and CREDITS
files) and was shot down.

I'd be happy to take another swing at this problem once the kbuild-2.5/CML2
transition is done.  But I don't think we should let it block us from
having the good results we can get from that change.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

"Those who make peaceful revolution impossible 
will make violent revolution inevitable."
	-- John F. Kennedy

Re: State of the new config & build system

[Larry McVoy]

More numbers.  I coded up a little program (included below) which
reads paths from stdin, lstats() them, and builds an MDBM of inode ->
pathname entries.  I ran that 10 times on the 2.4 kernel, which had 8679
files matching *.[chSs].  I did a little tuning of page size and inital
DB size (reduces page split costs) and got it down to 105 millisecs from
200, so we're at 12 usecs per item.  Then I removed the mdbm_store()
call so I was doing everything except that.  That took 7 usecs/item.

Write path summary: the mdbm_store() cost is about 5 usecs/item, which
is about right.  To build a DB of the same number of items as source
files in the kernel should cost less than 50 milliseconds for the DB
part of the work.  In other words, it's basically free.

OK, on to the read path.  I generated the list of inodes as an ascii file
and wrote another program to open the mdbm and fetch each one.  Ran that 
10 times, it cost 40 milliseconds to look up all the items, so that's
about 4 usecs/item including the read of the data from stdin.  That's 
slower than I think it should be and I may go look to see what is going
on, but it's plenty fast for the config/build system.

Here's the code.  Sorry about the perlisms, wait, no I'm not, I like those,
but it will make you look at it twice before it makes sense.

------------------------------------------------------------------------------

/*
 * inode.c - create an MDBM of inode -> path mappings
 */
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include "mdbm.h"
#define	unless(x)	if (!(x))
#define	fnext(buf, f)	fgets(buf, sizeof(buf), f)
#define	u32		unsigned int

void
chomp(char *s)
{
	unless (s && *s) return;
	while (*s && (*s != '\n')) s++;
	*s = 0;
}

u32
inode(char *path)
{
	struct	stat sb;

	if (lstat(path, &sb)) return (0);
	return ((u32)sb.st_ino);
}

int
main()
{
	char	buf[1024];
	MDBM	*m;
	datum	k, v;
	u32	ino;

	unlink("ino.mdbm");
	unless (m = mdbm_open("ino.mdbm", O_RDWR|O_CREAT, 0644, 4<<10)) {
		perror("ino.mdbm");
		exit(1);
	}
	mdbm_pre_split(m, 128);
	while (fnext(buf, stdin)) {
		chomp(buf);
		unless (ino = inode(buf)) {
			perror(buf);
			continue;
		}
		printf("%u\n", ino);
		k.dptr = (void*)&ino;
		k.dsize = sizeof(u32);
		v.dptr = buf;
		v.dsize = strlen(buf) + 1;
		if (mdbm_store(m, k, v, MDBM_INSERT)) {
			perror(buf);
			exit(1);
		}
	}
	mdbm_close(m);
	exit(0);
}

------------------------------------------------------------------------------

/*
 * read.c - read items from the mdbm
 */
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include "mdbm.h"
#define	unless(x)	if (!(x))
#define	fnext(buf, f)	fgets(buf, sizeof(buf), f)
#define	u32		unsigned int

int
main()
{
	char	buf[1024];
	MDBM	*m;
	datum	k, v;
	u32	ino;

	unless (m = mdbm_open("ino.mdbm", O_RDONLY, 0644, 0)) {
		perror("ino.mdbm");
		exit(1);
	}
	while (fnext(buf, stdin)) {
		ino = atoi(buf);
		continue;
		k.dptr = (void*)&ino;
		k.dsize = sizeof(u32);
		v = mdbm_fetch(m, k);
		unless (v.dsize) {
			perror(buf);
			exit(1);
		}
	}
	mdbm_close(m);
	exit(0);
}

Re: State of the new config & build system

[Kai Germaschewski]

[So far, I've generally been trying to keep away from the hot topics, but
I guess it's about time to make that experience. Also, let me add that my
opinion here is of course influenced by the fact that things didn't go the
way I would have liked...]

On Fri, 28 Dec 2001, Keith Owens wrote:

> On Thu, 27 Dec 2001 17:15:45 -0800, 
> Larry McVoy <lm@bitmover.com> wrote:
> >[talking about kbuild 2.5 speed]
> >Then it does seem reasonable to ask that the new one is at least as fast
> >as the old one.
> 
> kbuild 2.4 is fast but inaccurate, kbuild 2.5 is slower but accurate.
> Pick one.

Most problems which exist within kbuild 2.4 are fixable without the 
elaborate rewrite Keith did. The single biggest problem the current system 
has is that modversions get screwed up, since dependencies are screwed up, 
and yes, that's not easily fixable. However, this problem isn't even 
attacked in kbuild 2.5 AFAIK (I think modversions are simply disabled 
there). 

A couple of months ago, I came up with an alternative to kbuild 2.5. It 
doesn't try to have all the features kbuild 2.5 has, but solves the major 
problems with kbuild 2.4. It definitely has things in common with kbuild 
2.5, it also uses the "non-recursive" approach, i.e. the top level 
Makefile includes all the others. It also doesn't have "make dep" but 
builds dependencies with "gcc -MD" plus postprocessing.

I'm not claiming it is complete, and it doesn't even try to add the 
multiple source tree etc. features. Others said one should use proper 
version management instead, and I agree with that, but that's not the 
point.

Non-complete list of pros/cons:
o gets dependencies right, i.e. a new make "whatever" will really rebuild
  everything which is needed. Even *with* CONFIG_MODVERSIONS turned on.
o uses standard tools. I believe people said that one of the advantages
  of UNIX is that you don't need specialized tools for everything, but 
  combine existing tools to reach your goals. The new kbuild has the 
  disadvantage that most is implemented from scratch, the meat is in C 
  programs which probably nobody apart from Keith is familiar with.
  My solution used the standard tool for building, i.e. make + standard
  utilities like sh, sed, grep and the like. I only have one non-standard
  tool, that postprocesses a dependency list: replace 
  include/linux/autoconf.h with the /include/linux/config/options - this
  is needed so that a .config change doesn't cause an entire rebuild
  every time.
o It's actually pretty fast. On my laptop, the time to read all the 
  dependencies when doing a "make bzImage modules" is was about 5 seconds 
  with hot caches. That means a make takes about 5 seconds when there's 
  nothing to do - that's good enough IMHO. When
  doing a full rebuild, the time spent within make is definitely down in 
  the noise, if only a few files get rebuild, it's noticable, but still 
  faster than what the current kbuild system gives.
o The Makefiles in the SUBDIRS look basically the same as currently, only
  a somewhat simpler (no special $(LD) rules for composite objects etc). 
  Keith  implemented a whole new language - I supoose most coders are 
  familiar with normal Makefiles, they have yet to learn the new commands in
  kbuild-2.5 (which, however, is easy, of course)
o It's not nearly as feature-rich as Keith's approach is.
o Behind the scenes, the code is not exactly clear. make is pretty 
  flexible, but it really needs some hacks to do what's needed. So
  if someone wants to understand the build system, it takes some effort -
  same situation as in kbuild-2.4 and -2.5, though.
o I had the major problems solved and things worked fine in my tree. 
  However, I discontinued to work on it months ago, as I saw no way this 
  work would ever be useful for other people - maintaining a build system 
  just for personal use is a bit too much effort.

I don't claim that my work is superior to kbuild-2.5 or anything. (I still
think it may be "good enough", i.e. does solve the current problems - it
doesn't add features, though). But I'm dissatified that there never ever
was even consideration. When I posted ideas/patches to kbuild-devel, I
usually got a response like "this work isn't needed, I developed
kbuild-2.5, which will be the solution to all problems in 2.5".  I also
submitted non-intrusive changes for 2.4, which fixed/simplified things
there without breaking anything, but the answer was about the same,
"kbuild-2.4 is obsolete, for 2.5 it's irrelevant". Well, 2.4 will be
around for some time I guess...

When I replied (with technical arguments), I never heard anything back - 
compare the current thread about just silently dropping mails/patches 
;-(

That's why I decided to drop out of the kbuild business again. (BTW: note 
that this was about kbuild-2.5 only - nothing to do with CML2)

--Kai

Re: State of the new config & build system

[Eric S. Raymond]

Legacy Fishtank <garzik@havoc.gtf.org>:
> Note I am specifically NOT talking about MAINTAINERS and CREDITS.
> -PLEASE- don't obscure my point by mentioning them.

It's the same problem, Jeff.  Same solution, too.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

It will be of little avail to the people, that the laws are made by
men of their own choice, if the laws be so voluminous that they cannot
be read, or so incoherent that they cannot be understood; if they be
repealed or revised before they are promulgated, or undergo such
incessant changes that no man, who knows what the law is to-day, can
guess what it will be to-morrow. Law is defined to be a rule of
action; but how can that be a rule, which is little known, and less
fixed?	-- James Madison, Federalist Papers 62

Re: State of the new config & build system

[Legacy Fishtank]

I think one thing to note is that dependencies is that if you are smart
about it, dependencies -really- do not even change when your .config
changes.

What about a system where Linus runs "make deps" -once- before he
releases a tarball.  This in turn generates dependency information
(perhaps not in purely make format) which includes 'ifdef CONFIG_xxx'
information embedded within.  We know that make can support ifeq
CONFIG_xxx for example...

	Jeff

Re: State of the new config & build system

[Legacy Fishtank]

On Fri, Dec 28, 2001 at 03:45:37PM -0500, Eric S. Raymond wrote:
> Legacy Fishtank <garzik@havoc.gtf.org>:
> > For single-file drivers, I like Becker's (correct credit?) system...
> > about 10 lines of metadata is embedded in a C comment, and it includes
> > the Config.in and Configure.help info.
> 
> I proposed implementing something like this about a year ago (to
> replace the nasty centralized knowledge in the MAINTAINERS and CREDITS
> files) and was shot down.

Note I am specifically NOT talking about MAINTAINERS and CREDITS.
-PLEASE- don't obscure my point by mentioning them.

Dealing with MAINTAINERS and CREDITS in an automated fashion seems more
like pointless masturbation to me.  If you want to find out who needs to
be CC'd on patches, use your brain like I do.

	Jeff

Re: State of the new config & build system

[Linus Torvalds]

On Fri, 28 Dec 2001, Alan Cox wrote:
>
> > So if somebody really wants to help this, make scripts that generate
> > config files AND Configure.help files from a distributed set. And once you
> > do that, you could even imagine creating the old-style config files
>
> Something like:
>
> 	find $TOPDIR -name "*.cf" -exec cat {} \; > Configure.help

For old tools..

> or changing the tools to look for
>
> 	Documentation/Configure/CONFIG_SMALL_BANANA

"small banana"? Freud would go wild.

But no. I don't want it under the Documentation directory: I'd much rather
have them _together_ with the config file.

So the config file format could be something that includes the docs, and
you could do something like

	find . -name '*.cf' -exec grep '^+' {} \; > Configure.help

for old tools, and nw tools would just automatically get the docs from the
same place they get the config info.

And there would _never_ be more than a few entries per config file: you
can imagine having a separate config file for PCI 100Mbps ethernet drivers
and one for ISA drivers.

The current Configure.help is 25k _lines_, and over a megabyte in size. I
would never consider that good taste in programming, why should I consider
it good in documentation?

		Linus

Re: State of the new config & build system

[Linus Torvalds]

On Fri, 28 Dec 2001, Eric S. Raymond wrote:
>
> I'm not certain what you're objecting to, and I want to understand it.
> There are rules that use architecture symbols to suppress things like
> bus types.  I presume that's not a problem for you, but tell me if it is.

It _is_ a problem for me, because I want to do "diffstat" on a patch from
a PPC maintainer, and if I see anything non-PPC, loud ringing
noises go off in my head. I want that diffstat to say _only_

	arch/ppc/...
	include/asm-ppc/...

and nothing else. That way I know that I don't have to worry.

In contrast, if it starts talking about Documentation/Configure.help and
the main config file, I start worrying.

For example, that MATHEMU thing is just ugly. It was perfectly fine in the
per-architecture version, now it suddenly has magic dependencies just
because different architectures call it different things, and different
architectures have different rules on when it's needed.

		Linus

Re: State of the new config & build system

[Linus Torvalds]

On Fri, 28 Dec 2001, Legacy Fishtank wrote:
>
> I think one thing to note is that dependencies is that if you are smart
> about it, dependencies -really- do not even change when your .config
> changes.

Absolutely. I detest "gcc -MD", exactly because it doesn't get this part
right. "mkdep.c" gets this _right_.

		Linus

Re: State of the new config & build system

[Linus Torvalds]

On Fri, 28 Dec 2001, Eric S. Raymond wrote:
> Legacy Fishtank <garzik@havoc.gtf.org>:
> > Note I am specifically NOT talking about MAINTAINERS and CREDITS.
> > -PLEASE- don't obscure my point by mentioning them.
>
> It's the same problem, Jeff.  Same solution, too.

It's not.

We already have pre-file credits information - people can put stuff in
there for their own (C) messages etc. The MAINTAINERS file is a much
higher-level thing which is there to be human-readable.

Note that I do _not_ want to mess up source files with magic comments. I
absolutely detest those. They only detract from the real job of coding,
and do not make anybody happier.

We have a hierarchical filesystem. Most drivers already have

	driver.c
	driver.h

(in fact _very_ few drivers are single-file) and some have a subdirectory
of their own. So why not just have

	driver.conf

and be done with it. No point in messing up the C file with stuff that
doesn't add any information to either the programmer _or_ the compiler.

Then you can make the config file _truly_ readable, and make the format
something like

	define tristate
	CONFIG_SCSI_AIC7XXX: Adaptec AIC7xxx support

	  This driver supports all of Adaptec's PCI based SCSI controllers
	  (not the hardware RAID controllers though) as well as the aic7770
	  based EISA and VLB SCSI controllers (the 274x and 284x series).
	  This is an Adaptec sponsored driver written by Justin Gibbs.  It is
	  intended to replace the previous aic7xxx driver maintained by Doug
	  Ledford since Doug is no longer maintaining that driver.

		depends on CONFIG_SCSI
		depends on CONFIG_PCI
		depends on ...

	define integer
	CONFIG_AIC7XXX_CMDS_PER_DEVICE: Maximum number of TCQ commands per device

		depends on CONFIG_SCSI_AIC7XXX
		default value 253

	define integer
	CONFIG_AIC7XXX_RESET_DELAY_MS: Initial bus reset delay in milli-seconds

		depends on CONFIG_SCSI_AIC7XXX
		default value 15000

	....

and it's readable and probably trivially parseable into both the existing
format (ie some "find . -name '*.conf'" plus sed-scripts) and into cml2 or
whatever.

		Linus

Re: State of the new config & build system

[Martin Dalecki]

Larry McVoy wrote:

>On Fri, Dec 28, 2001 at 12:41:48PM +1100, Keith Owens wrote:
>
>>On Thu, 27 Dec 2001 17:37:39 -0800, 
>>Larry McVoy <lm@bitmover.com> wrote:
>>
>>>A couple of questions:
>>>
>>>a) will 2.5 be as fast as the current system?  Faster?
>>>
>>At the moment kbuild 2.5 ranges from 10% faster on small builds to 100%
>>slower on a full kernel build.  
>>
>
>I don't understand why it would be slower. 
>
Thank's go to basically to python and other excessfull overengineering 
there.

Re: State of the new config & build system

[Eric S. Raymond]

Linus Torvalds <torvalds@transmeta.com>: 
> On Fri, 28 Dec 2001, Eric S. Raymond wrote:
> > I'm not certain what you're objecting to, and I want to understand it.
> > There are rules that use architecture symbols to suppress things like
> > bus types.  I presume that's not a problem for you, but tell me if it is.
> 
> It _is_ a problem for me, because I want to do "diffstat" on a patch from
> a PPC maintainer, and if I see anything non-PPC, loud ringing
> noises go off in my head. I want that diffstat to say _only_
> 
> 	arch/ppc/...
> 	include/asm-ppc/...
> 
> and nothing else. That way I know that I don't have to worry.

Perhaps we're talking past each other.  I don't understand your objection
yet, and I want to so I can design (or redesign) to meet it.

When I talk about "rules that use architecture symbols to suppress
things like bus types" I have in mind things like this:

unless X86 suppress dependent MCA EISA
unless MIPS32 suppress dependent TC
unless (PCI and (X86 or SUPERH)) suppress pci_access
unless (ISA or PCI) suppress dependent IDE
unless PCI suppress dependent USB HOTPLUG_PCI
unless (X86 or ALPHA or MIPS32 or PPC) suppress usb
unless (X86 and PCI and EXPERIMENTAL) or PPC or ARM or SPARC suppress dependent IEEE1394
unless (M68K or ALL_PPC) suppress MACINTOSH_DRIVERS
unless SPARC suppress dependent FC4
unless ARCH_S390==n suppress buses

It seems to me *extremely* unlikely that a typical patch from a PPC maintainer
would mess with any of these!  They're rules that are likely to be written
once at the time a new port is added to the tree and seldom or ever changed
afterwards.

Thus I really don't think you have to worry about spurious spikes in
your diffstat.  The root rules.cml file will not change very often --
I know this is true, because I can look at the RCS history since I
broke it out in response to your request at the Kernel Summit and
*see* that changes have been few and sparse.

> In contrast, if it starts talking about Documentation/Configure.help and
> the main config file, I start worrying.

Rightly so in the latter case.  Configure.help patches shouldn't worry
you, I don't think.  It's not like they can actually break anything.
 
> For example, that MATHEMU thing is just ugly. It was perfectly fine in the
> per-architecture version, now it suddenly has magic dependencies just
> because different architectures call it different things, and different
> architectures have different rules on when it's needed.

It sounds to me like you're agreeing that it *shouldn't* be called
different things, and thus with my goal of cleaning this mess up the
rest of the way.  Yes?  No?

Guidance, please.  I am, as ever, willing to meet your concerns.  
But I have to understand clearly what they are in order to do that.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

"...The Bill of Rights is a literal and absolute document. The First
Amendment doesn't say you have a right to speak out unless the
government has a 'compelling interest' in censoring the Internet. The
Second Amendment doesn't say you have the right to keep and bear arms
until some madman plants a bomb. The Fourth Amendment doesn't say you
have the right to be secure from search and seizure unless some FBI
agent thinks you fit the profile of a terrorist. The government has no
right to interfere with any of these freedoms under any circumstances."
	-- Harry Browne, 1996 USA presidential candidate, Libertarian Party

Re: State of the new config & build system

[Martin Dalecki]

Linus Torvalds wrote:

>[ Btw, Jeff, any reason why you changed your name to "Legacy Fishtank"? It
>  took a few mails before I noticed that it also said "garzik" in the
>  fine print;]
>
>One thing that this big flame-war has brought up is that different people
>like different things. There may be a simpler solution to this: have the
>core dependency files generated from some other file format.
>
>My pet peeve is "centralized knowledge". I absolutely detested the first
>versions of cml2 for having a single config file, and quite frankly I
>don't think Eric has even _yet_ separated things out enough - why does the
>main "rules.cml" file have architecture-specific info, for example?
>
>That's a big step backwards as far as I'm concerned - we didn't use to
>have those stupid global files, and each architecture could do it's own
>config rules. Eric never got the point that to me, modularity is _the_
>most important thing for maintenance.
>
>Something I also asked for the config system at least a year ago was to
>have Configure.help split up. Never happened. It's still one large ugly
>file. Driver or architecture maintainers still can't just change _their_
>small fragment, they have to touch a global file that they don't "own".
>
>So if somebody really wants to help this, make scripts that generate
>config files AND Configure.help files from a distributed set. And once you
>do that, you could even imagine creating the old-style config files
>(without the automatic checking and losing some information) from the
>information.
>
If you go thus far... then I think, that the Configure.help stuff should 
be embedded inside the driver source code
itself. Like for example the postfix MTA code is embedding whole *man* 
pages there. And  *man* pages would be
anyway a more appriopriate and classical place where the current 
Configure.help information should be.

Just lift the code over from there (The extraction is even proper awk 
insead of some perl crap...) and be nearly done.

Re: State of the new config & build system

[Larry McVoy]

On Fri, Dec 28, 2001 at 09:56:53PM +0100, Kai Germaschewski wrote:
> A couple of months ago, I came up with an alternative to kbuild 2.5. It 
> doesn't try to have all the features kbuild 2.5 has, but solves the major 
> problems with kbuild 2.4. 

So has anyone looked at this?  Is this a viable choice?   I've heard nothing
since Kai posted this.  Keith?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Eric S. Raymond]

Martin Dalecki <dalecki@evision-ventures.com>:
> >>At the moment kbuild 2.5 ranges from 10% faster on small builds to 100%
> >>slower on a full kernel build.  
> >
> >I don't understand why it would be slower. 
> >
> Thank's go to basically to python and other excessfull overengineering 
> there.

Bzzzt!  Thank you for playing.

kbuild-2.5 doesn't use Python.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

The possession of arms by the people is the ultimate warrant
that government governs only with the consent of the governed.
        -- Jeff Snyder

Re: State of the new config & build system

[Eric S. Raymond]

Alan Cox <alan@lxorguk.ukuu.org.uk>:
> > I'd be happy to take another swing at this problem once the kbuild-2.5/CML2
> > transition is done.  But I don't think we should let it block us from
> > having the good results we can get from that change.
> 
> It would certainly fit nicely with the existing metadata. We already rip out
> code comments via kernel-doc, and extending it to rip out
> 
> 	-	Help text
> 	-	Web site
> 	-	Version information
> 	-	Man page for the driver
> 	-	Module options
> 
> etc, shouldn't be too challenging. Ok so kernel-doc is in perl and ugly perl
> but if someone hates it enough to rewrite it in python thats a win too 8)

I've been thinking about doing that very thing anyway.  Part of my master
plan to reduce the tree's external dependencies.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

I do not find in orthodox Christianity one redeeming feature.
	-- Thomas Jefferson

Re: State of the new config & build system

[Benjamin LaHaise]

On Fri, Dec 28, 2001 at 02:27:37PM -0800, Linus Torvalds wrote:
> and it's readable and probably trivially parseable into both the existing
> format (ie some "find . -name '*.conf'" plus sed-scripts) and into cml2 or
> whatever.

It's even doable within the .c file (and preferable for small drivers).  
Something like:

	/* mydriver.c .... header blah blah */
	config_requires(CONFIG_INET);
	config_option(CONFIG_MY_FAST_CHIP, "Help info for this");

which gets picked out of the .c files during depend phase, and nullified 
during compile by means of -Iconfig_system.h would even let us get rid of 
Makefiles for drivers.  Wouldn't being able to just drop a .c file (or a 
bunch of .c files) into the tree in the right place be great?  Eliminating 
makefiles means eliminating more conflicts, which might mean more time to 
respond to other issues...

		-ben
-- 
Fish.

Re: State of the new config & build system

[Linus Torvalds]

On Fri, 28 Dec 2001, Alan Cox wrote:
>
> It would certainly fit nicely with the existing metadata. We already rip out
> code comments via kernel-doc, and extending it to rip out
>
> 	-	Help text
> 	-	Web site
...

No no no.

The comments can at least be helpful to programmers, whether ripped out or
not.

Extra stuff is not helpful to anybody, and is just really irritating. I
personally despise source trees that start out with one page of copyright
statement crap, it just detracts from the real _point_ of the .c file,
which is to contain C code. Making it a comment requirement is

 - stupid:
	we have a filesystem, guys

 - slow:
	we don't need to parse every C file we encounter when we can just
	open another file based on filename

 - nonsensical:
	many config options are _not_ limited to one C file

 - hard to parse and read:
	why limit ourself to C comments, when just keeping the thing
	logically separated means that we don't have to.

Having per-function comment blocks, in contrast, makes sense to have
inline:

 - you read the comment when you read the function

 - you might even update the comment when you update the function

 - you have a reasonable 1:1 relationship.

_None_ of those are sensible for config file entries.

		Linus

Re: State of the new config & build system

[Martin Dalecki]

Linus Torvalds wrote:

>On Fri, 28 Dec 2001, Alan Cox wrote:
>
>>It would certainly fit nicely with the existing metadata. We already rip out
>>code comments via kernel-doc, and extending it to rip out
>>
>>	-	Help text
>>	-	Web site
>>
>...
>
>No no no.
>
>The comments can at least be helpful to programmers, whether ripped out or
>not.
>
>Extra stuff is not helpful to anybody, and is just really irritating. I
>personally despise source trees that start out with one page of copyright
>statement crap, it just detracts from the real _point_ of the .c file,
>which is to contain C code. Making it a comment requirement is
>
> - stupid:
>	we have a filesystem, guys
>
Not quite... It is making moving patches  through e-mail around easier...

>

Re: State of the new config & build system

[Alan Cox]

> I'd be happy to take another swing at this problem once the kbuild-2.5/CML2
> transition is done.  But I don't think we should let it block us from
> having the good results we can get from that change.

It would certainly fit nicely with the existing metadata. We already rip out
code comments via kernel-doc, and extending it to rip out

	-	Help text
	-	Web site
	-	Version information
	-	Man page for the driver
	-	Module options

etc, shouldn't be too challenging. Ok so kernel-doc is in perl and ugly perl
but if someone hates it enough to rewrite it in python thats a win too 8)

Alan

Re: State of the new config & build system

[Alan Cox]

> Frankly, I find it very amusing that advocates of i18n efforts tend to
> be either British or USAnians.  Folks, get real - your languages are
> too close to show where the problems are.  I can see how doing that
> gives you a warm fuzzy feeling, but could you please listen to those
> of us who have to deal with the resulting mess for real?

The biggest advocates I see are from the Middle-East and Japan. We already
have people providing translations for Configure.help in several languages.

Alan

Re: State of the new config & build system

[Stewart Smith]

dammit, didn't hit "reply all" grr....

On Saturday, December 29, 2001, at 05:02  AM, Linus Torvalds wrote:

<snip>
> My pet peeve is "centralized knowledge". I absolutely detested the first
> versions of cml2 for having a single config file, and quite frankly I
> don't think Eric has even _yet_ separated things out enough - why does 
> the
> main "rules.cml" file have architecture-specific info, for example?

agreed - it's something that really irritates me too. As Linux is 
running on so many different architectures (some of which are purely 
virtual, such as Usermode Linux and my whacky idea of running it ontop 
of MacOS X) so it seems that keeping all the options for architectures 
separate would make a lot of sense. I've never seen a cross-platform 
binary kernel (although have had scary dreams of one)

<snip>
> So if somebody really wants to help this, make scripts that generate
> config files AND Configure.help files from a distributed set. And once 
> you
> do that, you could even imagine creating the old-style config files
> (without the automatic checking and losing some information) from the
> information.


This shouldn't be too hard should it? In each module directory have a 
config and Configure.help file, then just

find . |grep config

and then cat all the files together. If I have some spare time today 
I'll see if I can hack something up.... :)

------------------------------
Stewart Smith
stewart@softhome.net
Ph: +61 4 3884 4332
ICQ: 6734154

Re: State of the new config & build system

[Kai Germaschewski]

On Fri, 28 Dec 2001, Linus Torvalds wrote:

> On Fri, 28 Dec 2001, Legacy Fishtank wrote:
> >
> > I think one thing to note is that dependencies is that if you are smart
> > about it, dependencies -really- do not even change when your .config
> > changes.
> 
> Absolutely. I detest "gcc -MD", exactly because it doesn't get this part
> right. "mkdep.c" gets this _right_.

Well, -MD gets this right. The dependencies it generates will cause a 
recompile when necessary. Unfortunately, though, it's too good, because 
the dependency on include/linux/autoconf.h will cause lots of unnecessary 
recompiles.

But yes, it seems possible to replace the -MD dependency file, which
depends on a specific config, with a generic dependency file, which knows
about our #ifdef CONFIG_XXX and translates them to the corresponding
ifeq(CONFIG_,) Makefile syntax. It'd make an interesting project, but it
effectively means re-implementing a C preprocessor.

I don't think you can blame gcc -MD for not knowing about the kernel's
CONFIG_ system, though ;-)

From

---
#ifdef CONFIG_XXX
#include <linux/xxx.h>
#endif

#ifdef CONFIG_YYY
const int nr = 10;
#else
const int nr = 100;
#endif
---

you'd have to generate

---
ifeq(CONFIG_XXX,y)
DEPS += include/linux/xxx.h
endif

DEPS += include/config/yyy
---

i.e. the include/config trick has to stay any way.

I don't think the above is necessary, though, the following does work
pretty good (I did it this way, inspired by mec, and I think kbuild-2.5
does it similarly):

Generate dependencies for a .o file when compiling it. 

[ Doing make dep in advance is unnessary. Actually, it's pretty stupid to
generate dependencies for *all* possible object files which you are never
going to compile (think arch/*). If you don't have the object yet, you
don't need to know the dependencies, dependencies only make sense for
recompiles. It's also cheaper to generate dependencies during the compile,
as you need to read the file anyway. Also, dependencies on generated files
cannot be found correctly until these files have been generated. ]

The generated dependencies will always include linux/autoconf.h, which is
correct, but will cause too many recompiles. So, replace linux/autoconf.h
with linux/config/xxx, where xxx are all the config options which appear
in all of the files used to build the object file (which is what -MD gave
you).

The result is still dependencies which are 100% correct. It's that simple. 
The object file gcc generates depends on the command line and all the 
files it reads during the compile. Why make it more complex?

--Kai

Re: State of the new config & build system

[Keith Owens]

cc: list trimmed.

On Fri, 28 Dec 2001 12:01:04 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Fri, Dec 28, 2001 at 08:42:44PM +1100, Keith Owens wrote:
>> "All" I need to do is have one server process that reads the big list
>> once and the other client processes talk to the server.  Much less data
>> involved means faster conversion from absolute to standardized names.
>
>Actually, if you use the mdbm code, you can have a server process which
>reads the data, stashes it in the db, touchs ./i_am_done, and exits.
>"client" processes do a 
>
>	while (!exists("i_am_done")) usleep(100000);
>	m = mdbm_open("db", O_RDONLY, 0, 0);
>	val = mdbm_fetch_str(m, "key");
>	etc.
>
>No sockets, no back and forth, runs at mmap speed.
>
>If you tell me what the data looks like that needs to be in the DB, i.e.,
>how to get it, I'll code you up the "server" side.

I also want updates from the dependency back end code, to remove the
phase 5 processing.  The "extract dependency" code runs after each
compile step so there can be multiple updates running in parallel.  My
gut feeling is that it will be faster to have one database server and
all the back ends talk to that server.  Otherwise each compile will
have overhead for lock, open, mmap, update, close, write back, unlock.
A single threading server removes the need for lock/unlock and can sync
the data to disk after n compiles instead of being forced to do it
after every compile.

If your experience says that doing updates from each compile step
without a server process would not be too slow, let me know.

Re: State of the new config & build system

[Larry McVoy]

> I also want updates from the dependency back end code, to remove the
> phase 5 processing.  The "extract dependency" code runs after each
> compile step so there can be multiple updates running in parallel.  My
> gut feeling is that it will be faster to have one database server and
> all the back ends talk to that server.  Otherwise each compile will
> have overhead for lock, open, mmap, update, close, write back, unlock.
> A single threading server removes the need for lock/unlock and can sync
> the data to disk after n compiles instead of being forced to do it
> after every compile.
> 
> If your experience says that doing updates from each compile step
> without a server process would not be too slow, let me know.

You certainly don't need a server process.   And as was pointed out
earlier, it's nice not to have them, then you don't have to worry 
about them still being there.

I can write you up a multi writer version using in file locks (which
work over NFS, we had do that for BK and I'm pretty sure it is platform
independent, I can't break it).  We have to do this sort of multi
reader/writer crud in BK all the time and have lots of experience with
locking, breaking locks, waiting, NFS, etc.  Much more experience than
we ever wanted :-)

You don't need to sync to disk at all, let the data sit in memory, that's
why mmap is cool.

Give me a spec for what you want, I'll crank out some code.  Maybe I'll 
finally actually be useful to the kernel after all these years...
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Legacy Fishtank]

On Fri, Dec 28, 2001 at 06:05:57PM -0500, Benjamin LaHaise wrote:
> On Fri, Dec 28, 2001 at 02:27:37PM -0800, Linus Torvalds wrote:
> > and it's readable and probably trivially parseable into both the existing
> > format (ie some "find . -name '*.conf'" plus sed-scripts) and into cml2 or
> > whatever.
> 
> It's even doable within the .c file (and preferable for small drivers).  
> Something like:
> 
> 	/* mydriver.c .... header blah blah */
> 	config_requires(CONFIG_INET);
> 	config_option(CONFIG_MY_FAST_CHIP, "Help info for this");

If Linus is willing to buy into "driver.conf" there is no need to stuff
things into the source.  [my previous post made the mistaken assumption
that Linus would not like an additional metadata file like driver.conf]

A per-driver metadata file is IMHO clearly the preferred solution.

	Jeff

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 16:16:03 -0500, 
Legacy Fishtank <garzik@havoc.gtf.org> wrote:
>I think one thing to note is that dependencies is that if you are smart
>about it, dependencies -really- do not even change when your .config
>changes.
>
>What about a system where Linus runs "make deps" -once- before he
>releases a tarball.  This in turn generates dependency information
>(perhaps not in purely make format) which includes 'ifdef CONFIG_xxx'
>information embedded within.  We know that make can support ifeq
>CONFIG_xxx for example...

Then people apply patches and break.  Please read the list of mkdep
bugs before suggesting partial fixes.

http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2,
makefile-2.5_make_dep.html.  I want a system that fixes _all_ the bugs,
not just some of them.

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 14:17:24 -0800 (PST), 
Linus Torvalds <torvalds@transmeta.com> wrote:
>
>On Fri, 28 Dec 2001, Legacy Fishtank wrote:
>>
>> I think one thing to note is that dependencies is that if you are smart
>> about it, dependencies -really- do not even change when your .config
>> changes.
>
>Absolutely. I detest "gcc -MD", exactly because it doesn't get this part
>right. "mkdep.c" gets this _right_.

Sorry, it does not.  Everybody is attacking little bits of the
dependency problem, any solution that does not fix _all_ 9 problems in
http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2,
makefile-2.5_make_dep.html is not a complete fix.

Yes, some of the problems with mkdep can be fixed in the current design
but there is one problem that is inherently unfixable.  make dep is a
manual process so it relies on users knowing when they have to rerun
make dep AND THEY DON'T DO IT!  Please do not say "I always run make
dep" after a change, I guarantee that you are the exception.  Users
apply patches and do not run make dep, then wonder why their kernel is
broken.

Dependencies _do_ change when your .config changes, the list of files
that are included varies.  gcc -MD gets this exactly right, gcc knows
which files it read.  mkdep does an incorrect approximation, see tyhe
bug list in makefile-2.5_make_dep.html.

The errors in mkdep were acceptable as long as only kernel hackers
built their own kernels, they could be relied upon to manually run
commands when necessary.  The target population has changed, more and
more beginners are building kernels and too many are getting it wrong.
I am aiming at the entire population, not that small subset who have
been building kernels since the year dot.

Any build system that silently fails when users forget to run a command
is a broken system.  kbuild 2.5 fixes _all_ 9 problems with mkdep, it
also positions us for correct modversion handling.  kbuild 2.4 is
faster, inaccurate and manual, kbuild 2.5 is slower, accurate and
totally automatic.

I know how to speed up 2.5.  What I don't have is time to rewrite the
code for speed, I am too busy tracking kernel changes because kbuild
2.5 is not in the kernel yet.

Linus, you have a choice between a known broken build system and a
clean and reliable system, which is slightly slower in mark 1.  Please
add kbuild 2.5 to the kernel, then I will have time to rewrite the core
programs for speed.  Mark 2 of the core code will be significantly
faster.

ps. I don't want mail discussing individual bug fixes to mkdep.  Code
    that does not fix _all_ 9 bugs listed in makefile-2.5_make_dep.html
    is pointless.

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 17:39:08 +0000 (GMT), 
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> * make dep is only run once
>> 
>> Personally, I don't see this as a big problem.  Just tell people to 
>
>I run make dep whenever I change config. Guess what - one cmp and I can
>automate that as part of make. If the .config doesnt match the
>.configwhendep then its time to make dep again

Don't forget that any change that affects a file's #include list
(including patches) requires you to run make dep.

make dep does not handle generated files at all, change the code that
generates a file and the file is regenerated but make dep did not pick
up the file the first time so kbuild 2.4 does not detect the change.
Another make dep will fix that of course.  So now we have

  Run make dep after -

    A change to .config.
    Any source change, it might have changed the #include list.
    Any source or command line change that affects generated files.

How do you automate that?  You end up saying that you always run make
dep.

>> That dependencies are absolute is also not a thing that has bothered me 
>> too much, it's always possible to run "make dep" after moving a tree, 
>> on the other hand, I don't use NFS a lot anymore, so I can see it being 
>> a problem in other environments.
>
>sed works too, as do symlinks

For people who know what they are doing, not for the larger population
that are struggling with the kernel build process.  kbuild 2.5 is
designed to work accurately and automatically for everyone, from the
high druids of the kernel down to the lowliest newbie.

Re: State of the new config & build system

[Alan Cox]

> dependency problem, any solution that does not fix _all_ 9 problems in
> http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2,
> makefile-2.5_make_dep.html is not a complete fix.

All well and good but "takes 100% longer" is number 10 on that list which
you missed off, and the same argument holds for that.

> but there is one problem that is inherently unfixable.  make dep is a
> manual process so it relies on users knowing when they have to rerun
> make dep AND THEY DON'T DO IT!  Please do not say "I always run make

So automate running make dep.

> Linus, you have a choice between a known broken build system and a

So broken its worked for say 5 years without major problem

> ps. I don't want mail discussing individual bug fixes to mkdep.  Code
>     that does not fix _all_ 9 bugs listed in makefile-2.5_make_dep.html
>     is pointless.

And bug number 10 you didnt mention

Re: State of the new config & build system

[Keith Owens]

On Sat, 29 Dec 2001 01:53:17 +0000 (GMT), 
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>> dependency problem, any solution that does not fix _all_ 9 problems in
>> http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2,
>> makefile-2.5_make_dep.html is not a complete fix.
>
>All well and good but "takes 100% longer" is number 10 on that list which
>you missed off, and the same argument holds for that.

You are missing the point Alan.

* The makefile rules are correct now.
* The build is correct now.
* kbuild 2.5 is faster on small compiles and much faster on recompiles
  after small changes.
* kbuild 2.5 is slower on large compiles.
* The speed problem is fixable, given time.  Correctness came first.
* I don't have time to keep tracking multiple kernels and architectures
  _and_ rewrite the core code.
* Once kbuild 2.5 is in the kernel I can spend far less time on
  tracking changes and can redesign the core programs for speed.
* It will get faster!

Why do you expect a change in a development kernel to be perfect first
time?  Look at all the bio changes, I just did a full 2.5.1 build and
had to disable 87 config options before the kernel would build, and
that is ignoring all the warning messages which point to out of date
function definitions.  Is anybody complaining that bio should have
worked first time?

Unlike bio, kbuild 2.5 works, it just needs to be a bit faster.  Put
kbuild 2.5 in the kernel and I will have a faster version within 2
weeks.

Re: State of the new config & build system

[Alan Cox]

> Unlike bio, kbuild 2.5 works, it just needs to be a bit faster.  Put
> kbuild 2.5 in the kernel and I will have a faster version within 2
> weeks.

Ok. I was assuming from what you had said that we were talking about months
before it got up to a sane speed. If its 2 weeks then I have absolutely no
problems with that.

Alan

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 14:51:13 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Fri, Dec 28, 2001 at 09:56:53PM +0100, Kai Germaschewski wrote:
>> A couple of months ago, I came up with an alternative to kbuild 2.5. It 
>> doesn't try to have all the features kbuild 2.5 has, but solves the major 
>> problems with kbuild 2.4. 
>
>So has anyone looked at this?  Is this a viable choice?   I've heard nothing
>since Kai posted this.  Keith?

I looked back through the kbuild mail for Kai's suggestions, I may not
have them all.

RFD: Tracking indirect dependencies [long]

  We knocked this back and forth for a while.  We both agree that
  extracting dependencies after compile is correct, where we differed
  was the mechanism.  In fact I have currently implemented Kai's
  approach (lots of little files) as a stepping stone to storing the
  data in a database.  It turns out that one of the reasons that kbuild
  2.5 is slow is handling all the little files containing dependency
  data.

[PATCH] removal of list-multi

  I agree with the patch but that was December 2000, in code freeze,
  and again in April 2001, AFAICR Linus had said "2.5 soon".  This
  patch is worth resurrecting for 2.4.

Auto detection of changed commands/flags

  That was a decent fix for part of the problem, but it did not address
  tracking user commands nor host compiles.  It did not allow for
  separate source and object trees, for read only source trees, nor did
  it handle the more esoteric cases like modules being built from
  multiple directories.

I am not interested in partial fixes, I want the whole kbuild problem
list to be cleared.  Fixes that only solve part of the problem tend to
be filed and ignored.

Kai, did I miss any of your patches?

Re: State of the new config & build system

[Legacy Fishtank]

On Sat, Dec 29, 2001 at 12:26:49PM +1100, Keith Owens wrote:
> On Fri, 28 Dec 2001 16:16:03 -0500, 
> Legacy Fishtank <garzik@havoc.gtf.org> wrote:
> >I think one thing to note is that dependencies is that if you are smart
> >about it, dependencies -really- do not even change when your .config
> >changes.
> >
> >What about a system where Linus runs "make deps" -once- before he
> >releases a tarball.  This in turn generates dependency information
> >(perhaps not in purely make format) which includes 'ifdef CONFIG_xxx'
> >information embedded within.  We know that make can support ifeq
> >CONFIG_xxx for example...
> 
> Then people apply patches and break.

s/break/update dependencies/

I assumed this was blindingly obvious, but I guess not.

	Jeff

Re: State of the new config & build system

[Legacy Fishtank]

On Sat, Dec 29, 2001 at 12:27:24PM +1100, Keith Owens wrote:
> Dependencies _do_ change when your .config changes, the list of files
> that are included varies.

1) "#ifdef CONFIG_FOO #include ..." is usually wrong and a bug.  But
that is a tangent and I digress.

2) Such changes can be expressed without regenerating all dependencies.


> Linus, you have a choice between a known broken build system and a
> clean and reliable system, which is slightly slower in mark 1.  Please
> add kbuild 2.5 to the kernel,

Your system is known broken because it is 100% slower.

My kernel builds work just fine now, your changes gain me nothing,
while COSTING me productivity.  I see no gains, only costs, with your
kbuild-2.5 system as it exists.

Keith the target audience is Linus and Alan and ME etc.  We are the
kernel hackers that perform kernel -development-.  Making end-user
builds easier is NOT a primary nor secondary nor tertiary goal here.
Make my life easier first.  Fuck Aunt Tillie.  Aunt Tillie can get
her kernels from a vendor.

	Jeff

Re: State of the new config & build system

[Legacy Fishtank]

On Sat, Dec 29, 2001 at 12:44:00PM +1100, Keith Owens wrote:
> For people who know what they are doing, not for the larger population
> that are struggling with the kernel build process.  kbuild 2.5 is
> designed to work accurately and automatically for everyone, from the
> high druids of the kernel down to the lowliest newbie.

Kernel building is not for newbies.

I'm all for lowering the barrier of entry until it affects the
productivity of people actually hacking on the code.

	Jeff

Re: State of the new config & build system

[Mike Castle]

On Fri, Dec 28, 2001 at 10:58:03PM -0500, Legacy Fishtank wrote:
> s/break/update dependencies/
> 
> I assumed this was blindingly obvious, but I guess not.

To YOU and other kernel hackers, yes.

But not to everyone.

Plus, as I understand it, it will be faster to:

apply a patch and rebuild with kbuild 2.5

than to:

apply a patch, make dep && make bzImage.

Correct?

mrc
-- 
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

Re: State of the new config & build system

[Keith Owens]

On Fri, 28 Dec 2001 20:21:39 -0800, 
Mike Castle <dalgoda@ix.netcom.com> wrote:
>On Fri, Dec 28, 2001 at 10:58:03PM -0500, Legacy Fishtank wrote:
>> s/break/update dependencies/
>> 
>> I assumed this was blindingly obvious, but I guess not.
>
>To YOU and other kernel hackers, yes.
>
>But not to everyone.
>
>Plus, as I understand it, it will be faster to:
>
>apply a patch and rebuild with kbuild 2.5
>
>than to:
>
>apply a patch, make dep && make bzImage.
>
>Correct?

As long as the patch does not change an include file that is used a
lot, yes, a patch and make will be significantly faster using kbuild
2.5.

What Mr. Fishtank seems to overlook is that kbuild 2.5 is far more
flexible and accurate than 2.4, including features that lots of people
want, like separate source and object trees.  Now that the overall
kbuild design is correct, the core code can be rewritten for speed.
And that will be done a couple of weeks after kbuild 2.5 goes into the
kernel, then I expect kbuild 2.5 to be faster than kbuild 2.4 even on
full builds.

Re: State of the new config & build system

[Arnaldo Carvalho de Melo]

Em Sat, Dec 29, 2001 at 03:44:10PM +1100, Keith Owens escreveu:
> On Fri, 28 Dec 2001 20:21:39 -0800, 
> Mike Castle <dalgoda@ix.netcom.com> wrote:
> >On Fri, Dec 28, 2001 at 10:58:03PM -0500, Legacy Fishtank wrote:
> >> s/break/update dependencies/
> >> 
> >> I assumed this was blindingly obvious, but I guess not.
> >
> >To YOU and other kernel hackers, yes.
> >
> >But not to everyone.
> >
> >Plus, as I understand it, it will be faster to:
> >
> >apply a patch and rebuild with kbuild 2.5
> >
> >than to:
> >
> >apply a patch, make dep && make bzImage.
> >
> >Correct?
> 
> As long as the patch does not change an include file that is used a
> lot, yes, a patch and make will be significantly faster using kbuild

And thats something I encourage people to work on: to reduce the includes
dependencies hell, I hope to have the cleanup I did to include/net/sock.h
removing the protocol specific headers from there and the cleanup that
Daniel Phillips is doing in include/net/fs.h in the tree soon, of course if
there's not issues, that we're very interesting to know.

- Arnaldo

Re: State of the new config & build system

[Nicholas Knight]

On Friday 28 December 2001 08:44 pm, Keith Owens wrote:
> On Fri, 28 Dec 2001 20:21:39 -0800,
>
> Mike Castle <dalgoda@ix.netcom.com> wrote:
> >On Fri, Dec 28, 2001 at 10:58:03PM -0500, Legacy Fishtank wrote:
> >> s/break/update dependencies/
> >>
> >> I assumed this was blindingly obvious, but I guess not.
> >
> >To YOU and other kernel hackers, yes.
> >
> >But not to everyone.
> >
> >Plus, as I understand it, it will be faster to:
> >
> >apply a patch and rebuild with kbuild 2.5
> >
> >than to:
> >
> >apply a patch, make dep && make bzImage.
> >
> >Correct?
>
> As long as the patch does not change an include file that is used a
> lot, yes, a patch and make will be significantly faster using kbuild
> 2.5.
>
> What Mr. Fishtank seems to overlook is that kbuild 2.5 is far more
> flexible and accurate than 2.4, including features that lots of
> people want, like separate source and object trees.  Now that the
> overall kbuild design is correct, the core code can be rewritten for
> speed. And that will be done a couple of weeks after kbuild 2.5 goes
> into the kernel, then I expect kbuild 2.5 to be faster than kbuild
> 2.4 even on full builds.
>

What, exactly, is the point of merging something that nobody is going 
to use unless they want to test it, in which case they can grab a patch 
from somewhere?
It's half the speed of the current system. The current system works, no 
matter how horrible its internals can be. That makes the NEW system 
BROKEN. If it's KNOWN to be BROKEN prior to merge then it shouldn't 
even be in a 2.5.*-pre#.

kbuild 2.4 works, but it's ugly! Let's create a new system!
You mean the new system is half the speed but it's more attractive? 
Well I guess we'll go with the new system anyway!

(sound familier?:
Oh look at this new processor with all this fancy shit from <company 
name withheld>! It even scales to absurdly high Mhz numbers!
Let's all spend a ton of money to get it!
Oh wait, you mean this old processor that costs nothing compared to the 
new one outperforms this new one? Oh well, let's get the new one anway!)

Re: State of the new config & build system

[Legacy Fishtank]

On Sat, Dec 29, 2001 at 03:44:10PM +1100, Keith Owens wrote:
> What Mr. Fishtank seems to overlook is that kbuild 2.5 is far more
> flexible and accurate than 2.4, including features that lots of people
> want, like separate source and object trees.

I don't see the masses, or, well, anybody on lkml, clamoring for this.

IIRC from the kernel summit SGI was the only entity clamoring for this.


> Now that the overall
> kbuild design is correct, the core code can be rewritten for speed.
> And that will be done a couple of weeks after kbuild 2.5 goes into the
> kernel, then I expect kbuild 2.5 to be faster than kbuild 2.4 even on
> full builds.

Ok... you want kbuild into 2.5 ASAP, only to submit a rewrite two weeks later?

If so it makes even less sense to get kbuild into 2.5.x now.

	Jeff

Re: State of the new config & build system

[Miles Lane]

On Fri, 2001-12-28 at 22:59, Nicholas Knight wrote:

<snip>

> > What Mr. Fishtank seems to overlook is that kbuild 2.5 is far more
> > flexible and accurate than 2.4, including features that lots of
> > people want, like separate source and object trees.  Now that the
> > overall kbuild design is correct, the core code can be rewritten for
> > speed. And that will be done a couple of weeks after kbuild 2.5 goes
> > into the kernel, then I expect kbuild 2.5 to be faster than kbuild
> > 2.4 even on full builds.
> >
> 
> What, exactly, is the point of merging something that nobody is going 
> to use unless they want to test it, in which case they can grab a patch 
> from somewhere?

You don't seem to be reading Keith's message.

The point of merging is that Keith needs time to fix the 
performance problem.  Plus, additional testing would probably 
be helpful.

> It's half the speed of the current system. The current system works, no 
> matter how horrible its internals can be. That makes the NEW system 
> BROKEN. 

No, it's known to be slow in some circumstances.

> If it's KNOWN to be BROKEN prior to merge then it shouldn't 
> even be in a 2.5.*-pre#.

Uh, many drivers cannot be built in the current 2.5 tree.  
Temporary brokenness is acceptable in the development tree.  
It is meant to be _unstable_.  I recall periods when the 
2.3 kernel was corrupting data for many users.  This period
lasted about a week, IIRC.  The kbuild 2.5 system will slow 
people down, but not hose their development system installations.
I personally think two weeks of working at a slower pace is 
an acceptable trade-off for the longterm benefits that Keith
has mentioned.  It seems odd that several people in this
discussion seem to have ignored the repeated statements that
Keith will have little time for dealing with the performance
rewrite until the multiple kernel tree synchronization 
time sink goes away.  Telling Keith that he needs to go on
spinning his wheels while he cannot find time to deal with
the problem you are asking him to fix seems sort of unhelpful.
Perhaps you'd be willing to assist him in the rewrite?

	Miles

Re: State of the new config & build system

[Nicholas Knight]

On Friday 28 December 2001 11:42 pm, Miles Lane wrote:
> On Fri, 2001-12-28 at 22:59, Nicholas Knight wrote:
>
> <snip>
>
> > > What Mr. Fishtank seems to overlook is that kbuild 2.5 is far
> > > more flexible and accurate than 2.4, including features that lots
> > > of people want, like separate source and object trees.  Now that
> > > the overall kbuild design is correct, the core code can be
> > > rewritten for speed. And that will be done a couple of weeks
> > > after kbuild 2.5 goes into the kernel, then I expect kbuild 2.5
> > > to be faster than kbuild 2.4 even on full builds.
> >
> > What, exactly, is the point of merging something that nobody is
> > going to use unless they want to test it, in which case they can
> > grab a patch from somewhere?
>
> You don't seem to be reading Keith's message.
>
> The point of merging is that Keith needs time to fix the
> performance problem.  Plus, additional testing would probably
> be helpful.

The performance problem is fixed by a rewrite, which will be done 
shortly after its merge, thus, what is the point of merging now? You 
don't seem to be reading either my message nor Keith's.

>
> > It's half the speed of the current system. The current system
> > works, no matter how horrible its internals can be. That makes the
> > NEW system BROKEN.
>
> No, it's known to be slow in some circumstances.
>
> > If it's KNOWN to be BROKEN prior to merge then it shouldn't
> > even be in a 2.5.*-pre#.
>
> Uh, many drivers cannot be built in the current 2.5 tree.
> Temporary brokenness is acceptable in the development tree.

Oh? Those drivers that cannot be built are a consequence of early work 
to get something ELSE working.
Merging kbuild 2.5 is not *neccisary* to get anything else working in 
2.5, thus so long as it's broken, it should not be merged. What is so 
hard to understand about this?

> It is meant to be _unstable_.  I recall periods when the
> 2.3 kernel was corrupting data for many users.  This period
> lasted about a week, IIRC.  The kbuild 2.5 system will slow

I don't recall there being any intention to cause corruption. There is 
clearly intent here to merge something that shouldn't be.

> people down, but not hose their development system installations.
> I personally think two weeks of working at a slower pace is
> an acceptable trade-off for the longterm benefits that Keith
> has mentioned.

So, it's acceptable to annoy developers and 2.5 testers for two weeks 
because a bad decision was made with full knowledge of the consequences?

> It seems odd that several people in this
> discussion seem to have ignored the repeated statements that
> Keith will have little time for dealing with the performance
> rewrite until the multiple kernel tree synchronization
> time sink goes away. 

Make it go away. This is intended to first be used in 2.5, right? So 
concentraite on getting it WORKING in 2.5, THEN worry about 
backporting. kbuild 2.4 works for 2.4, kbuild 2.5 does not need to be 
maintained for 2.4 now that 2.5 is up. Make it work in 2.5, then worry 
about backporting it. (is there an echo in here?)

> Telling Keith that he needs to go on
> spinning his wheels while he cannot find time to deal with
> the problem you are asking him to fix seems sort of unhelpful.

If he can't find time to fix his own project, it's not my fault, nor is 
it the fault of any kernel developer that he will be irritating the 
shit out of with both the merging and the subsequent problems.

> Perhaps you'd be willing to assist him in the rewrite?

Sure, first you'll need to make me a pro at the language(s) in use.
I've stated repeatedly that I am not a developer for the Linux kernel. 
I'm an annoying guy that tries to give the developers an occasional 
reality check, which usualy fails.

BTW, Jeff Garzik (Legacy Fishtank) just said practicaly the same thing 
I'm saying, why don't you go flame him now?

Re: State of the new config & build system

[Mike Castle]

On Sat, Dec 29, 2001 at 12:02:07AM -0800, Nicholas Knight wrote:
> The performance problem is fixed by a rewrite, which will be done 
> shortly after its merge, thus, what is the point of merging now? You 
> don't seem to be reading either my message nor Keith's.

Did you catch the point, mentioned several times, that Keith needs to be
relieved of the duty of constantly re-syncing the the patch so that he has
time to do the rewrite.

That is the point of merging now.

Otherwise you'll never get the speed-up you're whining about.

mrc
-- 
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

Re: State of the new config & build system

[Andrew Morton]

Legacy Fishtank wrote:
> 
> On Sat, Dec 29, 2001 at 03:44:10PM +1100, Keith Owens wrote:
> > What Mr. Fishtank seems to overlook is that kbuild 2.5 is far more
> > flexible and accurate than 2.4, including features that lots of people
> > want, like separate source and object trees.
> 
> I don't see the masses, or, well, anybody on lkml, clamoring for this.

Clamour.

The current system has some significant problems.  Pet peeves:

- Failure to rebuild the right things after you've applied a patch
- Doesn't work when the same tree is accessed via different paths
  (make dep on local machine, build across nfs)
- Mysterious recompilation of things which you've already compiled.

> IIRC from the kernel summit SGI was the only entity clamoring for this.
> 
> > Now that the overall
> > kbuild design is correct, the core code can be rewritten for speed.
> > And that will be done a couple of weeks after kbuild 2.5 goes into the
> > kernel, then I expect kbuild 2.5 to be faster than kbuild 2.4 even on
> > full builds.
> 
> Ok... you want kbuild into 2.5 ASAP, only to submit a rewrite two weeks later?

An optimisation of one bit, Keith says.  I'd guess that his two-week
estimate is optimistic because he'll have a busy two weeks supporting
the patch once it goes in, but whatever.
 
> If so it makes even less sense to get kbuild into 2.5.x now.

Keith says it speeds up builds where only a small number of files
have changed.  For me, that's the common case.

I'd like to hear more from Keith on where this 100% actually occurs,
but if he says it's fixable in a (give him four) week timeframe,
I believe him.

As you know, I'd be more concerned about moves to drop support
for the older and much faster gcc versions.  If you're not using
egcs-1.1.2, you're already a very patient person.

>         Jeff

Fish.

-

Re: State of the new config & build system

[Anton Blanchard]

 
Hi,

> Taking nothing away from Tridge, I like Tridge, I'd like to see numbers.
> I'm sure that Tridge's stuff is great, but we were very motivated to 
> go well beyond the normal effort when we wrote this code.

How large is your core db stuff? The thing I like about tdb is that it
is very simple, only recently growing over 1024 lines.

Anton

Re: State of the new config & build system

[Daniel Phillips]

On December 29, 2001 09:13 am, Andrew Morton wrote:
> Legacy Fishtank wrote:
> > 
> > On Sat, Dec 29, 2001 at 03:44:10PM +1100, Keith Owens wrote:
> > > What Mr. Fishtank seems to overlook is that kbuild 2.5 is far more
> > > flexible and accurate than 2.4, including features that lots of people
> > > want, like separate source and object trees.
> > 
> > I don't see the masses, or, well, anybody on lkml, clamoring for this.
> 
> Clamour.

Clamour.

Broke my tree yesterday, rm -rf was the fastest/easiest way out.

Immediately after make -j2 bzImage, make bzImage seems to rebuild about half 
the tree.

Many incidents of time-wasting breakage over the last couple of years for me.

> Keith says it speeds up builds where only a small number of files
> have changed.  For me, that's the common case.

Ooh, yes!

> I'd like to hear more from Keith on where this 100% actually occurs,
> but if he says it's fixable in a (give him four) week timeframe,
> I believe him.

He said something about reloading dependencies on each compile.

--
Daniel

Re: State of the new config & build system

[Wayne.Brown]

Legacy Fishtank <garzik@havoc.gtf.org> wrote:

>I don't see the masses, or, well, anybody on lkml, clamoring for this.

I'm one of the masses who are *not* clamoring for this.  Neither kbuild 2.5 nor
CML2 will provide any benefits for me; I'm going to be enduring them (because I
have no choice), rather than welcoming them.

Wayne

Re: State of the new config & build system

[Kai Germaschewski]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3878 bytes --]


[CC trimmed]

On Sat, 29 Dec 2001, Keith Owens wrote:

> [...] 
> Auto detection of changed commands/flags
> 
>   That was a decent fix for part of the problem, but it did not address
>   tracking user commands nor host compiles.  It did not allow for
>   separate source and object trees, for read only source trees, nor did
>   it handle the more esoteric cases like modules being built from
>   multiple directories.

Some people asked for what my patch looked like, so I resurrected it. I 
moved it to 2.5.2-pre3. As it caused lots of rejects (maintaining kbuild 
is a non-trivial task, isn't it, Keith?), I only fixed the problems for my 
.config. So the attached patch will break with most .configs. However, 
people who want to take a look can take my .config and play with that - 
it's strictly proof-of-concept only. 
(The breakage is easily fixable, it's just quite a bit of mostly trivial 
work).

And yes, it doesn't fix the separate source / objects tree issue. But IMO
that's not a problem that needs to be fixed, it's an additional feature
which I didn't even try to attack (it may be possible to get there with
VPATH, though).

To try it, patch your kernel,

mv ../config-2.5 .config
make oldconfig
make # builds bzImage and modules

A subsequent make will only rebuild what's needed:

[kai@vaio linux-2.5.2-pre3.kbuild]$ time make
make -f Makefile.build boot
make[1]: Entering directory 
`/home/kai/kernel/v2.5/linux-2.5.2-pre3.kbuild'
Generating include/linux/compile.h ...
Preprocessing arch/i386/boot/bsetup.s ...
Assembling (AS) arch/i386/boot/bsetup.o ...
Linking arch/i386/boot/bsetup ...
Compiling init/version.o ...
Linking vmlinux ...
Generating System.map ...
Building arch/i386/boot/compressed/piggy.o
Linking arch/i386/boot/compressed/bvmlinux
Copying to arch/i386/boot/compressed/bvmlinux.out
Building arch/i386/boot/bzImage
Root device is (3, 2)
Boot sector 512 bytes.
Setup is 4764 bytes.
System is 715 kB
make[1]: Leaving directory `/home/kai/kernel/v2.5/linux-2.5.2-pre3.kbuild'

real    0m10.230s
user    0m8.510s
sys     0m1.050s

For each make, compile.h is updated, thus causing init/version.o to be 
recompiled and vmlinux to be relinked. - I kept the behavior of the 
current kbuild. Not regenerating compile.h will give:

[kai@vaio linux-2.5.2-pre3.kbuild]$ time make
make -f Makefile.build boot
make[1]: Entering directory 
`/home/kai/kernel/v2.5/linux-2.5.2-pre3.kbuild'
make[1]: Nothing to be done for `boot'.
make[1]: Leaving directory `/home/kai/kernel/v2.5/linux-2.5.2-pre3.kbuild'

real    0m7.155s
user    0m6.790s
sys     0m0.290s


As an example, change one file:

[kai@vaio linux-2.5.2-pre3.kbuild]$ touch drivers/isdn/hisax/callc.c
[kai@vaio linux-2.5.2-pre3.kbuild]$ time make
make -f Makefile.build boot
make[1]: Entering directory 
`/home/kai/kernel/v2.5/linux-2.5.2-pre3.kbuild'
Compiling drivers/isdn/hisax/callc.o ...
Linking module drivers/isdn/hisax/hisax.o ...
make[1]: Leaving directory `/home/kai/kernel/v2.5/linux-2.5.2-pre3.kbuild'

real    0m9.433s
user    0m8.780s
sys     0m0.520s


$ diffstat pp-kbuild-2.5.2-pre3

 00_version                               |    5 
 Documentation/DocBook/Makefile           |    3 
 Makefile                                 |  507 +++----------------------
 Makefile.build                           |  460 ++++++++++++++++++++++
 Makefile.config                          |   32 +
 Makefile.end                             |   37 +
 Makefile.vars                            |  130 ++++++
 Rules.make                               |  326 ----------------
 arch/i386/Makefile                       |   70 ---
 [...]
 scripts/Makefile                         |   23 -
 scripts/fix_dep.c                        |  406 ++++++++++++++++++++
 scripts/mkdep.c                          |  628 -------------------------------
 169 files changed, 1849 insertions(+), 3107 deletions(-)


[-- Attachment #2: Type: APPLICATION/x-bzip2, Size: 43326 bytes --]

[-- Attachment #3: Type: APPLICATION/x-bzip2, Size: 5717 bytes --]

Re: State of the new config & build system

[Rik van Riel]

On Fri, 28 Dec 2001, Linus Torvalds wrote:

> Having per-function comment blocks, in contrast, makes sense to have
> inline:
>
>  - you read the comment when you read the function
>  - you might even update the comment when you update the function
>  - you have a reasonable 1:1 relationship.

Personally I'd like to see each C file have a header like
this too, describing in a few lines what the functions in
this file are supposed to do.

This should make it easier for people to figure out not just
what each C file is about, but also if they should spend their
time wading through this particular C file when in search of
some piece of code.

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: State of the new config & build system

[Rik van Riel]

On Sat, 29 Dec 2001, Keith Owens wrote:

> ps. I don't want mail discussing individual bug fixes to mkdep.  Code
>     that does not fix _all_ 9 bugs listed in makefile-2.5_make_dep.html
>     is pointless.

I guess you presented a good point to not ignore bug number
10 (the speed one) either. ;)

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: State of the new config & build system

[Larry McVoy]

On Sat, Dec 29, 2001 at 08:24:37PM +1100, Anton Blanchard wrote:
> How large is your core db stuff? The thing I like about tdb is that it
> is very simple, only recently growing over 1024 lines.

   1404    4736   32921 mdbm.c

-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: State of the new config & build system

[Christer Weinigel]

Keith Owens wrote:
>   Run make dep after -
> 
>     A change to .config.

Ok

>     Any source change, it might have changed the #include list.
>     Any source or command line change that affects generated files.

Not really, I only need to run "make dep" if I change a #include or
mess with generated files somehow.  When I'm hacking on a driver, I
think I'm supposed to be clever enough to realise when it's time to
run "make dep".

When I apply a patch from somebody else, I usually don't know exactly
what has changed, so to be on the safe side I should always run "make
oldconfig && make dep" after applying a patch.

If I move the kernel tree so that the absolute paths won't match any
more, I also run "make dep".

> How do you automate that?  You end up saying that you always run make
> dep.

I don't want to automate that.  In my opinion trying to automatically
figure out when to run make dep is overkill (except to run make dep a
first time when it hasn't been run before).

Anyway, from what you have said it seems as if the slowdowns are due
to two things, checking dependencies every time and doing the
translation of absolute to relative paths.  I'm not arguing against
kbuild-2.5, what I want to say is that I think that not all of the
nine bugs you mention are showstoppers.  If you can get kbuild-2.5
into the kernel without the absolute->relative translation, why not do
it?

I just noted the option:

    make NO_MAKEFILE_GEN=anything

which allows me to test things rather quickly (does this avoid the
absolute->relative conversion?) so it seems as if you've fixed this
already.  :-)

One thing I'd like to change though:

Error: the previous kbuild used NO_MAKEFILE_GEN, install is not safe
       You must do a clean kbuild without NO_MAKEFILE_GEN before doing
       install

I don't care that it is unsafe, if I know what I'm doing I don't want
the computer to say "Sorry Dave, I can't let you do that".  My current
way of working with embedded systems is to install etherboot on the
machine and the suck the kernel over tftp and mount the root file
system using NFS.  The way I work is usually:

    change a source file (e.g. drivers/char/scx200_watchdog.c)
    make SUBDIRS=drivers/char modules
    make INSTALL_MOD_PATH=/export/nfs/00601D1D1D52 modules_install
    reboot the target system and test the change

Of course I could just copy the module by hand, but when there is a
whole build system that does things, it would be nice to be able to
use it.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Re: State of the new config & build system

[Linus Torvalds]

On Fri, 28 Dec 2001, Legacy Fishtank wrote:
>
> A per-driver metadata file is IMHO clearly the preferred solution.

Note that it doesn't need to be per-driver: there are good reasons to have
"combined" files too. For example, things like "architecture config" could
all be in one file, along with similar drivers (ie "3com network devices",
whatever).

		Linus

Re: State of the new config & build system

[Linus Torvalds]

On Sat, 29 Dec 2001, Keith Owens wrote:
>
> Yes, some of the problems with mkdep can be fixed in the current design
> but there is one problem that is inherently unfixable.  make dep is a
> manual process so it relies on users knowing when they have to rerun
> make dep AND THEY DON'T DO IT!

Don't be silly.

Make the dependency file itself depend on all the files it describes, and
add a makefile rule to re-generate it. Poof, problem gone.

> Dependencies _do_ change when your .config changes,

Only if you do them wrong. Look at mkdep.c - it statically determines the
complete list of header files that _can_ be included, and does not care at
all about what config options there are.

> that are included varies.  gcc -MD gets this exactly right, gcc knows
> which files it read.

Bzzt, it knows the subset of files to read, nothing more.

		Linus

Re: State of the new config & build system

[Viktor Rosenfeld]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

Legacy Fishtank wrote:

> Kernel building is not for newbies.

Crap.  Back in 1995, I had to compile a kernel to get Linux installed,
because the packaged kernel did not include support for ATAPI CD-ROM
drives.  I had no Unix experience whatsoever, basically what you call a
newbie.

And, no, the situation has not changed.  There are

- people/cooperations without Linux kernel compilation experience, who
  might need a feature that's only available in a development kernel,
- *newbies*, that are generally interested in learning Linux in all its
  ways.

Your attitude strikes me as unnessicarily elitist.

Cheers,
Viktor
-- 
Viktor Rosenfeld
WWW: http://www.informatik.hu-berlin.de/~rosenfel/

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: State of the new config & build system

[Dave Jones]

On Sun, 30 Dec 2001, Viktor Rosenfeld wrote:

> > Kernel building is not for newbies.
> Crap.  Back in 1995, I had to compile a kernel to get Linux installed,
> because the packaged kernel did not include support for ATAPI CD-ROM
> drives.  I had no Unix experience whatsoever, basically what you call a
> newbie.

Things have changed dramatically since 1995.
In particular, distros got a lot friendlier to install, and customise.
If theres a valid reason for Aunt Tillie to rebuild her kernel, it means
her distro of choice is doing something wrong.

> - people/cooperations without Linux kernel compilation experience, who
>   might need a feature that's only available in a development kernel,
> - *newbies*, that are generally interested in learning Linux in all its
>   ways.
> Your attitude strikes me as unnessicarily elitist.

Dumbing down the learning curve doesn't necessary make things easier
for people to learn. Reluctance to read documentation for eg is something
that will plague us no matter how simple we make it to build kernels.

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: State of the new config & build system

[Viktor Rosenfeld]

[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]

Dave Jones wrote:

> Things have changed dramatically since 1995.
> In particular, distros got a lot friendlier to install, and customise.
> If theres a valid reason for Aunt Tillie to rebuild her kernel, it means
> her distro of choice is doing something wrong.

Yes, Aunt Tillie should not have to build a new kernel.  But 13-year-old
Joe Geek might want to try that out, as well as your next door neighbor,
who just might be an IT guy, trying to switch his department to Linux.

> Dumbing down the learning curve doesn't necessary make things easier
> for people to learn. Reluctance to read documentation for eg is something
> that will plague us no matter how simple we make it to build kernels.

I'm not talking about dumbing down the learning curve.  This is about
tools that do the job they're supposed to do in a correct way, without
having to rely on some inside voodoo magic, that is poorly documented,
if at all.

Cheers,
Viktor
-- 
Viktor Rosenfeld
WWW: http://www.informatik.hu-berlin.de/~rosenfel/

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: State of the new config & build system

[GOTO Masanori]

At Fri, 28 Dec 2001 23:20:01 +0000 (GMT),
Alan Cox wrote:
> > Frankly, I find it very amusing that advocates of i18n efforts tend to
> > be either British or USAnians.  Folks, get real - your languages are
> > too close to show where the problems are.  I can see how doing that
> > gives you a warm fuzzy feeling, but could you please listen to those
> > of us who have to deal with the resulting mess for real?
> 
> The biggest advocates I see are from the Middle-East and Japan. We already
> have people providing translations for Configure.help in several languages.

Yes. We JF Project (Japan) is still keeping translating Configure.help
into Japanese for the stable kernel version 2.0, 2.2, and 2.4.

We have some interest in distributing our translated-Configure.help,
but, such distribution needs so-high-precious technical translation.
I think to leave quality control of Configure.help from developer
is not good, we have to be so careful, and it's a big problem.

In addition, I think we need a framework for keeping up to date with
the latest Configure.help against translated Configure.help.
Consistency between original Configure.help and translated-Configure.help
must be kept. IMHO, for example, if CONFIG_FOO is changed between 2.4.16 
and 2.4.17, then (translated-)CONFIG_FOO must show in original English,
even if we have only 2.4.16-translated-CONFIG_FOO, and so on.

-- gotom

Re: State of the new config & build system

[Horst von Brand]

Alan Cox <alan@lxorguk.ukuu.org.uk> said:
> Something like:
> 
> 	find $TOPDIR -name "*.cf" -exec cat {} \; > Configure.help 

Make that:

     cat `find $TOPDIR -name "*.cf"` > Configure.help #;-)
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: State of the new config & build system

[Arnaldo Carvalho de Melo]

Em Mon, Dec 31, 2001 at 07:51:23PM -0300, Horst von Brand escreveu:
> Alan Cox <alan@lxorguk.ukuu.org.uk> said:
> > Something like:
> > 
> > 	find $TOPDIR -name "*.cf" -exec cat {} \; > Configure.help 
> 
> Make that:
> 
>      cat `find $TOPDIR -name "*.cf"` > Configure.help #;-)

whatever is faster, do you have trustable benchmark numbers? ;)

Yes, its a joke, have a nice 2002 all!

- Arnaldo

Re: State of the new config & build system

[Horst von Brand]

"Eric S. Raymond" <esr@thyrsus.com> said:

[...]

> Which is why there are organized translation groups that do periodic
> translation updates for software that has registered with them.  This
> doesn't eliminate the problem, but it can keep it within manageable bounds
> that make having localizations better than not.  I deal with this regularly
> with respect to fetchmail.

Translations do suck: In Spanish, there are several "dialects" of computer
terms in common use. Get a book written in one of those, and try to make
sense on texts using another convention. Or just try to read the original
English documentation...

To do a good translation you need (a) good understanding of the source
language (enough to be able to work around bugs in the original rendering),
(b) extensive knowledge of the target language, (c) knowledge of the task
at hand. Getting all three together is hard.

Besides, stuff like comments and help messages does suffer bitrot very much
as it doesn't (much) affect functioning, translations are much worse as
they have even less exposure (== even less selective pressure to stay right).

[...]

> No, not always.  I read French, Italian, and Spanish; I can puzzle out
> technical prose in a couple of other languages.  I can read
> fetchmail's .po files and *see* that they don't suck.  

You _know_ what the English text says. To be able to make sense of a text
when you have a rather clear idea what it says is a lot easier than trying
to puzzle it out when you have no clue.

Not to say that the files in fetchmail aren OK (I looked at them myself
(German, Spanish) a while back and found little to patch).
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: State of the new config & build system

[Peter Samuelson]

    [Alan Cox]
> > > 	find $TOPDIR -name "*.cf" -exec cat {} \; > Configure.help 

  [Horst von Brand]
> >      cat `find $TOPDIR -name "*.cf"` > Configure.help #;-)

[Arnaldo Carvalho de Melo]
> whatever is faster, do you have trustable benchmark numbers? ;)

Fewer forks vs. increased parallelism ... depends on the nature of your
bottlenecks, I guess, and cold vs. hot cache.  Or you could have it
both ways:

	find $TOPDIR -name \*.cf | xargs -n10 cat > Configure.help

...where 10 is tuned by benchmarking. (:

> Yes, its a joke, have a nice 2002 all!

Yeah, same from me..

Peter

Re: State of the new config & build system

[Mike Touloumtzis]

On Fri, Dec 28, 2001 at 12:57:58PM +1100, Keith Owens wrote:
> 
> Unlike the broken make dep, kbuild 2.5 extracts accurate dependencies
> by using the -MD option of cpp and post processing the cpp list.  The
> post processing code is slow because the current design requires every
> compile to read a complete list of all the files, giving O(n^2)
> effects.  Mark 2 of the core code will use a shared database with
> concurrent update so post processing is limited to looking up just the
> required files, instead of reading the complete list every time.

Why not use '$(GCC) -c -Wp,-MD,foo.d foo.c' to generate the dependencies
as a side effect of the regular compile step?  This enables you to skip
the initial dependency preprocessing step entirely, and could lead to a
speedup over even the current fastdep system.  You still have to massage
the dependencies but you can do it based on the side-effect dependency
output of the _previous_ build, to whatever degree that output exists.

This strategy allows for lazy dependency generation in those cases in
which the dependencies need not be known--for example, if floppy.o
doesn't exist, you know it needs to be built no matter which header
files floppy.c may include.  This breaks down in some cases (as when a
.c file depends on a generated .h file) but those breakdown cases can
be explicitly identified, and a full dependency tree be generated for
them in an eager, rather than a lazy, fashion.

It seems like it's worth it if it leads to a near 100% speedup over the
current kbuild 2.5.  The "build whole clean tree" case is a common one
even among kernel developers, e.g. for compile-testing patches before
resending them.

miket

Re: State of the new config & build system

[Keith Owens]

On Mon, 31 Dec 2001 20:03:59 -0800, 
Mike Touloumtzis <miket@bluemug.com> wrote:
>On Fri, Dec 28, 2001 at 12:57:58PM +1100, Keith Owens wrote:
>> 
>> Unlike the broken make dep, kbuild 2.5 extracts accurate dependencies
>> by using the -MD option of cpp and post processing the cpp list.  The
>> post processing code is slow because the current design requires every
>> compile to read a complete list of all the files, giving O(n^2)
>> effects.  Mark 2 of the core code will use a shared database with
>> concurrent update so post processing is limited to looking up just the
>> required files, instead of reading the complete list every time.
>
>Why not use '$(GCC) -c -Wp,-MD,foo.d foo.c' to generate the dependencies
>as a side effect of the regular compile step?  This enables you to skip
>the initial dependency preprocessing step entirely, and could lead to a
>speedup over even the current fastdep system.  You still have to massage
>the dependencies but you can do it based on the side-effect dependency
>output of the _previous_ build, to whatever degree that output exists.

That is exactly what kbuild 2.5 does.  The slowdown occurs when
massaging the -MD dependencies from absolute names to relative path
names.  To support separate source and object trees, renaming of trees,
different names in local and NFS mode etc., the massage code needs a
list of where all the files are before it can convert the absolute
dependencies produced by gcc.  Reading and indexing that file for every
compile is _slow_.

Larry McVoy has sent me the source code to an mmapped database (from
bitkeeper).  Using a shared mmapped database should speed the process
up considerably.

Re: State of the new config & build system

[Peter Samuelson]

[Mike Touloumtzis]
> Why not use '$(GCC) -c -Wp,-MD,foo.d foo.c' to generate the
> dependencies as a side effect of the regular compile step?

As Keith said, kbuild 2.5 *does* use 'gcc -MD' - although the *current*
system does not.

Linus has said that he doesn't like -MD, and he has a point: it only
extracts dependencies for your *current* compile, which means they have
to be rebuilt if you change CONFIG options.  However, those CONFIG
options would cause rebuilding of the file *anyway*, and -MD is almost
free since the preprocessor already has to read the files in question,
so I'm not convinced that it's a big deal.

> The "build whole clean tree" case is a common one even among kernel
> developers, e.g. for compile-testing patches before resending them.

One of the main points of kbuild 2.5 is that, unlike the current
system, it tracks dependencies perfectly.  Thus you should almost never
have to run 'make clean' before test compiling something - unless you
need to see non-fatal compile warnings.

It may take some time to get used to the soon-to-be new reality of "ok,
so I just applied eight kernel patches from three different places but
I know I don't need to bother with 'make clean' because the dependency
system is just *that good*."

Peter

Re: State of the new config & build system

[Pavel Machek]

Hi!

> > > On Sat, Dec 29, 2001 at 03:44:10PM +1100, Keith Owens wrote:
> > > > What Mr. Fishtank seems to overlook is that kbuild 2.5 is far more
> > > > flexible and accurate than 2.4, including features that lots of people
> > > > want, like separate source and object trees.
> > > 
> > > I don't see the masses, or, well, anybody on lkml, clamoring for this.
> > 
> > Clamour.
> 
> Clamour.

Clamour.

Being able to cp -a then build without full rebuild is good. Also make dep
takes  *long* and and bad things happen when you think it was not needed ;-).

									Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: State of the new config & build system

[Dave Jones]

On Thu, 3 Jan 2002, Pavel Machek wrote:

> Being able to cp -a then build without full rebuild is good. Also make dep
> takes  *long* and and bad things happen when you think it was not needed ;-).

And being able to NFS share 1 kernel tree, and be able to do parallel
builds on multiple boxes without having to wait until 1 is finished.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: State of the new config & build system

[Alexander Viro]

On Thu, 3 Jan 2002, Dave Jones wrote:

> On Thu, 3 Jan 2002, Pavel Machek wrote:
> 
> > Being able to cp -a then build without full rebuild is good. Also make dep
> > takes  *long* and and bad things happen when you think it was not needed ;-).
> 
> And being able to NFS share 1 kernel tree, and be able to do parallel
> builds on multiple boxes without having to wait until 1 is finished.

	Sigh...  As soon as we get to prototype change in
getattr()/setattr()/permission() - we get CoW fs.  I.e. equivalent of
*BSD unionfs.  I hope to get around to that stuff around 2.5.4 or so.

Re: State of the new config & build system

[Keith Owens]

On Thu, 3 Jan 2002 15:35:19 -0500 (EST), 
Alexander Viro <viro@math.psu.edu> wrote:
>On Thu, 3 Jan 2002, Dave Jones wrote:
>> And being able to NFS share 1 kernel tree, and be able to do parallel
>> builds on multiple boxes without having to wait until 1 is finished.
>
>	Sigh...  As soon as we get to prototype change in
>getattr()/setattr()/permission() - we get CoW fs.  I.e. equivalent of
>*BSD unionfs.  I hope to get around to that stuff around 2.5.4 or so.

Unionfs and cow fs will be nice but kernel build will not use it.
Users can build a Linux kernel on other operating systems, including
Solaris, Irix, Cygwin etc.  kbuild requires a Posix compliant fs and
GNU tools, but it must not use additional fs features that only exist
on Linux or only on specific versions of Linux.

Re: State of the new config & build system

[Alexander Viro]

On Fri, 4 Jan 2002, Keith Owens wrote:

> On Thu, 3 Jan 2002 15:35:19 -0500 (EST), 
> Alexander Viro <viro@math.psu.edu> wrote:
> >On Thu, 3 Jan 2002, Dave Jones wrote:
> >> And being able to NFS share 1 kernel tree, and be able to do parallel
> >> builds on multiple boxes without having to wait until 1 is finished.
> >
> >	Sigh...  As soon as we get to prototype change in
> >getattr()/setattr()/permission() - we get CoW fs.  I.e. equivalent of
> >*BSD unionfs.  I hope to get around to that stuff around 2.5.4 or so.
> 
> Unionfs and cow fs will be nice but kernel build will not use it.
> Users can build a Linux kernel on other operating systems, including
> Solaris, Irix, Cygwin etc.  kbuild requires a Posix compliant fs and
> GNU tools, but it must not use additional fs features that only exist
> on Linux or only on specific versions of Linux.

<shrug> kernel build doesn't have to use it - if I mount a writable layer
atop of the clean tree and build in the resulting tree, build system
doesn't need to have any idea of that fact.  That's the point - you are
emulating the thing that is generally useful and belongs to different
layer - namely, the kernel.

Re: State of the new config & build system

[Keith Owens]

On Thu, 3 Jan 2002 16:30:55 -0500 (EST), 
Alexander Viro <viro@math.psu.edu> wrote:
><shrug> kernel build doesn't have to use it - if I mount a writable layer
>atop of the clean tree and build in the resulting tree, build system
>doesn't need to have any idea of that fact.

I have one big problem with unionfs and make, it cannot handle this
scenario.

* Mount COW layer over clean tree.
* Edit a file, writing to the COW layer.
* Build the kernel.
* Decide that you don't want the change, delete the COW version,
  exposing the original version of the file, timestamp goes backwards.
* Build the kernel.
* make sees source timestamp < object timestamp and does not rebuild,
  the kernel source and object do not match.

Obviously this is a design flaw in make, using only timestamps has been
shown to be unreliable.  As long as people use the standard make
program, they will have problems with union filesystems.  The problem
is not restricted to unionfs, NFS timestamp skew also affects make, as
well as checking out code from source repositories when the timestamp
goes backwards.

>That's the point - you are
>emulating the thing that is generally useful and belongs to different
>layer - namely, the kernel.

I agree that unionfs is useful but it is not the panacea for kbuild
that you think it is.  The kbuild wrapper around make takes care of the
timestamp problems as well as handling separate source and object
trees, IOW it does unionfs plus a lot more work.

If make did not rely on timestamps I would have been pushing for
unionfs a long time ago, but as long as we are stuck with make's
design, unionfs is not a fix.  I thought about replacing make entirely
with another tool like Scons but decided that none of the other tools
on their own could cope with the peculiarities of the kernel build nor
were they stable enough yet.

Re: State of the new config & build system

[Alexander Viro]

On Fri, 4 Jan 2002, Keith Owens wrote:

> * Mount COW layer over clean tree.
> * Edit a file, writing to the COW layer.
> * Build the kernel.
> * Decide that you don't want the change, delete the COW version,
>   exposing the original version of the file, timestamp goes backwards.

ITYM "creating a whiteout entry".  unlink() on unionfs doesn't expose
the underlying object.

It looks so:

	* each directory in covering layer has a flag (is_transparent)
	* all children of non-transparent directory are non-transparent
	* lookup in non-transparent directory is a usual lookup in covering
layer.
	* lookup in transparent directory
		lookup in covering layer
		if found an object -> return it
		else if found whiteout -> no entry
		else
			do lookup in covered
			if not found -> no entry
			else if found is a directory
				create a directory in covering
				mark it transparent
				return new directory
			else -> return what we found
	* mkdir creates non-transparent directories
	* unlink and rmdir leave whiteout entry
	* attempt to modify file copies it into covering and modifies that copy.

That gives you real copy-on-write semantics - when you remove object it
stays removed; rm -rf foo && mkdir foo gives you an empty directory, etc.
rename() support is messy - especially when it comes to renaming directories
(if it was transparent you need to copy the entire subtree to covering layer).

Whiteouts are usually represented as directory entries with no inode and
type of entry being DT_WHT (14).  Adding support of these beasts into
ext2 is ~ 10 lines of patch.

Re: State of the new config & build system

[Keith Owens]

On Thu, 3 Jan 2002 17:11:37 -0500 (EST), 
Alexander Viro <viro@math.psu.edu> wrote:
>On Fri, 4 Jan 2002, Keith Owens wrote:
>
>> * Mount COW layer over clean tree.
>> * Edit a file, writing to the COW layer.
>> * Build the kernel.
>> * Decide that you don't want the change, delete the COW version,
>>   exposing the original version of the file, timestamp goes backwards.
>
>ITYM "creating a whiteout entry".  unlink() on unionfs doesn't expose
>the underlying object.

It does in kbuild 2.5.  You have a pristine source tree, start a
development layer, edit files, build a kernel, decide your edit was
wrong, delete the updated version, expose the original and kbuild 2.5
still gets it right.  IMHO that model better fits kernel development.

The whiteout model makes it more difficult to revert to the standard
kernel, you have to copy the original code to your writeable layer to
back out changes.  To satisfy the broken make design, you cannot copy
with timestamp.  When the base layer changes to a new release, the COW
version does not get upgraded, even though it is supposed to be
identical to the base layer.

Again, I am not disagreeing with unionfs, it has its uses.  kbuild
using make and relying solely on timestamps to detect changes is not
one of them.  Especially when kbuild has to run on other operating
systems.

Re: State of the new config & build system

[Andreas Bombe]

On Fri, Jan 04, 2002 at 08:50:52AM +1100, Keith Owens wrote:
> On Thu, 3 Jan 2002 16:30:55 -0500 (EST), 
> Alexander Viro <viro@math.psu.edu> wrote:
> ><shrug> kernel build doesn't have to use it - if I mount a writable layer
> >atop of the clean tree and build in the resulting tree, build system
> >doesn't need to have any idea of that fact.
> 
> I have one big problem with unionfs and make, it cannot handle this
> scenario.
> 
> * Mount COW layer over clean tree.
> * Edit a file, writing to the COW layer.
> * Build the kernel.
> * Decide that you don't want the change, delete the COW version,
>   exposing the original version of the file, timestamp goes backwards.
> * Build the kernel.
> * make sees source timestamp < object timestamp and does not rebuild,
>   the kernel source and object do not match.

Isn't that a thinko in there?  The build using the edited file would
happen in the same layer as that file or in another one on top.  If it's
the same, the build would be deleted with the change.

If it's in another one on top you'd be removing a layer in the middle.
I don't know if that should be possible without user intervention
(unmount build and change layers, delete change layer, mount build layer
over source).  There is something said about Unix and ropes, I remember.
Then again I don't know the unionfs idea in these details, so I may be
wrong.

Unionfs as I understand it would be great for editing/patching and
building.  Build a kernel in the pristine sources, mount a COW layer
over it where you patch/edit, build there.  In the ideal case the COW
layer would only build the changed file(s) and link vmlinux with all the
other objects from the pristine build.  This wouldn't affect the
pristine build itself at all, no make problems there when you remove the
COW build&change layer.

-- 
Andreas Bombe <bombe@informatik.tu-muenchen.de>    DSA key 0x04880A44

Re: State of the new config & build system

[Keith Owens]

On Fri, 4 Jan 2002 02:49:38 +0100, 
Andreas Bombe <bombe@informatik.tu-muenchen.de> wrote:
>Unionfs as I understand it would be great for editing/patching and
>building.  Build a kernel in the pristine sources, mount a COW layer
>over it where you patch/edit, build there.  In the ideal case the COW
>layer would only build the changed file(s) and link vmlinux with all the
>other objects from the pristine build.  This wouldn't affect the
>pristine build itself at all, no make problems there when you remove the
>COW build&change layer.

You are talking about removing an entire layer, I am talking about
removing individual files when you decide the edit failed.  Removing
the entire layer works, as long as all changes are always made to the
top layer.  Removing individual files gets into timestamp problems.

Re: State of the new config & build system

[Andreas Bombe]

On Fri, Jan 04, 2002 at 01:31:08PM +1100, Keith Owens wrote:
> On Fri, 4 Jan 2002 02:49:38 +0100, 
> Andreas Bombe <bombe@informatik.tu-muenchen.de> wrote:
> >Unionfs as I understand it would be great for editing/patching and
> >building.  Build a kernel in the pristine sources, mount a COW layer
> >over it where you patch/edit, build there.  In the ideal case the COW
> >layer would only build the changed file(s) and link vmlinux with all the
> >other objects from the pristine build.  This wouldn't affect the
> >pristine build itself at all, no make problems there when you remove the
> >COW build&change layer.
> 
> You are talking about removing an entire layer, I am talking about
> removing individual files when you decide the edit failed.  Removing
> the entire layer works, as long as all changes are always made to the
> top layer.  Removing individual files gets into timestamp problems.

Duh, you are right.  That can't be solved cleanly.

-- 
Andreas Bombe <bombe@informatik.tu-muenchen.de>    DSA key 0x04880A44