Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Dan Kegel]

Richard Gooch wrote:
> As Keith says, the new code is faster and more robust than the old
> code. Given that tracking kernel drift is a significant load on him,
> it makes sense to incorporate the new code now. Once it's in, let
> people get used to it and then we can look at optimising it, if need
> be. Delaying introduction into the kernel tree because it's not 100%
> optimised is wasteful.

Keith also says:
> I am temporarily omitting [modversions] which is (a) currently broken
> (b) is not being used in development kernels and (c) cannot be fixed
> without a radical redesign.  Modversions is not needed right now and
> will be added later.  Everything I have done in kbuild 2.5 is needed
> now

[Caveat: I'm not much of a kernel hacker.]
My only concern with kbuild 2.5 was the lack of modversions,
but since Richard is promising to add them in before the
distros need them, I would have no qualms about kbuild 2.5
totally replacing the old build system for the next 2.5 kernel.
I'm sick and tired of 'make dep'.

What does Alan Cox think?
- Dan

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Sun, 05 May 2002 09:42:35 -0700, 
Dan Kegel <dank@kegel.com> wrote:
>Keith also says:
>> I am temporarily omitting [modversions] which is (a) currently broken
>> (b) is not being used in development kernels and (c) cannot be fixed
>> without a radical redesign.  Modversions is not needed right now and
>> will be added later.  Everything I have done in kbuild 2.5 is needed
>> now
>
>[Caveat: I'm not much of a kernel hacker.]
>My only concern with kbuild 2.5 was the lack of modversions,
>but since Richard is promising to add them in before the
>distros need them,

You misquoted.  Richard is not promising to add modversions, I am.
I maintain both kbuild and modutils, the two halves of the modversion
problem.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Dan Kegel]

Keith Owens wrote:
> 
> On Sun, 05 May 2002 09:42:35 -0700,
> Dan Kegel <dank@kegel.com> wrote:
> >Keith also says:
> >> I am temporarily omitting [modversions] which is (a) currently broken
> >> (b) is not being used in development kernels and (c) cannot be fixed
> >> without a radical redesign.  Modversions is not needed right now and
> >> will be added later.  Everything I have done in kbuild 2.5 is needed
> >> now
> >
> >[Caveat: I'm not much of a kernel hacker.]
> >My only concern with kbuild 2.5 was the lack of modversions,
> >but since Richard is promising to add them in before the
> >distros need them,
> 
> You misquoted.  Richard is not promising to add modversions, I am.
> I maintain both kbuild and modutils, the two halves of the modversion
> problem.

Sorry, that was a fingerslip.  I meant to say Keith.

BTW I'm looking at your kbuild 2.5 performance measurements at
http://www.mail-archive.com/kbuild-devel%40lists.sourceforge.net/msg01434.html
Looks like 9 seconds to rebuild the kernel after a small change
on a quad 700MHz pentium, right?   (Or does 'make phase4' not actually build?)
What would the time be on a dual pentium?
- Dan

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Sun, 05 May 2002 17:02:30 -0700, 
Dan Kegel <dank@kegel.com> wrote:
>BTW I'm looking at your kbuild 2.5 performance measurements at
>http://www.mail-archive.com/kbuild-devel%40lists.sourceforge.net/msg01434.html
>Looks like 9 seconds to rebuild the kernel after a small change
>on a quad 700MHz pentium, right?   (Or does 'make phase4' not actually build?)

Those times are for the full timestamp, dependency and integrity
checking.  phase4 does not build.  Actual kernel build time depends on
what has changed, what needs to be rebuilt and whether you are
compressing the kernel.

>What would the time be on a dual pentium?

For phase1 through phase4, no difference, the startup code is almost
entirely sequential and uses one cpu.  After phase4 and into the
compile steps, more processors are better.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> [Caveat: I'm not much of a kernel hacker.]
> My only concern with kbuild 2.5 was the lack of modversions,
> but since Richard is promising to add them in before the

Keith  ITYM

> distros need them, I would have no qualms about kbuild 2.5
> totally replacing the old build system for the next 2.5 kernel.
> I'm sick and tired of 'make dep'.
> 
> What does Alan Cox think?

I believe we should end up with working Modversions. If they get disabled
for a bit while it gets there its no different to the state of IDE, the
block layer and many scsi drivers right now.

Alan

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Tomas Szepe]

> [Alan Cox <alan@lxorguk.ukuu.org.uk>, May-06 2002, Mon, 16:38 +0100]
>
> > distros need them, I would have no qualms about kbuild 2.5
> > totally replacing the old build system for the next 2.5 kernel.
> > I'm sick and tired of 'make dep'.
> > 
> > What does Alan Cox think?
> 
> I believe we should end up with working Modversions. If they get disabled
> for a bit while it gets there its no different to the state of IDE, the
> block layer and many scsi drivers right now.

Considering the recent threads and Alan's positive word on the topic I gather
the general opinion on kbuild-2.5 inclusion leans towards a warm "yes."  Looks
like the perfect time for Linus to speak...

T.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Ion Badulescu]

On Tue, 07 May 2002 09:54:29 +1000, Keith Owens <kaos@ocs.com.au> wrote:

> The documentation is correct, but incomplete.  It is correct for an end
> user kernel build, i.e. for people who do not change the code or apply
> patches, they just configure the kernel and build it.  It is incomplete
> for developers and for anybody who gets a patch and applies it.

So update the documentation if it's incomplete. Seriously.

This is more of the "world will come to an end if we don't adopt kbuild25
as-is" rhetoric...

> Add or delete #include in a source, add or delete a source, 

Of course it's not nearly as bad as you make it appear. You're still
touching source code which will cause a recompile of that source code.

You have to make one of the changes above, compile, then change an 
included header file _only_, then compile again, and only then the 
missing dependency will create a problem.

> add or delete a config option and you must run make dep to
> ensure that the changes rebuild what is required.

"add or delete" as in "create/remove a CONFIG_FOO definition and make use
of it in the source code", or "select/unselect CONFIG_FOO in .config"?

If the former, again you need to make use of that definition somewhere
so you're changing source which will get recompiled. Only after making 
a second round of changes you start running into missing dependencies. 
As for removing CONFIG_FOO, you'll get missing dependencies if
include/config/foo.h disappears after make *config.

If the latter, I'm at a loss as to how the current system fails to
handle this correctly.

> Modversions are even
> worse, after any change that might affect an exported symbol or
> structure, you must make mrproper (not dep) to calculate and apply the
> new hashes to the entire kernel.

1. kbuild25 doesn't even support modversions yet

2. you are (almost) contradicting yourself. You've stated earlier that
modversions are irrelevant for a development kernel; in that case, they
are equally irrelevant for any kernel compiled by a developer.

3. you have control over modutils so you could easily make the
modversions mismatch case clearer by printing some helpful advise.
Something like "you have a modversions conflict between module XXX
and the kernel, you should recompile both by running this and that
command (make mrproper, whatever)".

> The default for kernel build must be a safe and accurate build.

Perhaps, but this is overkill. It's irrelevant for non-developers,
it's almost irrelevant for wannabe patch testers who can't read the 
documentation, and it's getting in developers' way by slowing them
down unnecessarily:

$ time make -f Makefile-2.5 drivers/net/starfire.o 
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/kgcc' CPP='/usr/bin/kgcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
  phase 1 (find all inputs)
  phase 2 (convert all Makefile.in files)
  phase 3 (evaluate selections)
  phase 4 (integrity checks, write global makefile)
Starting phase 5 (build) for drivers/net/starfire.o
Phase 5 complete for drivers/net/starfire.o

real    0m32.941s
user    0m30.940s
sys     0m1.640s

It takes 32 seconds to do nothing -- and this is a P4/1400...

$ time make -f Makefile-2.5 NO_MAKEFILE_GEN=1 drivers/net/starfire.o 
Starting phase 5 (build) for drivers/net/starfire.o
  Updating global makefile with last set of commands
Phase 5 complete for drivers/net/starfire.o

real    0m21.505s
user    0m17.800s
sys     0m0.280s

First run with NO_MAKEFILE_GEN=1 takes 21 seconds to do nothing... and
still insists on updating the global makefile.

$ time make -f Makefile-2.5 NO_MAKEFILE_GEN=1 drivers/net/starfire.o 
Starting phase 5 (build) for drivers/net/starfire.o
Phase 5 complete for drivers/net/starfire.o

real    0m0.674s
user    0m0.390s
sys     0m0.280s

Second run is more acceptable.

I like the non-recursive make, and I like the ability to easily compile
only the target I need/want. So I won't even try to compare the times 
with those for a 2.4-style build, it's like comparing apples and oranges.
However, 30+ seconds is *slow* no matter how you look at it.

If something like the current kbuild25 gets into the kernel, I'll end up
doing the same thing I'm currently for 2.4: running gcc by hand, with all
the required arguments. Surely it's no worse than 2.4, but it will have
had the opportunity to get better, and missed it...

At the very least, give me the option of create a 
I_DONT_WANT_NO_STINKING_MAKEFILES_TO_BE_REGENERATED
file which does the equivalent of NO_MAKEFILE_GEN=1, preferrably without
the slowdown associated with the first run.

Also try to remember that solving 100% of the reported problems, at any
cost, is not necessarily a desirable goal. Many people here would be
happy to get only the more annoying ones fixed, and have reasonable
workarounds for the rest.

Don't get me wrong, there are many nice things in kbuild25 which I'd like
to see in the official tree. Makefile regeneration is not one of them,
though. Make a system that even a fool can use and only a fool will want
to use it...

Just MHO, of course.

---------------

Other minor things, not related to the discussion above:

- it would be nice if CFLAGS were also printed.
- I think looking for kgcc before gcc is a bad idea. If you really
want something like that, make it look for kgcc-2.5 instead.

Thanks,
Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

Thanks for the thoughtful reply, it makes a change to get a response
from somebody who has actually considered the problems.

Reading your reply, I get the impression that you might subscribe to
the "every kernel builder is an expert who never makes mistakes" model.
If that is the case then we will have to agree to disagree, kbuild 2.5
was designed to be safe when not used by an expert.  Otherwise, read on.

On Tue, 7 May 2002 19:48:48 -0400, 
Ion Badulescu <ionut@cs.columbia.edu> wrote:
>On Tue, 07 May 2002 09:54:29 +1000, Keith Owens <kaos@ocs.com.au> wrote:
>
>> The documentation is correct, but incomplete.  It is correct for an end
>> user kernel build, i.e. for people who do not change the code or apply
>> patches, they just configure the kernel and build it.  It is incomplete
>> for developers and for anybody who gets a patch and applies it.
>
>So update the documentation if it's incomplete. Seriously.

Yes, the documentation should be updated.  But that will not fix the
problem.  Users do not read documentation, they misunderstand it or
they forget about it.  kbuild 2.5 is designed to work for people who do
not read the documentation.

>This is more of the "world will come to an end if we don't adopt kbuild25
>as-is" rhetoric...

I don't see how you get from incomplete documentation to an end of the
world scenario.

>> Add or delete #include in a source, add or delete a source, 
>
>Of course it's not nearly as bad as you make it appear. You're still
>touching source code which will cause a recompile of that source code.
>
>You have to make one of the changes above, compile, then change an 
>included header file _only_, then compile again, and only then the 
>missing dependency will create a problem.

I absolutely agree.  Except you missed the most common cause of
problems with patches.  Users apply a patch, build once, make *config
and adjust a config that the change introduced.  Code affected by the
config change is not rebuilt because make dep was not run.

>As for removing CONFIG_FOO, you'll get missing dependencies if
>include/config/foo.h disappears after make *config.

Not true.  kbuild 2.4 uses $(wildcard) so disappearing config files are
ignored.  It has to use $(wildcard) because the source code already
contains references to non-existent CONFIG variables.

>> Modversions are even
>> worse, after any change that might affect an exported symbol or
>> structure, you must make mrproper (not dep) to calculate and apply the
>> new hashes to the entire kernel.
>
>1. kbuild25 doesn't even support modversions yet

Agreed, I was talking about the 2.4 documentation being incomplete.

>2. you are (almost) contradicting yourself. You've stated earlier that
>modversions are irrelevant for a development kernel; in that case, they
>are equally irrelevant for any kernel compiled by a developer.

What is a developer?  Is it a full blown kernel hacker, is it somebody
just starting on their first update or is it somebody who is following
instructions to apply a bug fix or install a driver that is not in the
kernel yet?

The first category always turn off modversions.  When I asked the
audience at the 2.5 kernel developers conference if they used
modversions, only 2-3 out of 80 did.  AFAIK they only use it for
distributions, not for day to day work.

It is the person starting on their first update or just following
instructions that gets bitten by modversions.  They are the least
likely to read the documentation or to rebuild the kernel from scratch
to turn off modversions first.

My aim in kbuild 2.5 is to support everybody, not just the full blown
hackers.

>3. you have control over modutils so you could easily make the
>modversions mismatch case clearer by printing some helpful advise.
>Something like "you have a modversions conflict between module XXX
>and the kernel, you should recompile both by running this and that
>command (make mrproper, whatever)".

Yes I could, but by then it is too late.  The user has already done the
compile, installed and booted.  Then they find out that kernel build
was stuffed, reboot back to original system and rebuild.

I want to fix the cause, not the symptom.

>> The default for kernel build must be a safe and accurate build.
>
>Perhaps, but this is overkill. It's irrelevant for non-developers,
>it's almost irrelevant for wannabe patch testers who can't read the 
>documentation

On the contrary, it is exactly these people who most need a safe and
accurate build.  They are the ones that get it wrong now, either
because they do not read documentation or because they are just
following instructions.

>and it's getting in developers' way by slowing them
>down unnecessarily:
>
>$ time make -f Makefile-2.5 drivers/net/starfire.o 
>Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/kgcc' CPP='/usr/bin/kgcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
>Generating global Makefile
>  phase 1 (find all inputs)
>  phase 2 (convert all Makefile.in files)
>  phase 3 (evaluate selections)
>  phase 4 (integrity checks, write global makefile)
>Starting phase 5 (build) for drivers/net/starfire.o
>Phase 5 complete for drivers/net/starfire.o
>
>real    0m32.941s
>user    0m30.940s
>sys     0m1.640s
>
>It takes 32 seconds to do nothing -- and this is a P4/1400...

The pre-processing code is currently unoptimized and has full debugging
turned on.  In scripts/Makefile-2.5, Add -O2 -DNDEBUG=1 to PP_CC_FLAGS
and those times will drop by 30%.

>$ time make -f Makefile-2.5 NO_MAKEFILE_GEN=1 drivers/net/starfire.o 
>Starting phase 5 (build) for drivers/net/starfire.o
>  Updating global makefile with last set of commands
>Phase 5 complete for drivers/net/starfire.o
>
>real    0m21.505s
>user    0m17.800s
>sys     0m0.280s
>
>First run with NO_MAKEFILE_GEN=1 takes 21 seconds to do nothing... and
>still insists on updating the global makefile.

For NO_MAKEFILE_GEN I generate a partial set of dependency information
to give make a better chance of detecting changes to headers and
configs.  There is no point in generating that data for every build, it
is only used in NO_MAKEFILE_GEN mode.  That is why I have to update the
global makefile the first time you switch from normal mode to
NO_MAKEFILE_GEN mode.  I will change the message to say

  Preparing build system for NO_MAKEFILE_GEN mode

>$ time make -f Makefile-2.5 NO_MAKEFILE_GEN=1 drivers/net/starfire.o 
>Starting phase 5 (build) for drivers/net/starfire.o
>Phase 5 complete for drivers/net/starfire.o
>
>real    0m0.674s
>user    0m0.390s
>sys     0m0.280s
>
>Second run is more acceptable.
>
>I like the non-recursive make, and I like the ability to easily compile
>only the target I need/want. So I won't even try to compare the times 
>with those for a 2.4-style build, it's like comparing apples and oranges.
>However, 30+ seconds is *slow* no matter how you look at it.

I agree.  Optimize PP_CC_FLAGS and it will go down a bit.  I am working
on speeding up the database code.  But I will not sacrifice build
accuracy for speed.

>If something like the current kbuild25 gets into the kernel, I'll end up
>doing the same thing I'm currently for 2.4: running gcc by hand, with all
>the required arguments. Surely it's no worse than 2.4, but it will have
>had the opportunity to get better, and missed it...

Accuracy first, speed later.

>At the very least, give me the option of create a 
>I_DONT_WANT_NO_STINKING_MAKEFILES_TO_BE_REGENERATED
>file which does the equivalent of NO_MAKEFILE_GEN=1, preferrably without
>the slowdown associated with the first run.

See above, 'Preparing build system for NO_MAKEFILE_GEN mode'.

>Also try to remember that solving 100% of the reported problems, at any
>cost, is not necessarily a desirable goal. Many people here would be
>happy to get only the more annoying ones fixed, and have reasonable
>workarounds for the rest.

But will people get the workarounds right?  The default has to be
something that works every time.  I am not aiming for Aunt Tillie's
level but kbuild 2.4 has proven that relying on human action does not
work, even for people who understand computers.

>- it would be nice if CFLAGS were also printed.

make ... PP_MAKEFILE5_FLAGS=-v gives you the exact command used for
each compile.  Ready for cut and paste if that is what you want to do.

>- I think looking for kgcc before gcc is a bad idea. If you really
>want something like that, make it look for kgcc-2.5 instead.

That came from one of the -ac trees.  No matter which order I use,
somebody will want a different order and complain.  At least kbuild 2.5
tells you what it is using, instead of silently defaulting to an
unexpected value.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> >- I think looking for kgcc before gcc is a bad idea. If you really
> >want something like that, make it look for kgcc-2.5 instead.
> 
> That came from one of the -ac trees.  No matter which order I use,
> somebody will want a different order and complain.  At least kbuild 2.5
> tells you what it is using, instead of silently defaulting to an
> unexpected value.

For 2.2 - for 2.4 using kgcc generally gets you egcs-1.1.2 which tends
to be a bad idea, and for 2.5 its a no go. I'd drop the kgcc search.

Alan

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Wed, 8 May 2002 01:37:38 +0100 (BST), 
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>For 2.2 - for 2.4 using kgcc generally gets you egcs-1.1.2 which tends
>to be a bad idea, and for 2.5 its a no go. I'd drop the kgcc search.

Will be dropped in core-12.

kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Content-Type: text/plain; charset=us-ascii

Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
It is faster, better documented, easier to write build rules in, has
better install facilities, allows separate source and object trees, can
do concurrent builds from the same source tree and is significantly
more accurate than the existing kernel build system.

The arch independent kbuild 2.5 code (core and common) is up to date
with 2.5.12, as are i386 and sparc64.  ia64 support is at 2.5.10, which
was the last ia64 patch against 2.5.  Work is proceeding on arch
dependent kbuild 2.5 rules for superh, s390[x] and ppc.

This version has only been tested on CML1.  kbuild 2.5 has support for
an older version of CML2 but it has not been tested on newer versions
of CML2.

Before I send you the kbuild 2.5 patch, how do you want to handle it?

* Coexist with the existing kernel build for one or two releases or
  delete the old build system when kbuild 2.5 goes in?

  Coexistence for a few days gives a backout, just in case.  It also
  gives a kernel release where the old and new code can be compared,
  useful for architectures that have not been converted yet.

  Deleting the old system at the same time means that unconverted
  architectures cannot build.  OTOH many architectures are already
  broken in the 2.5 kernel.

* I need a quiet period of 24-48 hours (no changes at all) after a new
  kernel release to bring kbuild 2.5 up to the latest release, before
  sending you the complete patch.  Which kernel release do you want
  kbuild 2.5 against?

I would like kbuild 2.5 to go in in the near future.  Keeping up to
date with kernel changes is a significant effort, Makefiles change all
the time, especially when major subsystems like sound and usb are
reorganised.  There are also some changes to architecture code to do it
right under kbuild 2.5 and tracking those against kernel changes can be
painful.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: Exmh version 2.1.1 10/15/1999

iD8DBQE8z/pii4UHNye0ZOoRAlZnAKCm+kmvXHZnGAAwRXl8sFj+cQ+U8ACgwgBG
2tKEQ0ADLtX7NuKxN7x1R4Y=
=cB0p
-----END PGP SIGNATURE-----

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Denis Vlasenko]

On 1 May 2002 12:23, Keith Owens wrote:
> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
> It is faster, better documented, easier to write build rules in, has
> better install facilities, allows separate source and object trees, can
> do concurrent builds from the same source tree and is significantly
> more accurate than the existing kernel build system.

I never used kbuild 2.5 (sorry Keith), but I am tired of
'make mrproper' bug in existing kernel build system.
Whenever my new kernel does not boot, I have to do
full build just to make sure I wasn't bitten
by it again.

I gather there is no such bug in kbuild 2.5.
That's good.
--
vda

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[tomas szepe]

> > Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
> > It is faster, better documented, easier to write build rules in, has
> > better install facilities, allows separate source and object trees, can
> > do concurrent builds from the same source tree and is significantly
> > more accurate than the existing kernel build system.
> 
> I never used kbuild 2.5 (sorry Keith), but I am tired of
> 'make mrproper' bug in existing kernel build system.
> Whenever my new kernel does not boot, I have to do
> full build just to make sure I wasn't bitten
> by it again.
> 
> I gather there is no such bug in kbuild 2.5.
> That's good.

Yeah, I have to say, kbuild 2.5 is definitely a nice thing..
Looking fw to seeing it included in mainline.

Btw, Keith, how's the bug (if it is a bug at all) w/ CONFIG_MODVERSIONS?
Whenever I built a kernel with it set using kbuild 2.5 everything went fine
up to the moment I tried to load a module into the new kernel -- not one would
actually work, complaining about unresolved symbols (at the same time, though,
depmod -ae had nothing to report). Since I couldn't find any info on this
I concluded it might be that CONFIG_MODVERSIONS was considered obsolete and
as such would no longer be supported?

Another question I'd like to ask (soooorry :D) -- would there be a little
cunning target in Makefile-2.5 that'd create the asm-$arch symlink for me
in include/ like kbuild 2.4 does? Whenever I run "make -f Makefile-2.5 clean"
followed by "make -f Makefile-2.5 menuconfig" I get some serious whipping
from kbuild 2.5, cos the asm-$arch symlink gets deleted in the cleaning,
and I have to resurrect it by hand.

cheers,
t.

-- 
"hello it's not like i read my mail so that you have where to offer to sell me
a giant turnip or anything else thankyou." -tomas szepe <kala@pinerecords.com>          

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Thu, 2 May 2002 12:38:10 +0200, 
tomas szepe <kala@pinerecords.com> wrote:
>Yeah, I have to say, kbuild 2.5 is definitely a nice thing..
>Looking fw to seeing it included in mainline.
>
>Btw, Keith, how's the bug (if it is a bug at all) w/ CONFIG_MODVERSIONS?
>Whenever I built a kernel with it set using kbuild 2.5 everything went fine
>up to the moment I tried to load a module into the new kernel -- not one would
>actually work, complaining about unresolved symbols (at the same time, though,
>depmod -ae had nothing to report). Since I couldn't find any info on this
>I concluded it might be that CONFIG_MODVERSIONS was considered obsolete and
>as such would no longer be supported?

kbuild 2.5 deliberately does not support modversions, you can turn it
on but it does nothing.  The original implementation of modversions
does not fit with the way that people build kernels now (apply patches,
change configs, rebuild without make mrproper).  To do modversions
right needs a new version of modutils as well, there is no chance of
that work being started until kbuild 2.5 is in the kernel.

>Another question I'd like to ask (soooorry :D) -- would there be a little
>cunning target in Makefile-2.5 that'd create the asm-$arch symlink for me
>in include/ like kbuild 2.4 does? Whenever I run "make -f Makefile-2.5 clean"
>followed by "make -f Makefile-2.5 menuconfig" I get some serious whipping
>from kbuild 2.5, cos the asm-$arch symlink gets deleted in the cleaning,
>and I have to resurrect it by hand.

It works for me.  menuconfig does not use include/asm-$ARCH.

make -f Makefile-2.5 clean
make -f Makefile-2.5 menuconfig
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/checklist.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/checklist.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/menubox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/menubox.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/textbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/textbox.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/yesno.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/yesno.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/inputbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/inputbox.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/util.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/util.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/msgbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/msgbox.c
gcc -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/checklist.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/menubox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/textbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/yesno.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/inputbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/util.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/msgbox.o -lncurses
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
  phase 1 (find all inputs)
Using defaults found in .config
Preparing scripts: functions, parsing......................................................................................done.

Saving your kernel configuration...

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Uz.ytkownik Keith Owens napisa?:

> kbuild 2.5 deliberately does not support modversions, you can turn it
> on but it does nothing.  The original implementation of modversions
> does not fit with the way that people build kernels now (apply patches,
> change configs, rebuild without make mrproper).  To do modversions
> right needs a new version of modutils as well, there is no chance of
> that work being started until kbuild 2.5 is in the kernel.

How many years was it that I was telling that symbol versioning is
a silly concept not solving any single problem and the implementation is to say
the least ugly?

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> > change configs, rebuild without make mrproper).  To do modversions
> > right needs a new version of modutils as well, there is no chance of
> > that work being started until kbuild 2.5 is in the kernel.
> 
> How many years was it that I was telling that symbol versioning is
> a silly concept not solving any single problem and the implementation is to say
> the least ugly?

Modversions solves a huge number of problems very very well. The fact that
you don't like it doesn't change the reality of the situation.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Uz.ytkownik Alan Cox napisa?:
>>>change configs, rebuild without make mrproper).  To do modversions
>>>right needs a new version of modutils as well, there is no chance of
>>>that work being started until kbuild 2.5 is in the kernel.
>>
>>How many years was it that I was telling that symbol versioning is
>>a silly concept not solving any single problem and the implementation is to say
>>the least ugly?
> 
> 
> Modversions solves a huge number of problems very very well. The fact that
> you don't like it doesn't change the reality of the situation.

Could you name *ONE* please? Maybe the following?

- It's solving the problem of applying quick security
fixes to vendor specific kern src.rpm packeges for the user
very well.

- It solves the problem of too fast kernel compiles as well fine.

- As an added bonus it makes you use
the force flag to insmod more often with binary only modules, which
everybody loves... This gives you the good feeling of polite
questions you have been missing from DOS for so long:
"Do you really wan't to delete this file Y/N"...

- And then we have no better use for our RAM then
storing some extendid redundant string information there of course
as well.

- And of course it is not annoying if you want to move
modules which you have just compiled yourself between
two machines. Or perhaps a compilation host and some testing systems.

Far better sollution then just versioning the kernel release
and expecting people to actually know what they do.
They are finally all loosers, becouse they use a system they
can mess with.

It is far better then providing clean submodule interfaces as well.
And finally it's of course a better sollution then versioning
with the granularity of a whole module, which we just don't
have right now. It would be ridiculous to have some
modules to provide the ABI version information they expose just
to let the clients check it explicitely in far too few bytes like
about 1 or maybe 2. The analogy with shared libraries would be far
too big - becouse of it course turned out there to be not sufficient and
the X11 people didn't show us what true compatibility means and the
glibc people don't know what real man programming is.

What are weak symbols for? Ah yes - we have to hold up the
a.out tradition in it's full glory!

Did I mention that the C++ solution to linker deficiencies is
inferior to module versioning of course as well, becouse
catching the type signature is not what we wan't.

Yes - versioning of every single piece is indeed a very good
solution to the above problems and a nice piece of SW design!

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Kai Germaschewski]

On Thu, 2 May 2002, Martin Dalecki wrote:

> > Modversions solves a huge number of problems very very well. The fact that
> > you don't like it doesn't change the reality of the situation.
> 
> Could you name *ONE* please? Maybe the following?

Are you sure you know what you're talking about?

> - It's solving the problem of applying quick security
> fixes to vendor specific kern src.rpm packeges for the user
> very well.

??? If you patch a kernel in a src.rpm, and rebuild from scratch, 
modversions won't be in your way.

> - It solves the problem of too fast kernel compiles as well fine.

I doubt you really notice the difference (make dep takes a bit longer, 
yes, but so what)

> - As an added bonus it makes you use
> the force flag to insmod more often with binary only modules, which
> everybody loves... This gives you the good feeling of polite
> questions you have been missing from DOS for so long:
> "Do you really wan't to delete this file Y/N"...

If the versioning goes wrong (which does happen with current kbuild, and 
that's a bug), insmod -f won't help you at all to cope with the unresolved 
symbols.

> - And then we have no better use for our RAM then
> storing some extendid redundant string information there of course
> as well.

Oh well, if you care about these couple of hundred bytes.

> - And of course it is not annoying if you want to move
> modules which you have just compiled yourself between
> two machines. Or perhaps a compilation host and some testing systems.

Well, if you have the same config, modules will be interchangeable, if 
not, modversions will prevent using the wrong modules, that's the very 
case it's been designed for. 

If you think you know better, or if any of the points above bother you too
much, you're free to turn off modversions, which I guess most developers
do. But because they're not useful to you, that doesn't mean they're not
useful for anyone.

--Kai

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> > Modversions solves a huge number of problems very very well. The fact that
> > you don't like it doesn't change the reality of the situation.
> 
> Could you name *ONE* please? Maybe the following?

It handles the case when you have modules that don't get rebuilt as part of
the base kernel. It allows you to build fixed kernel images without spending
ages rebuilding all the modules when they otherwise match

> - As an added bonus it makes you use
> the force flag to insmod more often with binary only modules, which
> everybody loves... This gives you the good feeling of polite

Unrelated IMHO

> - And then we have no better use for our RAM then
> storing some extendid redundant string information there of course
> as well.

Which occupies almost no space

> Far better sollution then just versioning the kernel release

The kernel release isnt useful info. Many interfaces are stable across
multiple kernels, many interface issues depend on things other than kernel
version. Lots of people apply patches and don't change the base kernel
version - in fact its hard to do so

> Yes - versioning of every single piece is indeed a very good
> solution to the above problems and a nice piece of SW design!

I think so. It solves the first 95% of the problem. Solving 100% isnt easy
enough to be worth it. Throwing it out and solving 0% of the problem is
not very bright either

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Mike Fedyk]

On Thu, May 02, 2002 at 04:17:36PM +0100, Alan Cox wrote:
> > Far better sollution then just versioning the kernel release
> 
> The kernel release isnt useful info. Many interfaces are stable across
> multiple kernels, many interface issues depend on things other than kernel
> version. Lots of people apply patches and don't change the base kernel
> version - in fact its hard to do so

Changing one line in Makefile is hard?  I do that every time I patch my
kernels.  It lets me track easily what kernels I have installed just by `ls
/boot`, and the name shows up in my kernel .deb. :)

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Sun, 5 May 2002 02:43:04 -0700, 
Mike Fedyk <mfedyk@matchmail.com> wrote:
>On Thu, May 02, 2002 at 04:17:36PM +0100, Alan Cox wrote:
>> The kernel release isnt useful info. Many interfaces are stable across
>> multiple kernels, many interface issues depend on things other than kernel
>> version. Lots of people apply patches and don't change the base kernel
>> version - in fact its hard to do so
>
>Changing one line in Makefile is hard?  I do that every time I patch my
>kernels.  It lets me track easily what kernels I have installed just by `ls
>/boot`, and the name shows up in my kernel .deb. :)

Not hard, just expensive.  From kbuild 2.5.

# FIXME: Current kernel source includes linux/version.h, mainly to get
# KERNEL_VERSION().  version.h also includes UTS_RELEASE which changes every
# time the kernel identifiers change.  The presence of UTS_RELEASE in version.h
# causes lots of unnecessary recompilations, very few places actually want
# UTS_RELEASE.  The new makefile generates separate linux/version.h and
# linux/uts_release.h, with version.h including utsname.h to avoid compilation
# errors.  Find all the source code that needs just UTS_RELEASE and change it to
# include uts_release.h, then remove #include <linux/uts_release.h> from the
# commands below.  KAO

There are less than 10 places in the kernel that really need
UTS_RELEASE.  Alas it is defined in version.h which is included by 99%
of the code, either directly or indirectly.  Change the top level
Makefile and 99% of the kernel gets recompiled.

I have deliberately not fixed this problem in kbuild 2.4.  The full
dependency chain goes Makefile -> version.h -> everything else.
Removing the dependency of Makefile -> version.h looks like a good
idea, but it exposes potential bugs that are currently hidden by the
extra recompiles.

Remove that dependency and other changes to Makefile such as target
changes will not cause rebuilds when they should.  kbuild 2.4 does not
have a complete list of dependencies on the top level Makefile, it
works by accident because of the chain Makefile -> version.h -> 99% of
the kernel.

All fixed in kbuild 2.5, of course.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Arjan van de Ven]

Martin Dalecki wrote:
> 
> Uz.ytkownik Alan Cox napisa?:
> >>>change configs, rebuild without make mrproper).  To do modversions
> >>>right needs a new version of modutils as well, there is no chance of
> >>>that work being started until kbuild 2.5 is in the kernel.
> >>
> >>How many years was it that I was telling that symbol versioning is
> >>a silly concept not solving any single problem and the implementation is to say
> >>the least ugly?
> >
> >
> > Modversions solves a huge number of problems very very well. The fact that
> > you don't like it doesn't change the reality of the situation.
> 
> Could you name *ONE* please? Maybe the following?

It fixes the problem where support people and people who get the
bugreports have
to stare at "impossible" problems for hours and hours to find out that
someone
has insmod'ed a module for a different kernel (be it an athlon module in
a i686 kernel
or someone using perl to replace the built-in kernel version in the .o
file)

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Richard Gooch]

Arjan van de Ven writes:
> someone using perl to replace the built-in kernel version in the .o
> file)

Oh, my! Do people really do that?!?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Użytkownik Richard Gooch napisał:
> Arjan van de Ven writes:
> 
>>someone using perl to replace the built-in kernel version in the .o
>>file)
> 
> 
> Oh, my! Do people really do that?!?

Yes they do, if they wont for example to get some
PCTEL win chip driver working. No matter what Alan and some others
think is good for them :-).

The main problem with mod-versions is the simple fact
that policy doesn't belong in to the kernel it belongs
in the user space. And mod-version is *just policy*.

See... if some impaired project manager at some
linux distro provider associated with hats,
who does decisions like for example basing the main OS
configuration tools on instable programming languages
like python or perl... does decide that he needs
the functionality of MODULEVERSIONS to get full controll about the
users of his crappy product. Why the hell doesn't he let all this
version checking be done for his own kernel cook-up entierly in
his patched mod-utils? And entierly in USER SPACE? He does
have the full scope of things which need the bless of versioning
over them at his hands and there is *no technical* argument why this
should be done in kernel space at all!

It just DOES NOT BELONG in to the kernel-space.

No matter what percentage of supposed problems it solves and
how many problems it in reality adds...

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> The main problem with mod-versions is the simple fact
> that policy doesn't belong in to the kernel it belongs
> in the user space. And mod-version is *just policy*.

Nope. modversions are information about the ABI/API and objects referenced
directly or indirectly from them.  The policy is entirely in modutils.
Modutils has the power to say "well that looks kind of the same I'll bind
that symbol name".

Kernel -> "Here is a helpful set of ABI compatibility hashes"
Modutils -> "This symbol doesnt match, what do we want to do about it. Lets
	     fail". It could equally pick something looking similar.

> It just DOES NOT BELONG in to the kernel-space.

People who start using capital letters always seem to have emotional rather
than logical reasons for their argument.

Alan
--
"Intel sued Cyrix five times and they never won. Intel they just love lawsuits"
		--  Wen Chi Chen, Via's CEO

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Uz.ytkownik Alan Cox napisa?:
>>The main problem with mod-versions is the simple fact
>>that policy doesn't belong in to the kernel it belongs
>>in the user space. And mod-version is *just policy*.
> 
> 
> Nope. modversions are information about the ABI/API and objects referenced
> directly or indirectly from them.  The policy is entirely in modutils.
> Modutils has the power to say "well that looks kind of the same I'll bind
> that symbol name".
> 
> Kernel -> "Here is a helpful set of ABI compatibility hashes"
> Modutils -> "This symbol doesnt match, what do we want to do about it. Lets
> 	     fail". It could equally pick something looking similar.
> 
> 
>>It just DOES NOT BELONG in to the kernel-space.
> 
> 
> People who start using capital letters always seem to have emotional rather
> than logical reasons for their argument.

You are wrong.

ar r module.a module-symbol-versions-copyright-or-whatever
ar r vmlinuz.a symbol-versions-from-System.map

(Perhaps the ld variant with some section magic would be
  looking prettier and technically more correct.)

Shared libraries for example don't look up stuff like this inside
themselfs. (Unless you look at DLL stubs...)
It's the ld.so programm which maintains such data.
modutils don't do anything different from classical late binding.
The natural place for such maintainance work could be for example
the init process, which serves already pretty a similar role for
the kernel like ld.so does for user land applications. It would
provide a convenient point for possible synchronization...

Another analogy is the rpm dependency maintainance.
It's the rpm program - which does checking here and not
the actual application itself during the file-system install.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> Shared libraries for example don't look up stuff like this inside
> themselfs. (Unless you look at DLL stubs...)

Nor does the kernel. Internal symbols are already resolved

> It's the ld.so programm which maintains such data.
> modutils don't do anything different from classical late binding.

Indeed

> The natural place for such maintainance work could be for example
> the init process, which serves already pretty a similar role for

Well actually the logical place to do it is in modutils. Which is where
we do it right now. We even precompute dependancies with depmod much like
the dynamic link caches

> Another analogy is the rpm dependency maintainance.
> It's the rpm program - which does checking here and not
> the actual application itself during the file-system install.

Actually for dynamic stuff the application also does some of it for late
binding and when triggers are used for relations between packages

All of which says the current modutils method is correct 

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Arjan van de Ven]

On Thu, May 02, 2002 at 05:36:26PM +0200, Martin Dalecki wrote:
> U¿ytkownik Richard Gooch napisa³:
> > Arjan van de Ven writes:
> > 
> >>someone using perl to replace the built-in kernel version in the .o
> >>file)
> > 
> > 
> > Oh, my! Do people really do that?!?
> 
> Yes they do, if they wont for example to get some
> PCTEL win chip driver working. No matter what Alan and some others
> think is good for them :-).
> 
> The main problem with mod-versions is the simple fact
> that policy doesn't belong in to the kernel it belongs
> in the user space. And mod-version is *just policy*.
> 
> See... if some impaired project manager at some
> linux distro provider associated with hats,
> who does decisions like for example basing the main OS
> configuration tools on instable programming languages
> like python or perl... does decide that he needs
> the functionality of MODULEVERSIONS to get full controll about the
> users of his crappy product. Why the hell doesn't he let all this
> version checking be done for his own kernel cook-up entierly in
> his patched mod-utils? And entierly in USER SPACE? He does
> have the full scope of things which need the bless of versioning
> over them at his hands and there is *no technical* argument why this
> should be done in kernel space at all!


Thank you for this insult and welcome to my .procmailrc

Oh and please don't forget your medicine tomorrow as you did today..

 
> It just DOES NOT BELONG in to the kernel-space.

That's why it's done by modutils.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Uz.ytkownik Arjan van de Ven napisa?:

> 
>>It just DOES NOT BELONG in to the kernel-space.
> 
> 
> That's why it's done by modutils.
> 

Last time I checked on Linux:

[root@kozaczek j2me]# ls -l /proc/ksyms
-r--r--r--    1 root     root            0 Mai  2 19:42 /proc/ksyms

Compare this with the following on Solaris:

www01:/kernel/drv# ls sy sy.conf
sy       sy.conf
www01:/kernel/drv# file sy
sy:             ELF 32-bit MSB relocatable SPARC Version 1
www01:/kernel/drv# cat sy.conf
#
# Copyright (c) 1992, by Sun Microsystems, Inc.
#
#ident  "@(#)sy.conf    1.4     93/06/03 SMI"

name="sy" parent="pseudo" instance=0;
www01:/kernel/drv#
www01:/kernel/drv# nm sy
0000000000000014 T _fini
0000000000000028 T _info
0000000000000000 T _init
                  U cdev_ioctl
                  U cdev_poll
                  U cdev_read
                  U cdev_write
  ....
0000000000000278 T syclose
0000000000000488 T syioctl
0000000000000140 T syopen
00000000000005b0 T sypoll
0000000000000280 T syread
0000000000000384 T sywrite
www01:/kernel/drv# strings sy
Indirect driver for tty 'sy'
www01:/kernel/drv#

And then think about the fact that they are able to even *patch*
running kernels. There is no way I can be convinced that the whole
versioning stuff is neccessary or a good design for any purpose.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[David S. Miller]

   From: Martin Dalecki <dalecki@evision-ventures.com>
   Date: Thu, 02 May 2002 18:53:23 +0200

   And then think about the fact that they are able to even *patch*
   running kernels. There is no way I can be convinced that the whole
   versioning stuff is neccessary or a good design for any purpose.

Hint: they never changes their ABIs for drivers.  This is why
they can't fix several large scale bugs in their OS.  When the
fix would break every third party Solaris driver on the planet
they simply don't do the fix until the next major relase of the
OS.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Uz.ytkownik David S. Miller napisa?:
>    From: Martin Dalecki <dalecki@evision-ventures.com>
>    Date: Thu, 02 May 2002 18:53:23 +0200
> 
>    And then think about the fact that they are able to even *patch*
>    running kernels. There is no way I can be convinced that the whole
>    versioning stuff is neccessary or a good design for any purpose.
> 
> Hint: they never changes their ABIs for drivers.  This is why
> they can't fix several large scale bugs in their OS.  When the
> fix would break every third party Solaris driver on the planet
> they simply don't do the fix until the next major relase of the
> OS.

Yes I know. But my main point is that they maintain the
whole module symbol and dependency data entierly in user space
and MODULEVERSION in the *linux kernel*, just simply isn't worth the
trouble. Similarly the whole Tainted not tained MODULE_AUTHOR
and so on information simply shouldn't me maintained
in precious kernel RAM space.

Information about module dependencies does get resolved at
the proper level: not centralized in a single one Makefile
alike repository but by an *.conf file placed alongside of it.
The module objects them-self look very much like a simple *stripped*
ELF shared objects and don't contain any hack-in of string arrays
in the .text segment just to provide data which can be trivially
reconstructed and maintained in user space. Module request handling for hot
pluggable devices for example is done entierly inside user
space and so on...

Version consistency get's handled at the proper level - namely
the file packaging. Once and not repeatedly during every time
a particular module get's loaded and so on... And then there is
simple late kernel binding. After all having a messed up /kernel
dir is not a single bit more dangerous then just having a
trashed vmlinuz file.

Overall even the fact aside that they have a much stronger policy
about releases, the overall design is much cleaner then on the
linux side of the world.

Gosh - the whole /devices file-system is an old hat for Solaris.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> Yes I know. But my main point is that they maintain the
> whole module symbol and dependency data entierly in user space

Actually thats also incorrect as far as I can tell

Alan

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Uz.ytkownik Alan Cox napisa?:
>>Yes I know. But my main point is that they maintain the
>>whole module symbol and dependency data entierly in user space
> 
> 
> Actually thats also incorrect as far as I can tell

They maintain a device driver tree there yes. But it's
a single directed tree there.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[David S. Miller]

   From: Alan Cox <alan@lxorguk.ukuu.org.uk>
   Date: Thu, 2 May 2002 20:11:59 +0100 (BST)

   > Yes I know. But my main point is that they maintain the
   > whole module symbol and dependency data entierly in user space
   
   Actually thats also incorrect as far as I can tell
   
To the best of my knowledge, they do something similar
to what modutils does right now when depmod is run, but
at module load time.  Ie. "oh module X needs symbol Y, who
provides symbol Y, lets load that first then retry X"

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> And then think about the fact that they are able to even *patch*
> running kernels. There is no way I can be convinced that the whole
> versioning stuff is neccessary or a good design for any purpose.

I wouldnt pick Solaris as an example. A long time ago they fixed a bug
in the streams code I found that let anyone reconfigure networking. It was
fixed in a day then not released for a year. It cost Sun a lot because
several customers wisely asked why it hadn't been fixed and went with
other products. To this day Sun has never explained officially why it took
a year to fix but I've been informed off the record by sun people I trust
that it was because it broke their module abi so had to be held over for
the next release

Now I don't actually give a hoot whether you implement the module binding
via /proc/kernel.so and C++ like mangling hacks or the _R stuff we do now
but don't confuse the Linux approach of putting a few million users before
a few binary module ISV's with the Solaris one.

Alan

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Kai Germaschewski]

On Thu, 2 May 2002, Keith Owens wrote:

> kbuild 2.5 deliberately does not support modversions, you can turn it
> on but it does nothing.  The original implementation of modversions
> does not fit with the way that people build kernels now (apply patches,
> change configs, rebuild without make mrproper).  To do modversions
> right needs a new version of modutils as well, there is no chance of
> that work being started until kbuild 2.5 is in the kernel.

I would like to object here. Getting dependencies right for modversions is
very much possible in principle, after all modversions are generated in a
deterministic process. (It's also possible in practise, though it's quite
a bit of work).

You're right that modversions are not perfect. It's possible that the ABI 
changes, but the checksum doesn't, but that's very rare. It's also 
possible that the ABI does not change but the checksum does. That happens 
a lot, but it's not really a big problem because that (if done right) will 
just cause spurious rebuilds - correctness isn't affected.

Of course, for people who are patching their kernels a lot, modversions
(again if done right) are a pain in the a**, since they cause a lot of not
really necessary rebuilds. But people who do that supposedly think they
have some idea of how the kernel works and can turn it off - if they get
bitten by ABI changes then, it's their problem. 

Modversions is really essential for distributions, where it's badly needed
to keep users from causing hard to track down crashes by inserting
self-compiled or obtained from whereever else modules into a kernel which
was compiled with a different config.

--Kai

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[David Woodhouse]

kai@tp1.ruhr-uni-bochum.de said:
> > kbuild 2.5 deliberately does not support modversions, you can turn it
> > on but it does nothing.  The original implementation of modversions
> > does not fit with the way that people build kernels now (apply
> > patches, change configs, rebuild without make mrproper).  To do
> > modversions right needs a new version of modutils as well, there
> > is no chance of that work being started until kbuild 2.5 is in 
> > the kernel.

> I would like to object here. Getting dependencies right for
> modversions is very much possible in principle, after all modversions
> are generated in a deterministic process. (It's also possible in
> practise, though it's quite a bit of work).

To what are you objecting? You aren't disagreeing with Keith here. He 
merely said that there's no chance of him working on modversions until
the newer build system that's sane w.r.t. dependencies is incorporated.

> Modversions is really essential for distributions, where it's badly
> needed to keep users from causing hard to track down crashes by
> inserting self-compiled or obtained from whereever else modules into a
> kernel which was compiled with a different config.

Distributions are unlikely to be shipping 2.5 kernels. As long as 
modversions can be reimplemented properly by the time 2.6 is released, 
what's the harm in disabling it for a while?

It's hard enough to keep kbuild-2.5 up to date with recent kernels as it 
is; let's not keep moving the goalposts by adding new requirements for the 
initial adoption -- once it's in and the makefiles are maintaining 
themselves, we can concentrate on reimplementing the niche features.

--
dwmw2

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Kai Germaschewski]

On Thu, 2 May 2002, David Woodhouse wrote:

> > I would like to object here. Getting dependencies right for
> > modversions is very much possible in principle, after all modversions
> > are generated in a deterministic process. (It's also possible in
> > practise, though it's quite a bit of work).
> 
> To what are you objecting? You aren't disagreeing with Keith here. He 
> merely said that there's no chance of him working on modversions until
> the newer build system that's sane w.r.t. dependencies is incorporated.

Well Keith's statement (as I read it) is: modversions are broken, I don't 
support them. My statement is: modversions work 95% of the time, why throw 
them out?

That doesn't mean the could be replaced by something which works more than 
95% of the time later (though 100% will be impossible to achieve anyway 
IMO).

> > Modversions is really essential for distributions, where it's badly
> > needed to keep users from causing hard to track down crashes by
> > inserting self-compiled or obtained from whereever else modules into a
> > kernel which was compiled with a different config.
> 
> Distributions are unlikely to be shipping 2.5 kernels. As long as 
> modversions can be reimplemented properly by the time 2.6 is released, 
> what's the harm in disabling it for a while?
> 
> It's hard enough to keep kbuild-2.5 up to date with recent kernels as it 
> is; let's not keep moving the goalposts by adding new requirements for the 
> initial adoption -- once it's in and the makefiles are maintaining 
> themselves, we can concentrate on reimplementing the niche features.

I merely disagree with the way how things are done here. Al Viro doesn't 
go like: here's a new VFS, everything is handled differently now - oh, and 
for the time being symlinks don't work, I'll fix that until 2.6 (I know 
this is a a bit extreme, but you get the point).

If Keith went like fixing issues one at a time, he wouldn't have that huge 
patch now, which replaces everything and is hard to keep up-to-date. 
There's a lot of orthogonal issues with kbuild which can be solved one at 
a time, e.g. correct dependency generation, cleaning up Makefiles (like 
getting rid of the explicit list-multi link rules), spurious rebuilds, 
building built-in and modular objects in one pass, non-recursive make, ...

My understanding is that the Linux way would have been the latter, going 
one step at a time, as Al Viro demonstrates perfectly with the VFS layer.

This way it would also have been possible to select which features are 
considered worthwhile and which aren't - not "you have to take it all or 
nothing"

Anyway, just my opinion, and yes, I'm admittedly preoccupied.

--Kai
 

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Thu, 2 May 2002 10:40:49 -0500 (CDT), 
Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de> wrote:
>Well Keith's statement (as I read it) is: modversions are broken, I don't 
>support them. My statement is: modversions work 95% of the time, why throw 
>them out?

Build a complete kernel with modversions.  Apply a patch or change a
config option that changes a structure size.  make bzImage, reboot.
modversions are _not_ rebuilt.  The kernel and modules were built using
different ABIs but modversions claims that they are identical.

Third party code is built using a copy of .config and modversions.h.
This assumes that modversions.h was generated using the same .config,
but it is not checked.  The module _may_ have used a different config
but asserts it was built using the same ABI as the kernel (same
modversions).  Result is a module that appears to match the kernel,
when really all you know is that the user claims it matches the kernel.

People think that modversions gives a strong check on ABI compatibility
for third party modules.  Wrong!  What it really gives is a weak
assumption that the user copied two files that are in sync.

Modversions only detect ABI changes if you make mrproper after any
change that affects the symbol versions.  That has to be done manually,
it cannot be automated.  Generation of new symbol versions requires a
recompile of everything marked export-objs after any source or config
change.  But there is no way of telling where the versioned symbols are
consumed, so any change to any versioned symbol requires a complete
kernel rebuild to ensure that every consumer picks up the new version.

Make any change, make mrproper, rebuild from scratch, it is the only
way to ensure that modversions are correct.  Modversions are fine for
distributors, a pain in the neck for everybody else.

95% working?  More like 95% broken!

I know how to do ABI versioning right.  But there is no chance of me
starting work on the correct method of ABI versioning until kbuild 2.5
is in.

>If Keith went like fixing issues one at a time, he wouldn't have that huge 
>patch now, which replaces everything and is hard to keep up-to-date. 
>There's a lot of orthogonal issues with kbuild which can be solved one at 
>a time, e.g. correct dependency generation, cleaning up Makefiles (like 
>getting rid of the explicit list-multi link rules), spurious rebuilds, 
>building built-in and modular objects in one pass, non-recursive make, ...

I have been waiting for somebody to raise the "why not do one bit a
time" argument for kbuild.  That is exactly what I have done!

Modversions are completely broken but are not required for a
development kernel, they will be done later.  There are 89 'FIXME'
comments in the Makefile.in files where changes to source code should
be done to clean up the include mess, those changes will be done later.

Changing from a recursive to non-recursive make is the big change in
kbuild 2.5.  If you think that you can do non-recursive make without
significant changes to the Makefiles, show me the code.

If you think that you can fix all the problems listed in
http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2
without making significant changes to the entire kbuild system, show me
the code.

I have no patience with people who pick the small problems out of
kbuild and fiddle with the Makefiles without considering the entire
problem list.  That is a classic case of ignoring the big problem and
concentrating on the little problem that you know how to fix.

kbuild 2.5 fixes _all_ the problems listed in the history file, except
for modversions which will be done later.  Once you decide to fix the
big problems, you will realise that fiddling with the old system to fix
the little problems is a waste of time and effort.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Martin Dalecki]

Uz.ytkownik Keith Owens napisa?:

> I know how to do ABI versioning right.  But there is no chance of me
> starting work on the correct method of ABI versioning until kbuild 2.5
> is in.

It's shown in the syscall part of the kernel :-) Just don't provide
a too big ABI and stick to it is one possible strategy.

And of course I'm sure you recognize that what we could use is ABI *checking*
and not ABI *versioning* thingee. If one really really want's to do this the
only true one way, well the solution is.... for example CORBA IDL and stuff
if you divide the remote part of CORBA out.

And hell I'm not expecting an ORB to appear in the kernel any time soon.
(However I remember someone once implementid such a beast...)

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Kai Germaschewski]

On Fri, 3 May 2002, Keith Owens wrote:

> Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de> wrote:
> >Well Keith's statement (as I read it) is: modversions are broken, I don't 
> >support them. My statement is: modversions work 95% of the time, why throw 
> >them out?
> 
> Build a complete kernel with modversions.  Apply a patch or change a
> config option that changes a structure size.  make bzImage, reboot.
> modversions are _not_ rebuilt.  The kernel and modules were built using
> different ABIs but modversions claims that they are identical.

Well, my experience is that modversions change, but only parts of the 
object files get rebuild, so what you end up with won't link. Anyway, no 
doubt we agree that this is broken (and I said so).

> Third party code is built using a copy of .config and modversions.h.
> This assumes that modversions.h was generated using the same .config,
> but it is not checked.  The module _may_ have used a different config
> but asserts it was built using the same ABI as the kernel (same
> modversions).  Result is a module that appears to match the kernel,
> when really all you know is that the user claims it matches the kernel.

True, that's one of my "5%" where modversions doesn't do the job - if you 
build, then change .config, then build extra stuff, it won't work. Anyway, 
how would you handle that stupidity, I don't see a way to do it without 
enclosing a copy of .config with every module (or a hash thereof). Which 
then means you have to rebuild everything if you change just any option in 
the .config.

> Modversions only detect ABI changes if you make mrproper after any
> change that affects the symbol versions.  That has to be done manually,
> it cannot be automated.  Generation of new symbol versions requires a
> recompile of everything marked export-objs after any source or config
> change.  But there is no way of telling where the versioned symbols are
> consumed, so any change to any versioned symbol requires a complete
> kernel rebuild to ensure that every consumer picks up the new version.
> 
> 95% working?  More like 95% broken!

Well, you did not quote all I said. My point was that the concept of
modversions catches incompatible ABI changes in 95% of the cases. I also
agreed with you that the build process is currently broken. (and I don't
think it's worth arguing if it's in 95% of the cases or whatever, it needs
fixing).

The dependencies you're describing above are wrong though, maybe that's 
why you couldn't get it right in kbuild-2.5? The dependencies are: If any 
of the objects in $(export-objs) are changed, you need to recalculate the 
checksum. If any checksum changes, you need to rebuild everything that 
includes modversions.h (i.e. every module). 

Yeah, that means a lot of recompilations, fortunately only a fraction of
all sources is in $(export-objs), so it doesn't happen all that often. It
may be possible to speed it up by figuring out which files actually used
the changed symbols, kind a like we currently do with dependencies on
CONFIG_XXX. But I think correctness goes first, the user asked for
modversions, so he may have to take the increased build time.

> Changing from a recursive to non-recursive make is the big change in
> kbuild 2.5.  If you think that you can do non-recursive make without
> significant changes to the Makefiles, show me the code.

Well, we had this thread a couple of months ago, I resurrected a 
(proof-of-concept only) patch I had to do so. You were on the CC list - 
why do I even bother talking to you when you don't listen anyway?

(The patch was not at all production quality, but yes, I check if the
things I'm claiming are doable, and it turns out they are. It's actually
fairly easy for Makefiles which only have standard rules (obj-... +=, etc)
Special cases are harder to handle, that's why I'm currently going through
the Makefiles, trying to remove them as far as possible).

However, there's two reasons to go to a non-recursive build:
o it may be faster than forking make for every subdir
o it handles the case where changes local to one subdir have global 
  effects

W.r.t. the first point, I'm not actually sure that's true for all cases. 
the recursive build doesn't even enter directories you didn't select in 
your .config, for most people, the kernel build never even descends into 
drivers/isdn.

In the non-recursive build, make ends up with a database for all potential 
objects, and has to build the dependency tree from that. Considering that 
every single source file has in the order of 100 files it depends on, 
that's a huge job. Even with a pretty small .config, I noticed make 
growing to >64M of RAM, so I suspect this approach may cause serious 
problems on small boxes - I didn't check that, though.

The second point is a serious one, as it affects correctness. However, the 
only case I'm aware of which is hard to handle with the recursive build is
modversions (here we go again), since changing a local file (list in 
$(export-objs)) will change a file in another subdir 
(include/linux/modules/xxx).

If I'm right that this is the only such case, it can actually be handled 
be adding a recursive pass through all Makefiles which regenerates the 
checksums. Considering that it's possible to get rid of the second pass 
for building modules, this gets us back to two passes for 
CONFIG_MODVERSIONS=y and only one pass otherwise.

However, let me state that I don't know what's the best solution here. I 
can see options like
o recursive build
o non-recursive build all handled within make
o using some external program that will generate a non-recursive Makefile
  to be used with make (note that this however, could be more or less a
  trivial parser which only adds the appropriate paths to object/source
  names in the individual Makefile fragments)

I think it would be worth figuring out what works best for the majority of 
people, though.

> If you think that you can fix all the problems listed in
> http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2
> without making significant changes to the entire kbuild system, show me
> the code.

I'm not claiming to have solved all the problems, I'm claiming to be able
to solve the important ones. - in particular, correctness, i.e.
dependencies done right and clean Makefiles. I didn't look into providing
a separate tree to put the built objects into, but I think it's doable. I 
won't even try to add shadow tree capabilities, as I don't believe that's 
something which is really needed - people probably are better off learning 
how to use a SCM anyway.

> I have no patience with people who pick the small problems out of
> kbuild and fiddle with the Makefiles without considering the entire
> problem list.  That is a classic case of ignoring the big problem and
> concentrating on the little problem that you know how to fix.

Actually, I'm not doing that. I don't think getting dependencies right is
a small problem, by the way, as you seem to (assuming you mean the big
problem is the non-recursive make). Even if I was, when I have N+1
orthogonal problems, I think solving the first N one in the usual
step-by-step approach isn't a bad idea at all, it means I can actually 
concentrate on the very problem when solving the last one.

> kbuild 2.5 fixes _all_ the problems listed in the history file, except
> for modversions which will be done later.  Once you decide to fix the
> big problems, you will realise that fiddling with the old system to fix
> the little problems is a waste of time and effort.

In one paragraph you claim that leaving the hard problem for later is a 
bad idea, in the next one you do it yourself?

--Kai

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Fri, 3 May 2002 09:48:04 -0500 (CDT), 
Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de> wrote:
>On Fri, 3 May 2002, Keith Owens wrote:
>> Changing from a recursive to non-recursive make is the big change in
>> kbuild 2.5.  If you think that you can do non-recursive make without
>> significant changes to the Makefiles, show me the code.
>
>Well, we had this thread a couple of months ago, I resurrected a 
>(proof-of-concept only) patch I had to do so. You were on the CC list - 
>why do I even bother talking to you when you don't listen anyway?

Because you have no working code!  kbuild 2.5 is up and running, you
are still discussing ideas and wasting my time.

>In the non-recursive build, make ends up with a database for all potential 
>objects, and has to build the dependency tree from that. Considering that 
>every single source file has in the order of 100 files it depends on, 
>that's a huge job. Even with a pretty small .config, I noticed make 
>growing to >64M of RAM, so I suspect this approach may cause serious 
>problems on small boxes - I didn't check that, though.

Which is why I don't let make do the dependency graph, I do it in
pp_makefile4.  Don't you understand?  I have already tried using
standard make processing for the kernel and found it was too expensive
for small systems.  Been there, done that, discarded the idea, wrote
the replacement code which is faster than what you are suggesting.

>However, let me state that I don't know what's the best solution here. I 
>can see options like
>o recursive build
>o non-recursive build all handled within make
>o using some external program that will generate a non-recursive Makefile
>  to be used with make (note that this however, could be more or less a
>  trivial parser which only adds the appropriate paths to object/source
>  names in the individual Makefile fragments)

I do.  make cannot cope with the complexity of the kernel, especially
with the two layer model of config plus timestamp.  You are still
looking at options that I investigated over a year ago and discarded as
unworkable.

>> http://prdownloads.sourceforge.net/kbuild/kbuild-2.5-history.tar.bz2
>> without making significant changes to the entire kbuild system, show me
>> the code.
>
>I'm not claiming to have solved all the problems, I'm claiming to be able
>to solve the important ones. - in particular, correctness, i.e.
>dependencies done right and clean Makefiles. I didn't look into providing
>a separate tree to put the built objects into, but I think it's doable.

Don't think, show me some working code or shut up.  I can tell you now
that it was several months of hard work to track down and fix all the
places in the source and Makefiles that assume source == object.  The
extreme difficulty of doing that with standard make rules is one of the
reasons that I went to a pre-processor.

>> kbuild 2.5 fixes _all_ the problems listed in the history file, except
>> for modversions which will be done later.  Once you decide to fix the
>> big problems, you will realise that fiddling with the old system to fix
>> the little problems is a waste of time and effort.
>
>In one paragraph you claim that leaving the hard problem for later is a 
>bad idea, in the next one you do it yourself?

No.  I am temporarily omitting a feature which is (a) currently broken
(b) is not being used in development kernels and (c) cannot be fixed
without a radical redesign.  Modversions is not needed right now and
will be added later.  Everything I have done in kbuild 2.5 is needed
now.

Kai, I am fed up with you suggesting ideas which I have already tried
and found not to work.  The only way that you will persuade me is to
come up with a *fully* working system that is simpler and faster than
kbuild 2.5.  Go away and do that, then you can argue your case.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alan Cox]

> possible that the ABI does not change but the checksum does. That happens 
> a lot, but it's not really a big problem because that (if done right) will 
> just cause spurious rebuilds - correctness isn't affected.

ccache is your friend on that one.

> Of course, for people who are patching their kernels a lot, modversions
> (again if done right) are a pain in the a**, since they cause a lot of not
> really necessary rebuilds. But people who do that supposedly think they

ccache is still your friend 8)

Alan

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Pavel Machek]

Hi!

> > possible that the ABI does not change but the checksum does. That happens 
> > a lot, but it's not really a big problem because that (if done right) will 
> > just cause spurious rebuilds - correctness isn't affected.
> 
> ccache is your friend on that one.
> 
> > Of course, for people who are patching their kernels a lot, modversions
> > (again if done right) are a pain in the a**, since they cause a lot of not
> > really necessary rebuilds. But people who do that supposedly think they
> 
> ccache is still your friend 8)

What is ccache?
									Pavel
-- 
(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Vikram]

>
> What is ccache?

maybe this?

<snip>
ccache is a compiler cache. It acts as a caching pre-processor to C/C++
compilers, using the -E compiler switch and a hash to detect when a
compilation can be satisfied from cache. This often results in a 5 to 10
times speedup in common compilations
</snip>

http://ccache.samba.org/

			Vikram


Pavel > --
> (about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
> no longer is classifiable as a democracy, but rather as a plutocracy." --hpa
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Fri, 3 May 2002 01:33:34 -0700 (PDT), 
Vikram <vvikram@stanford.edu> wrote:
><snip>
>ccache is a compiler cache. It acts as a caching pre-processor to C/C++
>compilers, using the -E compiler switch and a hash to detect when a
>compilation can be satisfied from cache. This often results in a 5 to 10
>times speedup in common compilations
></snip>
>
>http://ccache.samba.org/

Firstly kbuild 2.5 removes the need to make clean or make mrproper for
most compilations.  You need to make mrproper when changing to a new
architecture in the same directory (it is much better to use a separate
object directory for each architecture), but apart from that you should
not need to make clean or mrproper.  IMNSHO having to issue make clean
is a sign that your build system is broken, relying on human
intervention in an automated build is falt out wrong.  Automatic
detection of an arch switch is on the enhancement list for kbuild 2.5.

Secondly kbuild 2.5 keeps objects that were built but are not currently
selected, it just does not link or install them.  Build a kernel,
disable a set of drivers, build the kernel and it will just bump the
version number and relink vmlinux.  Enable the drivers again, kbuild
2.5 does not need to compile them, they are still there, it just bumps
the version number and relinks vmlinux.  Same with installing modules.
Various .tmp files list the objects and modules required for the
current .config.

So kbuild 2.5 removes the need to make clean after patches, changing
configs, etc.  It gets it right without human intervention.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Andrea Arcangeli]

On Fri, May 03, 2002 at 10:07:14PM +1000, Keith Owens wrote:
> On Fri, 3 May 2002 01:33:34 -0700 (PDT), 
> Vikram <vvikram@stanford.edu> wrote:
> ><snip>
> >ccache is a compiler cache. It acts as a caching pre-processor to C/C++
> >compilers, using the -E compiler switch and a hash to detect when a
> >compilation can be satisfied from cache. This often results in a 5 to 10
> >times speedup in common compilations
> ></snip>
> >
> >http://ccache.samba.org/
> 
> Firstly kbuild 2.5 removes the need to make clean or make mrproper for
> most compilations.  You need to make mrproper when changing to a new
> architecture in the same directory (it is much better to use a separate
> object directory for each architecture), but apart from that you should
> not need to make clean or mrproper.  IMNSHO having to issue make clean

you're right if we need a make clean it's because the buildsystem is
broken. However one thing that happens all the time to me, is that I
change an header like mm.h or sched.h and ~everything needs to be
rebuilt then. And since I cannot trust the current buildsystem I need to
`make clean` first just in case somebody is getting mm.h included
implicitly and fastdep so cannot notice it has to rebuild such object
too. But in such case make clean doesn't hurt much because almost
everything needs to be rebuilt anyways. Now the only regression I can
see is that kbuild was quite slower in compiling the kernel from scrach
(so I suspect that for me after editing mm.h it would take more time
with kbuild2.5 to reach the vmlinux generation than it took with the old
buildsystem after the make clean) Is that the case, or did you improved
the performance of kbuild recently?

Said that I look forward to avoid touching those .h so it becomes
possible to do parallel developement from two hardlinked trees.  Also
the ability of compile out of the tree is very clean feature, even if
it's a secondary need for me at least.

> is a sign that your build system is broken, relying on human
> intervention in an automated build is falt out wrong.  Automatic
> detection of an arch switch is on the enhancement list for kbuild 2.5.
> 
> Secondly kbuild 2.5 keeps objects that were built but are not currently
> selected, it just does not link or install them.  Build a kernel,
> disable a set of drivers, build the kernel and it will just bump the
> version number and relink vmlinux.  Enable the drivers again, kbuild
> 2.5 does not need to compile them, they are still there, it just bumps
> the version number and relinks vmlinux.  Same with installing modules.
> Various .tmp files list the objects and modules required for the
> current .config.
> 
> So kbuild 2.5 removes the need to make clean after patches, changing
> configs, etc.  It gets it right without human intervention.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


Andrea

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Dave Jones]

On Sat, May 18, 2002 at 03:14:10AM +0200, Andrea Arcangeli wrote:
 > you're right if we need a make clean it's because the buildsystem is
 > broken. However one thing that happens all the time to me, is that I
 > change an header like mm.h or sched.h and ~everything needs to be
 > rebuilt then.

Yep. Our includes dependancies suck bigtime.  Some work has been
done already in untangling the mess, but a lot more needs to be
done to really make a real difference.

Whats scary is that if you look at the dependancy graphs[1] of the
'best of the worst' includes, it's the same ugly mess we've
come to know and expect, and yet this is *after* some cleanups
already happened.

The 'dump everything into sched.h and friends' things really
needs splitting up some more, but it's a lot of work, and I don't
think kbuild2.5 alone is going to make that much difference
in this regard. Pulling out the component parts of the bigger
includes is probably the only way around this.

A driver that needs 'jiffies' defined should not be
inadvertantly pulling in a hundred include files.

    Dave.


[1] ftp://ftp.kernel.org/pub/linux/kernel/people/davej/misc/graphs/

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Oliver Xymoron]

On Sat, 18 May 2002, Dave Jones wrote:

> needs splitting up some more, but it's a lot of work, and I don't
> think kbuild2.5 alone is going to make that much difference
> in this regard.

No, but it does give much more incentive. With a broken makefile, you
still have to do "make clean dep all" after you've untangled the
headers, so not much is gained, with a working makefile, touching headers
doesn't mean clean build to get it right.

Keith, I would take silence from Linus to mean one of two things:
a) he's already decided he doesn't want it and he's being a dork
b) he's waiting to hear from someone who's taste he trusts before
   accepting it.

So you might try selling some of his lieutenants on your implementation
(even if doesn't make sense for them to adopt it into their trees until
Linus does).

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Gerhard Mack]

On Sat, 18 May 2002, Andrea Arcangeli wrote:

> you're right if we need a make clean it's because the buildsystem is
> broken. However one thing that happens all the time to me, is that I
> change an header like mm.h or sched.h and ~everything needs to be
> rebuilt then. And since I cannot trust the current buildsystem I need to
> `make clean` first just in case somebody is getting mm.h included
> implicitly and fastdep so cannot notice it has to rebuild such object
> too. But in such case make clean doesn't hurt much because almost
> everything needs to be rebuilt anyways. Now the only regression I can
> see is that kbuild was quite slower in compiling the kernel from scrach
> (so I suspect that for me after editing mm.h it would take more time
> with kbuild2.5 to reach the vmlinux generation than it took with the old
> buildsystem after the make clean) Is that the case, or did you improved
> the performance of kbuild recently?

recently==a month ago.


	Gerhard


--
Gerhard Mack

gmack@innerfire.net

<>< As a computer I find your faith in technology amusing.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Sat, 18 May 2002 03:14:10 +0200, 
Andrea Arcangeli <andrea@suse.de> wrote:
>
>you're right if we need a make clean it's because the buildsystem is
>broken. However one thing that happens all the time to me, is that I
>change an header like mm.h or sched.h and ~everything needs to be
>rebuilt then.

That is an orthogonal problem to kbuild 2.5.  The spaghetti that is the
include tree needs to be cleaned up, other people are working on that.

>Now the only regression I can
>see is that kbuild was quite slower in compiling the kernel from scrach
>(so I suspect that for me after editing mm.h it would take more time
>with kbuild2.5 to reach the vmlinux generation than it took with the old
>buildsystem after the make clean) Is that the case, or did you improved
>the performance of kbuild recently?

Since release 2.0 [1], kbuild 2.5 has been faster as well as more
accurate than the old build system.  A couple of people have complained
that some restricted operations are slower in kbuild 2.5 [2] but
overall it is faster, more accurate and provides more facilities,
especially for people building multiple kernels.

[1] http://www.lib.uaa.alaska.edu/linux-kernel/archive/2002-Week-13/0771.html
[2] http://marc.theaimsgroup.com/?l=linux-kernel&m=102064198628442&w=2

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Andrea Arcangeli]

On Sat, May 18, 2002 at 12:13:51PM +1000, Keith Owens wrote:
> On Sat, 18 May 2002 03:14:10 +0200, 
> Andrea Arcangeli <andrea@suse.de> wrote:
> >
> >you're right if we need a make clean it's because the buildsystem is
> >broken. However one thing that happens all the time to me, is that I
> >change an header like mm.h or sched.h and ~everything needs to be
> >rebuilt then.
> 
> That is an orthogonal problem to kbuild 2.5.  The spaghetti that is the

of course.

> include tree needs to be cleaned up, other people are working on that.
> 
> >Now the only regression I can
> >see is that kbuild was quite slower in compiling the kernel from scrach
> >(so I suspect that for me after editing mm.h it would take more time
> >with kbuild2.5 to reach the vmlinux generation than it took with the old
> >buildsystem after the make clean) Is that the case, or did you improved
> >the performance of kbuild recently?
> 
> Since release 2.0 [1], kbuild 2.5 has been faster as well as more

Ok.

> accurate than the old build system.  A couple of people have complained
> that some restricted operations are slower in kbuild 2.5 [2] but
> overall it is faster, more accurate and provides more facilities,
> especially for people building multiple kernels.
> 
> [1] http://www.lib.uaa.alaska.edu/linux-kernel/archive/2002-Week-13/0771.html
> [2] http://marc.theaimsgroup.com/?l=linux-kernel&m=102064198628442&w=2

Thanks for the two pointers.

Andrea

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Miles Bader]

Keith Owens <kaos@ocs.com.au> writes:
> Since release 2.0 [1], kbuild 2.5 has been faster as well as more
> accurate than the old build system.  A couple of people have complained
> that some restricted operations are slower in kbuild 2.5 [2] but
> overall it is faster

Yeah, but from your descriptions, it appears that the `restricted
operations' where KB 2.5 is slower are perhaps the _most common_ case
when debugging -- where you've change one or two source files and want
to rebuild the kernel to reflect that.

-Miles
-- 
"I distrust a research person who is always obviously busy on a task."
   --Robert Frosch, VP, GM Research

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[tomas szepe]

> >Another question I'd like to ask (soooorry :D) -- would there be a little
> >cunning target in Makefile-2.5 that'd create the asm-$arch symlink for me
> >in include/ like kbuild 2.4 does? Whenever I run "make -f Makefile-2.5 clean"
> >followed by "make -f Makefile-2.5 menuconfig" I get some serious whipping
> >from kbuild 2.5, cos the asm-$arch symlink gets deleted in the cleaning,
> >and I have to resurrect it by hand.
>
> It works for me.  menuconfig does not use include/asm-$ARCH.
> 
> make -f Makefile-2.5 clean
> make -f Makefile-2.5 menuconfig
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/checklist.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/checklist.c
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/menubox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/menubox.c
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/textbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/textbox.c
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/yesno.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/yesno.c
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/inputbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/inputbox.c
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/util.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/util.c
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog.c
> gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/msgbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/msgbox.c
> gcc -o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/checklist.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/menubox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/textbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/yesno.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/inputbox.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/util.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/lxdialog.o /build/kaos/2.5.12-kbuild-2.5/scripts/lxdialog/msgbox.o -lncurses
> Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
> Generating global Makefile
>   phase 1 (find all inputs)
> Using defaults found in .config
> Preparing scripts: functions, parsing......................................................................................done.
> 
> Saving your kernel configuration...

Hmmm.

kala@nibbler:/usr/src$ rm -Rf linux-2.5.12
kala@nibbler:/usr/src$ tar xzf linux-2.5.12.tgz
kala@nibbler:/usr/src$ rm -f linux; ln -s linux-2.5.12 linux
kala@nibbler:/usr/src$ cd linux-2.5.12
kala@nibbler:/usr/src/linux-2.5.12$ zcat ../kbuild-2.5-core-9.gz ../kbuild-2.5-common-2.5.12-1.gz ../kbuild-2.5-i386-2.5.12-1.gz| patch -sp1
kala@nibbler:/usr/src/linux-2.5.12$ cp ../.config .
kala@nibbler:/usr/src/linux-2.5.12$ make oldconfig
...

kala@nibbler:/usr/src/linux-2.5.12$ make -f Makefile-2.5 oldconfig
...

kala@nibbler:/usr/src/linux-2.5.12$ make -f Makefile-2.5 installable
spec value %p not found
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
...

kala@nibbler:/usr/src/linux-2.5.12$ make -f Makefile-2.5 clean
spec value %p not found
kala@nibbler:/usr/src/linux-2.5.12$ make -f Makefile-2.5 menuconfig
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/checklist.o /usr/src/linux-2.5.12/scripts/lxdialog/checklist.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/menubox.o /usr/src/linux-2.5.12/scripts/lxdialog/menubox.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/textbox.o /usr/src/linux-2.5.12/scripts/lxdialog/textbox.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/yesno.o /usr/src/linux-2.5.12/scripts/lxdialog/yesno.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/inputbox.o /usr/src/linux-2.5.12/scripts/lxdialog/inputbox.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/util.o /usr/src/linux-2.5.12/scripts/lxdialog/util.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/lxdialog.o /usr/src/linux-2.5.12/scripts/lxdialog/lxdialog.c
gcc -Wall -Wstrict-prototypes -g  -DLOCALE  -I/usr/include/ncurses -DCURSES_LOC="<ncurses.h>" -c -o /usr/src/linux-2.5.12/scripts/lxdialog/msgbox.o /usr/src/linux-2.5.12/scripts/lxdialog/msgbox.c
gcc -o /usr/src/linux-2.5.12/scripts/lxdialog/lxdialog /usr/src/linux-2.5.12/scripts/lxdialog/checklist.o /usr/src/linux-2.5.12/scripts/lxdialog/menubox.o /usr/src/linux-2.5.12/scripts/lxdialog/textbox.o /usr/src/linux-2.5.12/scripts/lxdialog/yesno.o /usr/src/linux-2.5.12/scripts/lxdialog/inputbox.o /usr/src/linux-2.5.12/scripts/lxdialog/util.o /usr/src/linux-2.5.12/scripts/lxdialog/lxdialog.o /usr/src/linux-2.5.12/scripts/lxdialog/msgbox.o -lncurses
In file included from /usr/include/bits/errno.h:25,
                 from /usr/include/errno.h:36,
                 from /usr/src/linux-2.5.12/scripts/mdbm/common.h:19,
                 from /usr/src/linux-2.5.12/scripts/mdbm/debug.c:6:
/usr/include/linux/errno.h:4: asm/errno.h: No such file or directory
In file included from /usr/include/netinet/in.h:212,
                 from /usr/src/linux-2.5.12/scripts/mdbm/mdbm.h:185,
                 from /usr/src/linux-2.5.12/scripts/mdbm/common.h:24,
                 from /usr/src/linux-2.5.12/scripts/mdbm/debug.c:6:
/usr/include/bits/socket.h:305: asm/socket.h: No such file or directory
make: *** [/usr/src/linux-2.5.12/scripts/mdbm/debug.o] Error 1

'/usr/include/asm' points to '/usr/src/linux/include/asm', which doesn't
exist at this moment. It seems to me that kbuild 2.5 makes the assumption
that the 'asm' symlink in /usr/include already determines the machine
architecture type by pointing to a concrete asm-$arch
in /usr/src/linux/include.


Tomas

-- 
"hello it's not like i read my mail so that you have where to offer to sell me
a giant turnip or anything else thankyou." -tomas szepe <kala@pinerecords.com>          

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Dave Jones]

On Thu, May 02, 2002 at 11:34:44PM +0200, tomas szepe wrote:
 > '/usr/include/asm' points to '/usr/src/linux/include/asm'

Therein lies your problem.
/usr/include/asm should NOT be a symlink.  At least, not in this century.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[John Covici]

So what should it point to?  I have had more trouble when some Debian
package made it not a symlink and if I tried to compile something
which needed correct headers for the version I am using I get very
strange errors which are hard to diagnose.

On Thu, 2 May 2002, Dave Jones wrote:

> On Thu, May 02, 2002 at 11:34:44PM +0200, tomas szepe wrote:
>  > '/usr/include/asm' points to '/usr/src/linux/include/asm'
>
> Therein lies your problem.
> /usr/include/asm should NOT be a symlink.  At least, not in this century.
>
>

-- 
         John Covici
         covici@ccs.covici.com

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Thu, 2 May 2002 21:19:53 -0400 (EDT), 
John Covici <covici@ccs.covici.com> wrote:
>So what should it point to?  I have had more trouble when some Debian
>package made it not a symlink and if I tried to compile something
>which needed correct headers for the version I am using I get very
>strange errors which are hard to diagnose.

Linus has spoken.  /usr/include/{linux,asm} must not be a symlink that
points to kernel code that is updated.  glibc must contain the linux
and asm files that were used to build glibc and those files must not
change until you change glibc.  glibc must take a copy of the kernel
headers at glibc build time or (much less desirable) it can symlink to
a set of kernel headers that are guaranteed to never change.

Having glibc linking to some random set of kernel headers is a recipe
for disaster.  kbuild 2.5 deliberately handles the asm symlink
differently from the old kbuild system, to detect and correct broken
installations.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[tomas szepe]

> >So what should it point to?  I have had more trouble when some Debian
> >package made it not a symlink and if I tried to compile something
> >which needed correct headers for the version I am using I get very
> >strange errors which are hard to diagnose.
> 
> Linus has spoken.  /usr/include/{linux,asm} must not be a symlink that
> points to kernel code that is updated.  glibc must contain the linux
> and asm files that were used to build glibc and those files must not
> change until you change glibc.  glibc must take a copy of the kernel
> headers at glibc build time or (much less desirable) it can symlink to
> a set of kernel headers that are guaranteed to never change.
> 
> Having glibc linking to some random set of kernel headers is a recipe
> for disaster.  kbuild 2.5 deliberately handles the asm symlink
> differently from the old kbuild system, to detect and correct broken
> installations.

Fair enough. I suggest, though, that you put a similar explanation
into kbuild 2.5 documentation.

T.

-- 
"hello it's not like i read my mail so that you have where to offer to sell me
a giant turnip or anything else thankyou." -tomas szepe <kala@pinerecords.com>          

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alexander Viro]

On Thu, 2 May 2002, John Covici wrote:

> So what should it point to?  I have had more trouble when some Debian
> package made it not a symlink and if I tried to compile something

"some package" being libc6-dev.  I.e. the first thing that puts something
in /usr/include...

> which needed correct headers for the version I am using I get very
> strange errors which are hard to diagnose.

Fix your application.  The rules are very simple - /usr/include/linux contains
versions of headers used to build libc.  If you are linking against libc,
you don't want to have different parts of resulting executable to be
compiled with different versions of these headers.  If you want several
definitions from headers of your current kernel - extract them (and make
damn sure that you don't pull a conflict with libc headers).

IOW, create a private header with definitions you need.  And you'd better
make sure that stuff you are pulling is stable, obviously - if it changes
from version to version you are going to run into serious trouble at
runtime.  "Rebuild whenever you boot into new kernel" is not a good idea...

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Davide Libenzi]

On Thu, 2 May 2002, Alexander Viro wrote:

> Fix your application.  The rules are very simple - /usr/include/linux contains
> versions of headers used to build libc.  If you are linking against libc,
> you don't want to have different parts of resulting executable to be
> compiled with different versions of these headers.  If you want several
> definitions from headers of your current kernel - extract them (and make
> damn sure that you don't pull a conflict with libc headers).

i do not know how glibc uses to export things but they should not export
any kernel related structure. the theory that went around in various
kernel update howto was to make linux, asm and scsi links to kernel
include dirs. even if glibc would export kernel dependent stuff ( ie
ioctl() params ) it should let the pointer go through w/out any
manipulation, otherwise it would be linked to a specific kernel version.

# find /usr/include/ -name '*.h' -exec grep -H 'include <linux/' \{} \;

/usr/include/pci/pci.h:#include <linux/pci.h>
/usr/include/pci/pci.h:#include <linux/types.h>
/usr/include/apm.h:#include <linux/apm_bios.h>
/usr/include/a.out.h:# include <linux/a.out.h>
/usr/include/bits/errno.h:# include <linux/errno.h>
/usr/include/bits/local_lim.h:#include <linux/limits.h>
/usr/include/net/ethernet.h:#include <linux/if_ether.h>
/usr/include/net/if_slip.h:#include <linux/if_slip.h>
/usr/include/net/ppp-comp.h:#include <linux/ppp-comp.h>
/usr/include/net/ppp_defs.h:#include <linux/ppp_defs.h>
/usr/include/netatalk/at.h:#include <linux/atalk.h>
/usr/include/netinet/if_ether.h:#include <linux/if_ether.h>
/usr/include/netinet/if_fddi.h:#include <linux/if_fddi.h>
/usr/include/netinet/if_tr.h:#include <linux/if_tr.h>
/usr/include/netinet/igmp.h:#include <linux/igmp.h>
/usr/include/nfs/nfs.h:#include <linux/nfs.h>
/usr/include/sys/kd.h:#include <linux/kd.h>
/usr/include/sys/param.h:#include <linux/limits.h>
/usr/include/sys/param.h:#include <linux/param.h>
/usr/include/sys/pci.h:#include <linux/pci.h>
/usr/include/sys/prctl.h:#include <linux/prctl.h>
/usr/include/sys/soundcard.h:#include <linux/soundcard.h>
/usr/include/sys/sysctl.h:#include <linux/sysctl.h>
/usr/include/sys/sysinfo.h:#include <linux/kernel.h>
/usr/include/sys/ultrasound.h:#include <linux/ultrasound.h>
/usr/include/sys/vt.h:#include <linux/vt.h>
/usr/include/libnet.h:#include <linux/igmp.h>


i've always used the symlink approach and if you're right i should
consider spending some time in Las Vegas ;)




- Davide

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alexander Viro]

On Thu, 2 May 2002, tomas szepe wrote:
 
> '/usr/include/asm' points to '/usr/src/linux/include/asm', which doesn't
> exist at this moment. It seems to me that kbuild 2.5 makes the assumption
> that the 'asm' symlink in /usr/include already determines the machine
> architecture type by pointing to a concrete asm-$arch
> in /usr/src/linux/include.

Sigh...  Configurations with /usr/include/{linux,asm} being symlinks
are BROKEN.  Please, look through the archives - it had been discussed
a lot of times.  Userland has no business using kernel headers directly
and that's precisely what had bitten you - setup where /usr/include/asm
comes not from libc but from the (currently being built) kernel.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[tomas szepe]

> > '/usr/include/asm' points to '/usr/src/linux/include/asm', which doesn't
> > exist at this moment. It seems to me that kbuild 2.5 makes the assumption
> > that the 'asm' symlink in /usr/include already determines the machine
> > architecture type by pointing to a concrete asm-$arch
> > in /usr/src/linux/include.
> Sigh...  Configurations with /usr/include/{linux,asm} being symlinks
> are BROKEN.  Please, look through the archives - it had been discussed
> a lot of times.  Userland has no business using kernel headers directly
> and that's precisely what had bitten you - setup where /usr/include/asm
> comes not from libc but from the (currently being built) kernel.

My apologies then... Actually, this is how Slackware-8.0 came (and
slackware-current AFAIK still comes). Apparently I must've missed
the transition, and so has Patrick Volkerding.

Also I'm sorry for bringing up the MODVERSIONS issue. If I had known
what flamewar it would trigger, I'd never have raised the topic. *sigh*

Now let's see what's to be found in glibc-2.2.5.tar.gz. :)

-Tomas

-- 
"hello it's not like i read my mail so that you have where to offer to sell me
a giant turnip or anything else thankyou." -tomas szepe <kala@pinerecords.com>          

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Mark H. Wood]

On Thu, 2 May 2002, Alexander Viro wrote:
[quote snipped]
> Sigh...  Configurations with /usr/include/{linux,asm} being symlinks
> are BROKEN.  Please, look through the archives - it had been discussed
> a lot of times.  Userland has no business using kernel headers directly
> and that's precisely what had bitten you - setup where /usr/include/asm
> comes not from libc but from the (currently being built) kernel.

There is a reason that this issue keesp rising from the grave.  I just
downloaded the glibc 2.2.5 source tarball and in INSTALL I find
this:

[begin quote]
Specific advice for Linux systems
=================================

   If you are installing GNU libc on a Linux system, you need to have
the header files from a 2.2 kernel around for reference.  You do not
need to use the 2.2 kernel, just have its headers where glibc can access
at them.  The easiest way to do this is to unpack it in a directory
such as `/usr/src/linux-2.2.1'.  In that directory, run `make config'
and accept all the defaults.  Then run `make include/linux/version.h'.
Finally, configure glibc with the option
`--with-headers=/usr/src/linux-2.2.1/include'.  Use the most recent
kernel you can get your hands on.

   An alternate tactic is to unpack the 2.2 kernel and run `make
config' as above.  Then rename or delete `/usr/include', create a new
`/usr/include', and make the usual symbolic links of
`/usr/include/linux' and `/usr/include/asm' into the 2.2 kernel
sources.  You can then configure glibc with no special options.  This
tactic is recommended if you are upgrading from libc5, since you need
to get rid of the old header files anyway.

   Note that `/usr/include/net' and `/usr/include/scsi' should *not* be
symlinks into the kernel sources.  GNU libc provides its own versions
of these files.
[end quote]

Note the bit about "usual symbolic links...into the...kernel sources".

-- 
Mark H. Wood, Lead System Programmer   mwood@IUPUI.Edu
MS Windows *is* user-friendly, but only for certain values of "user".

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Kurt Wall]

Scribbling feverishly on May 03, Mark H. Wood managed to emit:
> On Thu, 2 May 2002, Alexander Viro wrote:
> [quote snipped]
> > Sigh...  Configurations with /usr/include/{linux,asm} being symlinks
> > are BROKEN.  Please, look through the archives - it had been discussed
> > a lot of times.  Userland has no business using kernel headers directly
> > and that's precisely what had bitten you - setup where /usr/include/asm
> > comes not from libc but from the (currently being built) kernel.
> 
> There is a reason that this issue keesp rising from the grave.  I just
> downloaded the glibc 2.2.5 source tarball and in INSTALL I find
> this:

Indeed.

> [begin quote]
> Specific advice for Linux systems
> =================================
> 
>    If you are installing GNU libc on a Linux system, you need to have
> the header files from a 2.2 kernel around for reference.  You do not
> need to use the 2.2 kernel, just have its headers where glibc can access
> at them.  The easiest way to do this is to unpack it in a directory
> such as `/usr/src/linux-2.2.1'.  In that directory, run `make config'
> and accept all the defaults.  Then run `make include/linux/version.h'.
> Finally, configure glibc with the option
> `--with-headers=/usr/src/linux-2.2.1/include'.  Use the most recent
> kernel you can get your hands on.

I've had no trouble (or, no *known* trouble) building glibc against
the current (2.4.18) kernel headers. So, are references to the 2.2 kernel
in glibc's INSTALL document out of date in this respect? "Use the most
recent kernel you can get your hands on." suggests this is the case.

>    An alternate tactic is to unpack the 2.2 kernel and run `make
> config' as above.  Then rename or delete `/usr/include', create a new
> `/usr/include', and make the usual symbolic links of
> `/usr/include/linux' and `/usr/include/asm' into the 2.2 kernel
> sources.  You can then configure glibc with no special options.  This
> tactic is recommended if you are upgrading from libc5, since you need
> to get rid of the old header files anyway.
> 
>    Note that `/usr/include/net' and `/usr/include/scsi' should *not* be
> symlinks into the kernel sources.  GNU libc provides its own versions
> of these files.
> [end quote]
> 
> Note the bit about "usual symbolic links...into the...kernel sources".

What, then, is the best way to proceed? Build and install glibc,
copy $KERNELSRC/include/asm to /usr/include/asm and
$KERNELSRC/include/linux to /usr/include/linux?

Kurt
-- 
Happiness, n.:
	An agreeable sensation arising from contemplating the misery of
another.
		-- Ambrose Bierce, "The Devil's Dictionary"

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Mike Fedyk]

On Fri, May 03, 2002 at 04:05:52PM -0500, Mark H. Wood wrote:
> There is a reason that this issue keesp rising from the grave.  I just
> downloaded the glibc 2.2.5 source tarball and in INSTALL I find
> this:
> 
> [begin quote]
> Specific advice for Linux systems
> =================================
> 
>    If you are installing GNU libc on a Linux system, you need to have
> the header files from a 2.2 kernel around for reference.  You do not
> need to use the 2.2 kernel, just have its headers where glibc can access
> at them.  The easiest way to do this is to unpack it in a directory
> such as `/usr/src/linux-2.2.1'.  In that directory, run `make config'
> and accept all the defaults.  Then run `make include/linux/version.h'.
> Finally, configure glibc with the option
> `--with-headers=/usr/src/linux-2.2.1/include'.  Use the most recent
> kernel you can get your hands on.
> 
>    An alternate tactic is to unpack the 2.2 kernel and run `make
> config' as above.  Then rename or delete `/usr/include', create a new
> `/usr/include', and make the usual symbolic links of
> `/usr/include/linux' and `/usr/include/asm' into the 2.2 kernel
> sources.  You can then configure glibc with no special options.  This
> tactic is recommended if you are upgrading from libc5, since you need
> to get rid of the old header files anyway.
> 
>    Note that `/usr/include/net' and `/usr/include/scsi' should *not* be
> symlinks into the kernel sources.  GNU libc provides its own versions
> of these files.
> [end quote]

I believe this is only for building Glibc, but all apps that depend on Glibc
should use whatever kernel headers that glibc is using...

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Pavel Machek]

Hi!

> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
> It is faster, better documented, easier to write build rules in, has
> better install facilities, allows separate source and object trees, can
> do concurrent builds from the same source tree and is significantly
> more accurate than the existing kernel build system.

Significantly more accurate, or actually *acurate* as in "never
forgets to rebuild anything"?
									Pavel
PS: Okay, modulo bugs...

-- 
(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Fri, 3 May 2002 00:54:27 +0200, 
Pavel Machek <pavel@ucw.cz> wrote:
>kaos wrote
>> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
>> It is faster, better documented, easier to write build rules in, has
>> better install facilities, allows separate source and object trees, can
>> do concurrent builds from the same source tree and is significantly
>> more accurate than the existing kernel build system.
>
>Significantly more accurate, or actually *acurate* as in "never
>forgets to rebuild anything"?
>									Pavel
>PS: Okay, modulo bugs...

Never forgets to rebuild anything, modulo bugs.  I would like to claim
100% build accuracy but ... "All code has at least one bug".

If you make a change and the kernel does not rebuild correctly, that is
a severity 1 bug in kbuild 2.5 and I will fix it.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Randy.Dunlap]

Hi Keith-

On Thu, 2 May 2002, Keith Owens wrote:

[snipped]

| Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
| It is faster, better documented, easier to write build rules in, has
| better install facilities, allows separate source and object trees, can
| do concurrent builds from the same source tree and is significantly
| more accurate than the existing kernel build system.

I kinda like to do 'make bzImage' without making modules also.
Would that be difficult to do in kbuild 2.5?
Oh, but then I would also (still) need 'make modules'...

| Before I send you the kbuild 2.5 patch, how do you want to handle it?
|
| * Coexist with the existing kernel build for one or two releases or
|   delete the old build system when kbuild 2.5 goes in?
|
|   Coexistence for a few days gives a backout, just in case.  It also
|   gives a kernel release where the old and new code can be compared,
|   useful for architectures that have not been converted yet.

So is there a downside to the coexisting method?
If not, let's do it.  (One reason: see below.)

| I would like kbuild 2.5 to go in in the near future.  Keeping up to
| date with kernel changes is a significant effort, Makefiles change all
| the time, especially when major subsystems like sound and usb are
| reorganised.  There are also some changes to architecture code to do it
| right under kbuild 2.5 and tracking those against kernel changes can be
| painful.

For sure.

Any ideas about this error?  user error??

$ make oldconfig menuconfig

... and then

[rddunlap@midway linux-2513-pv]$ make -f Makefile-2.5
spec value %p not found
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc
-E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
  phase 1 (find all inputs)
Error: The CML input files have changed since .config was created.
       Always make one of xconfig menuconfig oldconfig defconfig
config randconfig allyes allno allmod after changing CML files
make: *** [/usr/linsrc/linux-2513-pv/.config] Error 1
[rddunlap@midway linux-2513-pv]$

I removed all .tmp* files & dir., reran 'make oldconfig menuconfig',
and got the same results.

-- 
~Randy

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Thu, 2 May 2002 21:17:43 -0700 (PDT), 
"Randy.Dunlap" <rddunlap@osdl.org> wrote:
>I kinda like to do 'make bzImage' without making modules also.
>Would that be difficult to do in kbuild 2.5?
>Oh, but then I would also (still) need 'make modules'...

Sample testing targets, to see if you made any typing errors.

  make vmlinux
  make arch/i386/boot/bzImage
  make drivers/acpi (non-recursive)
  make drivers/acpi-r (recursive)

Do it with NO_MAKEFILE_GEN=1 for much, much! faster builds.  But you
should really do a clean make installable (which will do modules as
well) before make install.

>Any ideas about this error?  user error??
>
>$ make oldconfig menuconfig
>
>... and then
>
>[rddunlap@midway linux-2513-pv]$ make -f Makefile-2.5
>spec value %p not found
>Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc
>-E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
>Generating global Makefile
>  phase 1 (find all inputs)
>Error: The CML input files have changed since .config was created.
>       Always make one of xconfig menuconfig oldconfig defconfig
>config randconfig allyes allno allmod after changing CML files
>make: *** [/usr/linsrc/linux-2513-pv/.config] Error 1

You mixed the old kbuild 2.4 make *config with a kbuild 2.5 build.
Don't do that.

One of the downsides of coexistence, users can get it wrong.

make -f Makefile-2.5 menuconfig installable

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Randy.Dunlap]

On Fri, 3 May 2002, Keith Owens wrote:

| On Thu, 2 May 2002 21:17:43 -0700 (PDT),
| "Randy.Dunlap" <rddunlap@osdl.org> wrote:
| >I kinda like to do 'make bzImage' without making modules also.
| >Would that be difficult to do in kbuild 2.5?
| >Oh, but then I would also (still) need 'make modules'...
|
| Sample testing targets, to see if you made any typing errors.
|
|   make vmlinux
|   make arch/i386/boot/bzImage
|   make drivers/acpi (non-recursive)
|   make drivers/acpi-r (recursive)

      make -f Makefile-2.5 <target>

      Yes, that works fine.

      So if my .config file has lots of USB modules =m,
      I can just make drivers/usb-r and it will make all
      of the specified modules ?
      Yes, I did that.

| Do it with NO_MAKEFILE_GEN=1 for much, much! faster builds.  But you
| should really do a clean make installable (which will do modules as
| well) before make install.
|
| >Any ideas about this error?  user error??
| >
| >$ make oldconfig menuconfig
| >
| >... and then
| >
...
| >make: *** [/usr/linsrc/linux-2513-pv/.config] Error 1
|
| You mixed the old kbuild 2.4 make *config with a kbuild 2.5 build.
| Don't do that.

DDT.  Got it.  Thanks.

| One of the downsides of coexistence, users can get it wrong.

Right.

| make -f Makefile-2.5 menuconfig installable

Working now.  I appreciate your help and kbuild 2.5.

-- 
~Randy

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Gerd Knorr]

>  Do it with NO_MAKEFILE_GEN=1 for much, much! faster builds.

What exactly is the reason for this hack, i.e. why kbuild wants to
rebuild the Makefiles every time?  Isn't it enougth to do that only
if .config has been touched?

  Gerd

-- 
You can't please everybody.  And usually if you _try_ to please
everybody, the end result is one big mess.
				-- Linus Torvalds, 2002-04-20

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On 3 May 2002 10:06:04 GMT, 
Gerd Knorr <kraxel@bytesex.org> wrote:
>>  Do it with NO_MAKEFILE_GEN=1 for much, much! faster builds.
>
>What exactly is the reason for this hack, i.e. why kbuild wants to
>rebuild the Makefiles every time?  Isn't it enougth to do that only
>if .config has been touched?

Or any of the Makefile.in files have changed.  Or any of the command
line options have changed.  Or various environment variables have
changed.  Or a target file has been altered outside kbuild control.
Or the compiler has changed.  Or ... the list goes on.

Coding a special case to work out if the existing global makefile can
be reused is horribly error prone.  And it would take just as long as
rebuilding the global makefile from scratch.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Gerd Knorr]

Keith Owens wrote:
>  On 3 May 2002 10:06:04 GMT, 
>  Gerd Knorr <kraxel@bytesex.org> wrote:
> >>  Do it with NO_MAKEFILE_GEN=1 for much, much! faster builds.
> >
> >What exactly is the reason for this hack, i.e. why kbuild wants to
> >rebuild the Makefiles every time?  Isn't it enougth to do that only
> >if .config has been touched?
>  
>  Or any of the Makefile.in files have changed.

.config + all Makefile.in's is still a small number of files where
make has to check the timestamp to see whenever a rebuild of the global
makefile is needed.

>  Or any of the command line options have changed.

Which command line options?

>  Or various environment variables have changed.
>  Or the compiler has changed.

Which environment variables?  CC / CFLAGS?  Well, with other CFLAGS
and/or compiler you have to do a full rebuild anyway, thus using "make
clean" would work equally well ...

>  Or a target file has been altered outside kbuild control.

Which is the users fault IMO.  I don't see the point in trying to catch
this and make kbuild idiot proof.  I doubt it is possible to make kbuild
clever enougth to handle all evil things a user could do.  AI isn't that
good.

>  Coding a special case to work out if the existing global makefile can
>  be reused is horribly error prone.

Special case?  I'd say it is the common case when doing kernel
development.  At least I don't use another compiler for every second
make.  I usually hack some piece of code and recompile the module then.

  Gerd

-- 
You can't please everybody.  And usually if you _try_ to please
everybody, the end result is one big mess.
				-- Linus Torvalds, 2002-04-20

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On 3 May 2002 12:05:01 GMT, 
Gerd Knorr <kraxel@bytesex.org> wrote:
>Keith Owens wrote:
>>  Coding a special case to work out if the existing global makefile can
>>  be reused is horribly error prone.
>
>Special case?  I'd say it is the common case when doing kernel
>development.  At least I don't use another compiler for every second
>make.  I usually hack some piece of code and recompile the module then.

We learnt the hard way in kbuild 2.4 that trying to optimize the build
by checking for special cases wasted far more time in solving wierd
problems than it ever saved, every time the optimization was wrong we
generated invalid kernels and modules.  Experience showed that
over-optimization was a BIG mistake.

The pre-processing programs do more than just build the global
makefile, they do a ton of integrity checking as well.  If you bypass
the makefile generation then I cannot guarantee the results.  Most of
the time is taken in finding all the files and doing the integrity
checks, the actual generation step is pretty fast.

You are complaining about a system that is already far more accurate
than the old build system, has more features and it still manages to be
faster than kbuild 2.4.  I am not going to sacrifice build accuracy for
the sake of shaving a few more seconds off the build time, it is
already faster than the old system.  Especially when I have to add and
maintain extra code in order to make the build less reliable!


I suspect that some users are put off by the small amount of output
from kbuild 2.5, they are used to all the noise from kbuild 2.4 and
they equate noise with "something is being done".  Perhaps I should add
CONFIG_PROVIDE_RANDOM_NOISE_TO_KEEP_THE_USER_AMUSED to kbuild 2.5, with
sub-options for dots, hashes, twirling bars and

              \|/ ____ \|/
              "@'/ ,. \`@"
              /_| \__/ |_\
                 \__U_/			(R) davem


Users who "know" that nothing has changed except a source file can tell
kbuild 2.5 of their opinion with NO_MAKEFILE_GEN=1.  Just don't blame
me if you get it wrong.  I provide the gun, but you have to aim it at
your own foot and pull the trigger.  Like everything else on kbuild
2.5, NO_MAKEFILE_GEN=1 is more accurate than the 2.4 equivalent
(SUBDIRS= is broken) and faster as well.  If you cannot type 18 extra
characters on the command line, you have other problems.

For the really foolish people who "know" that the build is complete and
who want to bypass all the integrity checks that are performed during
makefile generation -

  make NO_MAKEFILE_GEN=1 \
  	I_KNOW_THAT_THIS_BUILD_MAY_BE_INCOMPLETE_BUT_I_WANT_TO_INSTALL_IT_ANYWAY=1 \
	install

When it blows up on you, you get to keep the pieces.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Paul Mackerras]

Keith Owens writes:

> Coding a special case to work out if the existing global makefile can
> be reused is horribly error prone.  And it would take just as long as
> rebuilding the global makefile from scratch.

I seriously doubt that last statement.  Building the global makefile
takes about 20 seconds on the box I compile on.  On a kernel tree
without object files I can read all the files in the kernel tree in
about 0.8 seconds, and I can calculate an md5sum of every file in 3.2
seconds.  I can do an md5sum of all the Makefile.in's in 0.1 seconds.
This is with pp_makefile* compiled with -O2 -DNDEBUG=1.

Paul.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Paul Mackerras]

I wrote:

> I seriously doubt that last statement.  Building the global makefile
> takes about 20 seconds on the box I compile on.  On a kernel tree
> without object files I can read all the files in the kernel tree in
> about 0.8 seconds, and I can calculate an md5sum of every file in 3.2
> seconds.  I can do an md5sum of all the Makefile.in's in 0.1 seconds.
> This is with pp_makefile* compiled with -O2 -DNDEBUG=1.

I made a mistake, the time to build the global makefile is in fact
around 12 seconds with -O2 -DNDEBUG=1.  And I should point out that
this is a machine with enough RAM to keep the whole kernel tree in
memory (so disk bandwidth is not an issue).

I still think that the time to build the global makefile is big enough
and obvious enough that many people (including myself) will want to
see it optimized, either by making the process itself more efficient
or by caching the result and re-using it where possible.

Paul.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Mike Fedyk]

On Sat, May 04, 2002 at 06:03:54PM +1000, Paul Mackerras wrote:
> I still think that the time to build the global makefile is big enough
> and obvious enough that many people (including myself) will want to
> see it optimized, either by making the process itself more efficient
> or by caching the result and re-using it where possible.

But it's still faster than kbuild-2.4.

This shouldn't keep it from going into mainline.  Just like anything else,
the itch will be scratched if there's enough irritation, and inclusion of
the new kbuild should at first *reduce* the irritation that's already there
now.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Paul Mackerras]

Mike Fedyk writes:

> On Sat, May 04, 2002 at 06:03:54PM +1000, Paul Mackerras wrote:
> > I still think that the time to build the global makefile is big enough
> > and obvious enough that many people (including myself) will want to
> > see it optimized, either by making the process itself more efficient
> > or by caching the result and re-using it where possible.
> 
> But it's still faster than kbuild-2.4.
> 
> This shouldn't keep it from going into mainline.  Just like anything else,
> the itch will be scratched if there's enough irritation, and inclusion of
> the new kbuild should at first *reduce* the irritation that's already there
> now.

Oh, I agree totally.  I'm just saying that I think there will be
enough irritation.  And this is a new irritation, which is for that
reason more noticeable than the old irritations which we are used to,
even if the old irritations are actually worse. :)

Paul.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Sat, 4 May 2002 16:44:08 +1000 (EST), 
Paul Mackerras <paulus@samba.org> wrote:
>Keith Owens writes:
>> be reused is horribly error prone.  And it would take just as long as
>> rebuilding the global makefile from scratch.
>
>I seriously doubt that last statement.  Building the global makefile
>takes about 20 seconds on the box I compile on.  On a kernel tree
>without object files I can read all the files in the kernel tree in
>about 0.8 seconds, and I can calculate an md5sum of every file in 3.2
>seconds.  I can do an md5sum of all the Makefile.in's in 0.1 seconds.
>This is with pp_makefile* compiled with -O2 -DNDEBUG=1.

Those times do not look right, the build times look too long.  On a
Pentium III 700MHz with 1GiB ram, I get

# cd $KBUILD_SRCTREE_000
# time md5sum -c ../sums > /dev/null
7.10user 0.73system 0:07.82elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
# time make -f $KBUILD_SRCTREE_000/Makefile-2.5 phase4
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/kgcc' CPP='/usr/bin/kgcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
  phase 1 (find all inputs)
4.31user 0.15system 0:04.45elapsed 100%CPU (0text+0data 0max)k
  phase 2 (convert all Makefile.in files)
1.66user 0.03system 0:01.69elapsed 99%CPU (0text+0data 0max)k
  phase 3 (evaluate selections)
1.10user 0.91system 0:01.91elapsed 104%CPU (0text+0data 0max)k
  phase 4 (integrity checks, write global makefile)
10.02user 0.08system 0:10.10elapsed 99%CPU (0text+0data 0max)k
17.40user 1.40system 0:18.59elapsed 101%CPU (0avgtext+0avgdata 0maxresident)k

Doing all the setup work on this machine takes ~2.5 times as long as
md5sum, but it does more work than just md5sum.  phase4 also does all
the integrity checks, cleans up dead files, checks for timestamp skew,
runs the config dependency chains etc.

What would it require to optimize for the "no config/makefile change"?

md5sums alone are not enough, people touch source or header files, even
config options and expect objects to be rebuilt, timestamps are
required as well.  A change to the KBUILD_SRCTREE_nnn environment
variables adds or deletes entire trees.  So phase1 is still required,
to find all the files in all the trees and get their current
timestamps.  "Optimizing" will not save any time there, kbuild always
needs current timestamp data.

That gives 7.8 seconds to check the md5sums compared to 14.2 seconds to

* convert the Makefile.in files (using the latest values for the kbuild
  variables from the environment and the command line)
* evaluate the selections (which can be overridden on the command line)
* run the config dependency chains
* do all the integrity checks
* handle special cases like asking for a .i or .s file on the command
  line
* write the global makefile.

Not a huge difference, especially considering it is doing more than a
simple checksum.

It is the config dependency, integrity checks and special case
processing that take the bulk of phase4, writing the makefile is a
small percentage.  "Optimizing" could only save a small amount of time
and would require extra code and time to work out if I could save any
time.

When I build the pp_ programs with -O2 -NDEBUG=1, the times go down to

  phase 1 (find all inputs)
2.78user 0.11system 0:02.88elapsed 100%CPU (0text+0data 0max)k
  phase 2 (convert all Makefile.in files)
1.09user 0.02system 0:01.10elapsed 100%CPU (0text+0data 0max)k
  phase 3 (evaluate selections)
1.04user 0.97system 0:01.89elapsed 106%CPU (0text+0data 0max)k
  phase 4 (integrity checks, write global makefile)
6.10user 0.10system 0:06.19elapsed 100%CPU (0text+0data 0max)k
11.35user 1.38system 0:12.53elapsed 101%CPU (0avgtext+0avgdata 0maxresident)k

Not bad for something that is doing a lot more work than a simple
checksum, 1.6 times as long as md5sum for complete kbuild support.  As
it stands kbuild 2.5 provides additional features, is far more accurate
and is 30% faster than kbuild 2.4.  I even provide an option for
bypassing everything and going straight to the build step, that option
is also faster and more accurate than the kbuild 2.4 equivalent.

If all of that is not enough justification for replacing the old
system, then shaving a few seconds off the startup code is not going to
make any difference.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Russell King]

On Sat, May 04, 2002 at 07:03:02PM +1000, Keith Owens wrote:
> md5sums alone are not enough, people touch source or header files, even
> config options and expect objects to be rebuilt, timestamps are
> required as well.  A change to the KBUILD_SRCTREE_nnn environment
> variables adds or deletes entire trees.  So phase1 is still required,
> to find all the files in all the trees and get their current
> timestamps.  "Optimizing" will not save any time there, kbuild always
> needs current timestamp data.

So you're *reading* all source files in the kernel tree each time you
kick off a build?  Do you have shares in a SDRAM memory manufacturer
by any chance?

If that's the case, I'd rather the existing 2.4 build system stayed.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Paul Mackerras]

Keith Owens writes:

> Those times do not look right, the build times look too long.  On a
> Pentium III 700MHz with 1GiB ram, I get

I made a mistake, the 20 seconds was without -O2 -DNDEBUG=1, with
those options it was 12 seconds (dual 1GHz G4 powermac with 1GB of
RAM).

> md5sums alone are not enough, people touch source or header files, even

Surely the dependencies on the dates of source and header files are
handled by make itself?  The global makefile wouldn't change just
because I touch a source file would it?

> config options and expect objects to be rebuilt, timestamps are
> required as well.  A change to the KBUILD_SRCTREE_nnn environment
> variables adds or deletes entire trees.  So phase1 is still required,
> to find all the files in all the trees and get their current

Finding all the files in a tree and stating them doesn't take very
long:

bash-2.05a$ touch foo
bash-2.05a$ time find . -newer foo

real	0m0.100s
user	0m0.020s
sys	0m0.090s

So that is not why phase1 takes a couple of seconds.

> That gives 7.8 seconds to check the md5sums compared to 14.2 seconds to
> 
> * convert the Makefile.in files (using the latest values for the kbuild
>   variables from the environment and the command line)
> * evaluate the selections (which can be overridden on the command line)
> * run the config dependency chains
> * do all the integrity checks
> * handle special cases like asking for a .i or .s file on the command
>   line
> * write the global makefile.
> 
> Not a huge difference, especially considering it is doing more than a
> simple checksum.

But when have you known a kernel hacker to be satisfied with just
"faster than the previous system", as distinct from "as fast as I can
reasonably make it go"? ;-)

> It is the config dependency, integrity checks and special case
> processing that take the bulk of phase4, writing the makefile is a
> small percentage.  "Optimizing" could only save a small amount of time
> and would require extra code and time to work out if I could save any
> time.

Actually, from the profiles I have done it looks to me like it is
spending the bulk of the time inside mdbm.  So presumably what is
taking up most of the time is fetching and storing the persistent data
needed for the processing, not the actual processing itself.

> Not bad for something that is doing a lot more work than a simple
> checksum, 1.6 times as long as md5sum for complete kbuild support.  As
> it stands kbuild 2.5 provides additional features, is far more accurate
> and is 30% faster than kbuild 2.4.  I even provide an option for
> bypassing everything and going straight to the build step, that option
> is also faster and more accurate than the kbuild 2.4 equivalent.
> 
> If all of that is not enough justification for replacing the old
> system, then shaving a few seconds off the startup code is not going to
> make any difference.

Don't get me wrong, I think it's great to have all the advantages that
kbuild-2.5 brings.  However, I also think that those seconds spent in
the startup code will tend to have a disproportionate effect on
people's perceptions of the new system.  I know you have already spent
a lot of effort on this, but I want to get in and have a look myself
to see if I can spot anything that could be improved there.

Paul.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Sat, 4 May 2002 20:33:53 +1000 (EST), 
Paul Mackerras <paulus@samba.org> wrote:
>Keith Owens writes:
>> md5sums alone are not enough, people touch source or header files, even
>
>Surely the dependencies on the dates of source and header files are
>handled by make itself?  The global makefile wouldn't change just
>because I touch a source file would it?

phase1 gathers timestamps to use for _all_ kbuild processing, not just
generating the global makefile.  NFS timestamp skew between edit and
build systems can make timestamps go backwards.  Edit a file, decide
you made a mistake, restore from a backup or a repository and the
timestamp goes backwards.

make does not detect timestamps going backwards, it is a common source
of build error, especially over NFS.  kbuild 2.5 detects _all_
timestamp changes, forwards and backwards, using the data gathered in
phase1.  phase1 must always run, you cannot optimize it away.

>Finding all the files in a tree and stating them doesn't take very
>long:
>So that is not why phase1 takes a couple of seconds.

It is the database processing, see scripts/pp_makefile1.c.  Besides
gathering timestamps, there are 6 passes over the kbuild database in
phase1 to get the data ready for the rest of kbuild.

The comments on each phase are only summaries.  Every phase does more
than you think, please look at the code before assuming that you know
everything about the kbuild processing.

>But when have you known a kernel hacker to be satisfied with just
>"faster than the previous system", as distinct from "as fast as I can
>reasonably make it go"? ;-)

make NO_MAKEFILE_GEN=1.  Bypass everything and go straight to the
build.  No integrity checks of course.

>> It is the config dependency, integrity checks and special case
>> processing that take the bulk of phase4, writing the makefile is a
>> small percentage.  "Optimizing" could only save a small amount of time
>> and would require extra code and time to work out if I could save any
>> time.
>
>Actually, from the profiles I have done it looks to me like it is
>spending the bulk of the time inside mdbm.  So presumably what is
>taking up most of the time is fetching and storing the persistent data
>needed for the processing, not the actual processing itself.

I have not done any tuning on mdbm.  The source came from Larry McVoy
and I do not want to change it.

>> kbuild 2.5 provides additional features, is far more accurate
>> and is 30% faster than kbuild 2.4.  I even provide an option for
>> bypassing everything and going straight to the build step, that option
>> is also faster and more accurate than the kbuild 2.4 equivalent.
>> 
>> If all of that is not enough justification for replacing the old
>> system, then shaving a few seconds off the startup code is not going to
>> make any difference.
>
>Don't get me wrong, I think it's great to have all the advantages that
>kbuild-2.5 brings.  However, I also think that those seconds spent in
>the startup code will tend to have a disproportionate effect on
>people's perceptions of the new system.  I know you have already spent
>a lot of effort on this, but I want to get in and have a look myself
>to see if I can spot anything that could be improved there.

Code can always be improved.  At the moment kbuild 2.5 is stable and
ready to go into the kernel.  This is not the time to redo the
pre-processing programs or to try tuning the database code.  I am only
tracking kernel changes and doing bug fixes to kbuild 2.5 until it has
been accepted into the kernel.  Once it is in the kernel I have a list
of enhancements to be done, one change at a time.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Gerd Knorr]

> >Surely the dependencies on the dates of source and header files are
> >handled by make itself?  The global makefile wouldn't change just
> >because I touch a source file would it?
>  
>  phase1 gathers timestamps to use for _all_ kbuild processing, not just
>  generating the global makefile.  NFS timestamp skew between edit and
>  build systems can make timestamps go backwards.  Edit a file, decide
>  you made a mistake, restore from a backup or a repository and the
>  timestamp goes backwards.

Ah, *that* is the point of rebuilding the Makefile every time.  Sounds
like you are tried to write a better make utility, not better Makefiles.

Just curious:  If kbuild does all the work usually done by make (i.e.
check timestamps, look what needs rebuilding, ...), why do you need make
at all?  IMHO this is bad designed:  People know what make is and how it
works, but kbuild (ab)uses make in different ways.  Which is bad from
the usability point of view because people simply don't expect that.
That is the reason why the question about the Makefile generation comes
up again and again.  And I'm pretty sure that will never stop if you
keep that design.

I think you should either use make the usual way, i.e. let make do all
the timestamp checking (I know it is less strict, but I don't think it
is a big issue because developers know how make works and what the
pitfalls are).  Or don't use make at all.

  Gerd

-- 
You can't please everybody.  And usually if you _try_ to please
everybody, the end result is one big mess.
				-- Linus Torvalds, 2002-04-20

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On 6 May 2002 08:40:07 GMT, 
Gerd Knorr <kraxel@bytesex.org> wrote:
>Just curious:  If kbuild does all the work usually done by make (i.e.
>check timestamps, look what needs rebuilding, ...), why do you need make
>at all?

kbuild does not do all the work.  It is a wrapper around make to
overcome problems which have bitten kbuild in the past and will
continue to bite us as long as we do things that make was not designed
to handle.  Check out the replacements being written for make, you find
that almost all of them handle timestamps going backwards.

kbuild requires other processing that is not handled by make nor by any
of the proposed replacements.  In particular, the two level dependency
chain on configs as well as timestamps.

>IMHO this is bad designed:  People know what make is and how it
>works, but kbuild (ab)uses make in different ways.  Which is bad from
>the usability point of view because people simply don't expect that.

kbuild has abused make for years.  Look at all the code in the top
level Makefile, in Rules.make, the .depend and .hdepend files.  All of
it is special processing for kbuild to do things that make does not do
automatically.  Those requirements did not go away, I just handled it
in a cleaner method in kbuild 2.5.

>I think you should either use make the usual way, i.e. let make do all
>the timestamp checking (I know it is less strict, but I don't think it
>is a big issue because developers know how make works and what the
>pitfalls are).

You obviously believe in the "every kernel builder is an expert who
never makes mistakes" model.  I don't.  Everybody makes mistakes,
kernel build is too important to rely on fallible human actions.

>Or don't use make at all.

make does a very good job once it has been given a global makefile and
the timestamp skew has been handled.  If I did not use make, I would
have write my own program which did exactly the same, pointless.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Richard Gooch]

Paul Mackerras writes:
> But when have you known a kernel hacker to be satisfied with just
> "faster than the previous system", as distinct from "as fast as I can
> reasonably make it go"? ;-)
[...]
> Don't get me wrong, I think it's great to have all the advantages
> that kbuild-2.5 brings.  However, I also think that those seconds
> spent in the startup code will tend to have a disproportionate
> effect on people's perceptions of the new system.  I know you have
> already spent a lot of effort on this, but I want to get in and have
> a look myself to see if I can spot anything that could be improved
> there.

As Keith says, the new code is faster and more robust than the old
code. Given that tracking kernel drift is a significant load on him,
it makes sense to incorporate the new code now. Once it's in, let
people get used to it and then we can look at optimising it, if need
be. Delaying introduction into the kernel tree because it's not 100%
optimised is wasteful.

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Urban Widmark]

On Thu, 2 May 2002, Keith Owens wrote:

> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
> It is faster, better documented, easier to write build rules in, has
> better install facilities, allows separate source and object trees, can
> do concurrent builds from the same source tree and is significantly
> more accurate than the existing kernel build system.

Faster ... ?

The time I care about is the module rebuild time. From various posts I had
the impression that it was significantly improved. I don't find that to be
the case unless I'm being "foolhardy" and specify various flags or
otherwise bypass the system.

The time to rebuild everything isn't that interesting to me as it's too
long anyway to sit around and wait for it to complete. Maybe that is where
the improvements are.


Here is how I work. First I configure and build a complete kernel for the
.config I use. Something like:

cp ../.config-2.5 .config
make -f Makefile-2.5 oldconfig
make -f Makefile-2.5 installable

Then I will do a number of rebuild modules & install cycles, where the
environment does not change (.config, tools, etc) except for the module I
am working on. The times I get for various things are:

A - Time to rebuild smbfs.o with proc.c changed:
	time make -f Makefile-2.5 fs/smbfs/smbfs.o
	51.910u 2.760s 0:54.95 99.4%    0+0k 0+0io 102368pf+0w

B - Same, with NO_MAKEFILE_GEN=1, 1st run:
	time make -f Makefile-2.5 NO_MAKEFILE_GEN=1 fs/smbfs/smbfs.o
	28.280u 0.600s 0:28.91 99.8%    0+0k 0+0io 27014pf+0w

C - Same, with NO_MAKEFILE_GEN=1, 2nd run:
	time make -f Makefile-2.5 NO_MAKEFILE_GEN=1 fs/smbfs/smbfs.o
	0.910u 0.420s 0:01.33 100.0%    0+0k 0+0io 21928pf+0w

D - Installing as root:
	time make -f Makefile-2.5 install
	1.110u 0.960s 0:02.63 78.7%     0+0k 0+0io 34905pf+0w

E - proc.c changed, module rebuild+install with the 2.4 build system
   (separate tree from above):
	time make modules modules_install
	21.160u 2.020s 0:23.12 100.2%   0+0k 0+0io 50927pf+0w

(Built on 2.5.8, 2xPIII 500, 512M, kbuild core-11, 2.5.13 tree)


A + D > 2 * E, and this is the way it's supposed to be used, no?
B + D > E
C + D < E, but C becomes a B after installing as the install wants to be
run with NO_MAKEFILE_GEN not set.

The difference between B and C is that pp_makefile4 is run.


I have seen you claim that it is faster, and others have repeated that. I
just wonder at doing what?

A full build it was slightly faster 12:00.47 vs 12:20.55, but here the
kbuild-2.5 build was run twice since I missed that it otherwise uses kgcc
(which makes it ~3 minutes faster) and the "2.4" build did an install
also. So that's about equal.

make oldconfig is also slower 0:21.06 vs, 0:04.55 since it runs phase1
first, and that was done outside the timed command.

Shadow trees sounds interesting. I'm sure others see great benefit from
being able to build over NFS or having stricter integrity checks. I just
don't get the faster bit, but maybe that's just me.

/Urban

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Sun, 5 May 2002 19:23:11 +0200 (CEST), 
Urban Widmark <urban@teststation.com> wrote:
>On Thu, 2 May 2002, Keith Owens wrote:
>
>> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
>> It is faster, better documented, easier to write build rules in, has
>> better install facilities, allows separate source and object trees, can
>> do concurrent builds from the same source tree and is significantly
>> more accurate than the existing kernel build system.
>
>Faster ... ?
>
>The time I care about is the module rebuild time. From various posts I had
>the impression that it was significantly improved. I don't find that to be
>the case unless I'm being "foolhardy" and specify various flags or
>otherwise bypass the system.
>[times snipped]
>Shadow trees sounds interesting. I'm sure others see great benefit from
>being able to build over NFS or having stricter integrity checks. I just
>don't get the faster bit, but maybe that's just me.

You are not comparing like with like.  Much of your speed difference
from kbuild 2.4 to 2.5 is because you have omitted the make dep time.
kbuild 2.5 does not have a seperate make dep pass.  Instead it checks
the dependencies every time, during phase4.

Checking the dependencies only once in kbuild 2.4 is a very common
source of build error.  Users change their code, forget to rerun make
dep then wonder why their kernel and module build is broken.  In your
case, you "know" that your change does not affect the dependencies so
you omit the make dep run.  That is the equivalent of bypassing the
integrity checks in kbuild 2.5, i.e. it is the equivalent of
NO_MAKEFILE_GEN=1.

Also you specified make modules.  You are bypassing all the checks to
see if any part of the main kernel needs to be rebuilt because of your
changes.  Whether that bypass is correct or not is unknown, you are
asserting that it is and bypassing the dependency checks on the rest of
the kernel.  BTW, if you have a lot of modules you will find that your
make modules time in 2.4 is significantly higher than the times you
quoted.

So you found a case that appears to make kbuild 2.4 faster than 2.5.
You did it by omitting the kbuild 2.4 step that does integrity checking
and by specifying command line options that bypass most of the build.
The result is that you are comparing an inaccurate 2.4 build against an
accurate 2.5 build.

I have never considered "fast but inaccurate" to be a sensible default
goal for a kernel build.  If you want that, use NO_MAKEFILE_GEN=1.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Urban Widmark]

On Mon, 6 May 2002, Keith Owens wrote:

> >being able to build over NFS or having stricter integrity checks. I just
> >don't get the faster bit, but maybe that's just me.
> 
> You are not comparing like with like.  Much of your speed difference
> from kbuild 2.4 to 2.5 is because you have omitted the make dep time.
> kbuild 2.5 does not have a seperate make dep pass.  Instead it checks
> the dependencies every time, during phase4.

make dep isn't part of a module rebuild given the constriants I work
under, the changes are local to the module (which they are).

In my world make dep is only relevant for the first build, and the
times I mentioned for the full build includes a dependency build.
(I know the presentation of that part was crap ... but so was the
 measurements :)


> Checking the dependencies only once in kbuild 2.4 is a very common
> source of build error.  Users change their code, forget to rerun make
> dep then wonder why their kernel and module build is broken.  In your
> case, you "know" that your change does not affect the dependencies so
> you omit the make dep run.  That is the equivalent of bypassing the
> integrity checks in kbuild 2.5, i.e. it is the equivalent of
> NO_MAKEFILE_GEN=1.

NO_MAKEFILE_GEN is still slower for me than the way I use make modules.

What you are saying is that I should never do:
make modules

but always:
make dep && make bzImage modules

Ok, then I see what you meant by kbuild-2.5 being faster.

Documentation/kbuild/commands.txt (2.4.18 kernel, don't have anything more
recent at hand) has a section on make dep that says I only have to run it
once after the first time I configure the kernel. Maybe that is where I
picked up that habit.


> the kernel.  BTW, if you have a lot of modules you will find that your
> make modules time in 2.4 is significantly higher than the times you
> quoted.

Sure. I was talking about me, my .config (~30 modules) and my builds
specifically.

Btw, I can see other benefits from kbuild-2.5 (and I can bypass the things
I don't want to run all the time more easily than in "2.4" :) and I'm not
against it, but it didn't live up to the faster claim at anything I
normally do. So I had to ask.

/Urban

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Mon, 6 May 2002 13:33:18 +0200 (CEST), 
Urban Widmark <urban@teststation.com> wrote:
>On Mon, 6 May 2002, Keith Owens wrote:
>
>> >being able to build over NFS or having stricter integrity checks. I just
>> >don't get the faster bit, but maybe that's just me.
>> 
>> You are not comparing like with like.  Much of your speed difference
>> from kbuild 2.4 to 2.5 is because you have omitted the make dep time.
>> kbuild 2.5 does not have a seperate make dep pass.  Instead it checks
>> the dependencies every time, during phase4.
>
>make dep isn't part of a module rebuild given the constriants I work
>under, the changes are local to the module (which they are).
>
>In my world make dep is only relevant for the first build, and the
>times I mentioned for the full build includes a dependency build.
>(I know the presentation of that part was crap ... but so was the
> measurements :)

kbuild 2.4 defaults to doing a (possibly) inaccurate build after
changes.  You have to manually run extra commands to ensure that kbuild
2.4 does an accurate build.  If you believe that your change has not
changed the build dependency graph, it _may_ be safe to omit the extra
commands.

kbuild 2.5 defaults to always doing an accurate build, no matter what
has changed.  You have to specify a command line option to bypass the
full processing and make it (possibly) inaccurate.  If you believe that
your change has not changed the build dependency graph, it _may_ be
safe to specify the command line option.

You have to specify a command line option on kbuild 2.5, to get the
same behaviour that kbuild 2.4 does by default.  It comes down to what
should the default be for a build?

* kbuild 2.4 defaults to assuming that nobody ever makes misteaks.
* kbuild 2.5 defaults to assuming that mistakes occur.

This is almost a religious argument, should the build be unsafe and
assume that everybody is an expert or should the build be safe and
provide facilities for experts to override it?  I am a true believer in
"the build must be safe" model.

>What you are saying is that I should never do:
>make modules
>
>but always:
>make dep && make bzImage modules
>
>Ok, then I see what you meant by kbuild-2.5 being faster.

That is the only way to ensure an accurate build on kbuild 2.4.  Yes,
if you know that your change has no side effects then you can omit make
dep bzImage.  The problem is that many people automatically omit the
extra commands, without considering the implications.

>Documentation/kbuild/commands.txt (2.4.18 kernel, don't have anything more
>recent at hand) has a section on make dep that says I only have to run it
>once after the first time I configure the kernel. Maybe that is where I
>picked up that habit.

The documentation is correct, but incomplete.  It is correct for an end
user kernel build, i.e. for people who do not change the code or apply
patches, they just configure the kernel and build it.  It is incomplete
for developers and for anybody who gets a patch and applies it.

You need to run make dep after any change that affects the build
dependency graph.  Add or delete #include in a source, add or delete a
source, add or delete a config option and you must run make dep to
ensure that the changes rebuild what is required.  Modversions are even
worse, after any change that might affect an exported symbol or
structure, you must make mrproper (not dep) to calculate and apply the
new hashes to the entire kernel.

As more and more people apply patches (how many variants of the kernel
tree are there now?), more and more people are forgetting to run make
dep and are building incomplete kernels.

The default for kernel build must be a safe and accurate build.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alex Riesen]

On Thu, May 02, 2002 at 12:23:33AM +1000, Keith Owens wrote:
> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
> It is faster, better documented, easier to write build rules in, has
> better install facilities, allows separate source and object trees, can
> do concurrent builds from the same source tree and is significantly
> more accurate than the existing kernel build system.

I do not like the new(core-11) "make *config" behaviour. Now it starts
build immediately after finishing, make xconfig effectively does
make xconfig installabled. I usually cook up the .config first, and
than decide when to compile the kernel. Now i have to interrupt the
build.

"make oldconfig" is broken btw, if the .config contains something
unknown (i.e. NEW). It used to ask for possible choices before.

And the last: kbuild-2.5 (as well as kbuild-2.4) keeps to be a good
stress/benchmark-test. Just tried to "make -f Makefile-2.5 -j" on
2.4.19-pre2-ac2... And decided to reboot after 15min (i'm at work now).
:)

-alex

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Mon, 6 May 2002 12:54:35 +0200, 
Alex Riesen <Alexander.Riesen@synopsys.com> wrote:
>On Thu, May 02, 2002 at 12:23:33AM +1000, Keith Owens wrote:
>> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
>> It is faster, better documented, easier to write build rules in, has
>> better install facilities, allows separate source and object trees, can
>> do concurrent builds from the same source tree and is significantly
>> more accurate than the existing kernel build system.
>
>I do not like the new(core-11) "make *config" behaviour. Now it starts
>build immediately after finishing, make xconfig effectively does
>make xconfig installabled. I usually cook up the .config first, and
>than decide when to compile the kernel. Now i have to interrupt the
>build.

I do not see either of these symptoms, and nobody else has reported
them.

# make -f $KBUILD_SRCTREE_000/Makefile-2.5 xconfig
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
  phase 1 (find all inputs)
  (cd /build/kaos/object-2.5.14/.tmp_config/links/ && /build/kaos/object-2.5.14/scripts/tkparse < config.in-2.5) >> /build/kaos/object-2.5.14/scripts/kconfig.tk

xconfig menu displays, clicking save and exit ends xconfig and drops
back to the command prompt, it does not do anything else.

>"make oldconfig" is broken btw, if the .config contains something
>unknown (i.e. NEW). It used to ask for possible choices before.

# make -f $KBUILD_SRCTREE_000/Makefile-2.5 oldconfig
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
  phase 1 (find all inputs)
#
# Using defaults found in .config
#
*
* Code maturity level options
*
Prompt for development and/or incomplete code/drivers (CONFIG_EXPERIMENTAL) [N/y/?] 
*
* General setup
*
Networking support (CONFIG_NET) [N/y/?] 
System V IPC (CONFIG_SYSVIPC) [N/y/?] 
BSD Process Accounting (CONFIG_BSD_PROCESS_ACCT) [N/y/?] 
Sysctl support (CONFIG_SYSCTL) [N/y/?] 
*
* Loadable module support
*
Enable loadable module support (CONFIG_MODULES) [Y/n/?] 
  Set version information on all module symbols (CONFIG_MODVERSIONS) [N/y/?] 
  Kernel module loader (CONFIG_KMOD) [N/y/?] (NEW) 

Works for me.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Alex Riesen]

On Wed, May 08, 2002 at 12:54:39PM +1000, Keith Owens wrote:
> On Mon, 6 May 2002 12:54:35 +0200, 
> Alex Riesen <Alexander.Riesen@synopsys.com> wrote:
> >On Thu, May 02, 2002 at 12:23:33AM +1000, Keith Owens wrote:
> >> Linus, kbuild 2.5 is ready for inclusion in the main 2.5 kernel tree.
> >> It is faster, better documented, easier to write build rules in, has
> >> better install facilities, allows separate source and object trees, can
> >> do concurrent builds from the same source tree and is significantly
> >> more accurate than the existing kernel build system.
> >
> >I do not like the new(core-11) "make *config" behaviour. Now it starts
> >build immediately after finishing, make xconfig effectively does
> >make xconfig installabled. I usually cook up the .config first, and
> >than decide when to compile the kernel. Now i have to interrupt the
> >build.
> 
> I do not see either of these symptoms, and nobody else has reported
> them.

i've found the reason: the make's stdin was redirected in /dev/null
(my make is aliased to a program prettifying output).
It still starts to compile immediately after *config, though.

/compile/kb-test-o gmake -f ../kb-test/Makefile-2.5 xconfig
Using ARCH='i386' AS='as' LD='ld' CC='/usr/bin/gcc' CPP='/usr/bin/gcc -E' AR='ar' HOSTAS='as' HOSTLD='gcc' HOSTCC='gcc' HOSTAR='ar'
Generating global Makefile
  phase 1 (find all inputs)
(cd /export/home/riesen-pc0/riesen/compile/kb-test-o/.tmp_config/links/ && /export/home/riesen-pc0/riesen/compile/kb-test-o/scripts/tkparse < config.in-2.5) >> /export/home/riesen-pc0/riesen/compile/kb-test-o/scripts/kconfig.tk
<pressed "Save&Exit" here...>
  phase 2 (convert all Makefile.in files)
  phase 3 (evaluate selections)
  phase 4 (integrity checks, write global makefile)
Starting phase 5 (build) for installable
  CC arch/i386/kernel/setup.o
...
looks like my *config targets aren't like yours and belong to phase5
targets 8)

-alex

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Wed, 8 May 2002 19:25:57 +0200, 
Alex Riesen <Alexander.Riesen@synopsys.com> wrote:
>i've found the reason: the make's stdin was redirected in /dev/null
>(my make is aliased to a program prettifying output).

Use standard make.

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Daniel Jacobowitz]

On Thu, May 09, 2002 at 10:10:19AM +1000, Keith Owens wrote:
> On Wed, 8 May 2002 19:25:57 +0200, 
> Alex Riesen <Alexander.Riesen@synopsys.com> wrote:
> >i've found the reason: the make's stdin was redirected in /dev/null
> >(my make is aliased to a program prettifying output).
> 
> Use standard make.

Wouldn't you call it a bug anyway if running
'make -f Makefile-2.5 oldconfig < /dev/null' went on to build a kernel? 
That's pretty surprising behavior.

(Not saying that that does happen, but it's how I read Alex's message)

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: kbuild 2.5 is ready for inclusion in the 2.5 kernel

[Keith Owens]

On Wed, 8 May 2002 20:55:46 -0400, 
Daniel Jacobowitz <dan@debian.org> wrote:
>On Thu, May 09, 2002 at 10:10:19AM +1000, Keith Owens wrote:
>> On Wed, 8 May 2002 19:25:57 +0200, 
>> Alex Riesen <Alexander.Riesen@synopsys.com> wrote:
>> >i've found the reason: the make's stdin was redirected in /dev/null
>> >(my make is aliased to a program prettifying output).
>> 
>> Use standard make.
>
>Wouldn't you call it a bug anyway if running
>'make -f Makefile-2.5 oldconfig < /dev/null' went on to build a kernel? 
>That's pretty surprising behavior.
>
>(Not saying that that does happen, but it's how I read Alex's message)

It would be a bug, that is not what is causing the problem.
make oldconfig < /dev/null breaks on both kbuild 2.4 and 2.5, oldconfig
requires input.  If you want no prompts with oldconfig taking defaults
for new values then

  yes '' | make oldconfig (kbuild 2.4)
  make defconfig (kbuild 2.5)

I cannot reproduce Alex's problems using standard make or gmake.  It is
almost certainly a problem with his version of make or the wrapper
around it, it is introducing spurious targets.  If the problem occurs
using standard GNU make then I will look at it.