kbuild: Implicit dependence on the C compiler

kbuild: Implicit dependence on the C compiler

[H. Peter Anvin]

Okay, this is driving me utterly crazy...

How the heck do I get kbuild to *not* think that because I'm using a
different C compiler (including "gcc" versus "distcc"), or I'm on a
different host, that it has to rebuild every single object file in my
directory?  This is an unbelievable headache.

	-hpa

Re: kbuild: Implicit dependence on the C compiler

[Sam Ravnborg]

On Mon, Jan 17, 2005 at 09:40:55PM +0000, H. Peter Anvin wrote:
> Okay, this is driving me utterly crazy...
> 
> How the heck do I get kbuild to *not* think that because I'm using a
> different C compiler (including "gcc" versus "distcc"), or I'm on a
> different host, that it has to rebuild every single object file in my
> directory?  This is an unbelievable headache.

It better be difficult. You want to recompile when changing gcc.
Try this untested patch.

There is no way to tell kbuild "ignore gcc change"

	Sam


===== Makefile.lib 1.27 vs edited =====
--- 1.27/scripts/Makefile.lib	2004-10-27 00:06:46 +02:00
+++ edited/Makefile.lib	2005-01-17 22:58:44 +01:00
@@ -187,9 +187,7 @@
 # >'< substitution is for echo to work, >$< substitution to preserve $ when reloading .cmd file
 # note: when using inline perl scripts [perl -e '...$$t=1;...'] in $(cmd_xxx) double $$ your perl vars
 
-if_changed = $(if $(strip $? \
-		          $(filter-out $(cmd_$(1)),$(cmd_$@))\
-			  $(filter-out $(cmd_$@),$(cmd_$(1)))),\
+if_changed = \
 	@set -e; \
 	$(if $($(quiet)cmd_$(1)),echo '  $(subst ','\'',$($(quiet)cmd_$(1)))';) \
 	$(cmd_$(1)); \
@@ -199,9 +197,7 @@
 # execute the command and also postprocess generated .d dependencies
 # file
 
-if_changed_dep = $(if $(strip $? $(filter-out FORCE $(wildcard $^),$^)\
-		          $(filter-out $(cmd_$(1)),$(cmd_$@))\
-			  $(filter-out $(cmd_$@),$(cmd_$(1)))),\
+if_changed_dep = \ 
 	@set -e; \
 	$(if $($(quiet)cmd_$(1)),echo '  $(subst ','\'',$($(quiet)cmd_$(1)))';) \
 	$(cmd_$(1)); \
@@ -213,9 +209,7 @@
 # will check if $(cmd_foo) changed, or any of the prequisites changed,
 # and if so will execute $(rule_foo)
 
-if_changed_rule = $(if $(strip $? \
-		               $(filter-out $(cmd_$(1)),$(cmd_$@))\
-			       $(filter-out $(cmd_$@),$(cmd_$(1)))),\
+if_changed_rule = \
 			@set -e; \
 			$(rule_$(1)))
 

Re: kbuild: Implicit dependence on the C compiler

[H. Peter Anvin]

Sam Ravnborg wrote:
> 
> It better be difficult. You want to recompile when changing gcc.
> Try this untested patch.
> 

Sorry, but that's baloney.  Saying "it better be difficult" is 
equivalent to saying "kbuild is smarter than you."

I don't mind the current default, but saying I shouldn't be able to 
override it is asinine.

It also means "make install" is largely unusable.

> There is no way to tell kbuild "ignore gcc change"

There really needs to be one.

	-hpa

Re: kbuild: Implicit dependence on the C compiler

[Sam Ravnborg]

On Mon, Jan 17, 2005 at 02:03:41PM -0800, H. Peter Anvin wrote:
> I don't mind the current default, but saying I shouldn't be able to 
> override it is asinine.
No-one asked for it until now.
Any preferred syntax to disable this dependency check?

> 
> It also means "make install" is largely unusable.

Maybe we should not let install be dependent on vmlinux then?

	Sam

Re: kbuild: Implicit dependence on the C compiler

[H. Peter Anvin]

Sam Ravnborg wrote:
> On Mon, Jan 17, 2005 at 02:03:41PM -0800, H. Peter Anvin wrote:
> 
>>I don't mind the current default, but saying I shouldn't be able to 
>>override it is asinine.
> 
> No-one asked for it until now.

That's of course perfectly fair, and I'm not flaming you for saying that 
noone had asked for it.  Saying that it *shouldn't* be done is another 
matter, that's all.

> Any preferred syntax to disable this dependency check?

How about "make CCDEP=0"?

> 
>>It also means "make install" is largely unusable.
> 
> Maybe we should not let install be dependent on vmlinux then?

Maybe not.  I don't think "modules_install" have any explicit 
dependencies, it's up to the user to make sure things are already built, 
so we might as well do the same thing with "make install".

	-hpa

Re: kbuild: Implicit dependence on the C compiler

[Sam Ravnborg]

Hi hpa
> >It better be difficult. You want to recompile when changing gcc.
> >Try this untested patch.
> Sorry, but that's baloney.  Saying "it better be difficult" is 
> equivalent to saying "kbuild is smarter than you."

To give some background info about why kbuild does what it does.
A kernel being compiled partly with and partly without say -regparm=3
will result in a non-workable kernel.

The same goes for a kernel that is partly built using gcc 2.96, partly
using 3.3.4 for example.

So kbuild pr. default will force a recompile for any .o file where
opions to gcc differ, or name of gcc has changed. Today no check has
been implemented to check the actual gcc executable timestamp - and
neither is this planned.

Default behaviour today is to recompile if anything change.

But as hpa points outs this hits us with nfs mounted kernel tree when
performing a make install - because install has vmlinux as prerequsite.
So this leaves us with at least two possibilitites:
1) Unconditionally execute make install assuming vmlinux is up-to-date.
   make modules_install run unconditionally, so this is already know
   practice
2) Detect that aother user is running the build - and therefore skip
   the kernel and the module build.
   This is a rather intrusive change since with current kbuild structure
   it is rather difficult to stop this in all relevant cases.

	Sam

Re: kbuild: Implicit dependence on the C compiler

[H. Peter Anvin]

Followup to:  <20050118190513.GA16120@mars.ravnborg.org>
By author:    Sam Ravnborg <sam@ravnborg.org>
In newsgroup: linux.dev.kernel
> 
> To give some background info about why kbuild does what it does.
> A kernel being compiled partly with and partly without say -regparm=3
> will result in a non-workable kernel.
> 
> The same goes for a kernel that is partly built using gcc 2.96, partly
> using 3.3.4 for example.
> 
> So kbuild pr. default will force a recompile for any .o file where
> opions to gcc differ, or name of gcc has changed. Today no check has
> been implemented to check the actual gcc executable timestamp - and
> neither is this planned.
> 

I would argue that "name of gcc has changed" is possibly a condition
that does more harm than good.  It is just as frequently used to have
wrappers, like distcc, as it is to have different versions.

(FWIW, nothing is more obnoxious than having the kernel tree blown
away when you try to compile in the one missing driver needed to talk
to the rest of your distcc cluster.)

> Default behaviour today is to recompile if anything change.
> 
> But as hpa points outs this hits us with nfs mounted kernel tree when
> performing a make install - because install has vmlinux as prerequsite.
> So this leaves us with at least two possibilitites:
> 1) Unconditionally execute make install assuming vmlinux is up-to-date.
>    make modules_install run unconditionally, so this is already know
>    practice
> 2) Detect that aother user is running the build - and therefore skip
>    the kernel and the module build.
>    This is a rather intrusive change since with current kbuild structure
>    it is rather difficult to stop this in all relevant cases.

I say unconditionally do make install, but there really, REALLY, need
to be a way to override this check manually.

	-hpa

Re: kbuild: Implicit dependence on the C compiler

[Matt Mackall]

On Tue, Jan 18, 2005 at 07:35:43PM +0000, H. Peter Anvin wrote:
> Followup to:  <20050118190513.GA16120@mars.ravnborg.org>
> By author:    Sam Ravnborg <sam@ravnborg.org>
> In newsgroup: linux.dev.kernel
> > 
> > To give some background info about why kbuild does what it does.
> > A kernel being compiled partly with and partly without say -regparm=3
> > will result in a non-workable kernel.
> > 
> > The same goes for a kernel that is partly built using gcc 2.96, partly
> > using 3.3.4 for example.
> > 
> > So kbuild pr. default will force a recompile for any .o file where
> > opions to gcc differ, or name of gcc has changed. Today no check has
> > been implemented to check the actual gcc executable timestamp - and
> > neither is this planned.
> > 
> 
> I would argue that "name of gcc has changed" is possibly a condition
> that does more harm than good.  It is just as frequently used to have
> wrappers, like distcc, as it is to have different versions.

Disagree. I switch compilers all the time and kbuild does the right
thing for me.

I do occassionally feel your 'make install' pain and some sort of
'make __install' might be called for.

-- 
Mathematics is the supreme nostalgia of our time.

Re: kbuild: Implicit dependence on the C compiler

[H. Peter Anvin]

Matt Mackall wrote:
>>
>>I would argue that "name of gcc has changed" is possibly a condition
>>that does more harm than good.  It is just as frequently used to have
>>wrappers, like distcc, as it is to have different versions.
> 
> Disagree. I switch compilers all the time and kbuild does the right
> thing for me.
> 
> I do occassionally feel your 'make install' pain and some sort of
> 'make __install' might be called for.
> 

As I said, I don't mind the default being there, it's certainly 
consistent with the default being safe.  However, not being able to 
override it is bad.

	-hpa

Re: kbuild: Implicit dependence on the C compiler

[Marcin Dalecki]

On 2005-01-19, at 04:35, H. Peter Anvin wrote:

> Matt Mackall wrote:
>>>
>>> I would argue that "name of gcc has changed" is possibly a condition
>>> that does more harm than good.  It is just as frequently used to have
>>> wrappers, like distcc, as it is to have different versions.
>> Disagree. I switch compilers all the time and kbuild does the right
>> thing for me.
>> I do occassionally feel your 'make install' pain and some sort of
>> 'make __install' might be called for.
>
> As I said, I don't mind the default being there, it's certainly 
> consistent with the default being safe.  However, not being able to 
> override it is bad.

Just please consider

CC ?= gcc

instead of

CC = gcc

in Makefiles. I assume it does precisely what you want. So I think 
anybody arguing against
you is just arguing about a single ASCII character...

Re: kbuild: Implicit dependence on the C compiler

[Sam Ravnborg]

On Mon, Jan 17, 2005 at 02:03:41PM -0800, H. Peter Anvin wrote:
> >There is no way to tell kbuild "ignore gcc change"
> 
> There really needs to be one.

make KBUILD_NOCMDDEP=1
will do what you want - at least I have it in my tree now.
I could not just ignore 'gcc' - but had to ignore the full commandline.

This is due to more complex commands like:
rm -f file; $(LD) ...

Within the Makefile.lib when I check KBUILD_NOCMDDEP there is no
knowledge of the actual command being executed. And an implmentation
that just filtered out $(CC) was too ugly.
And due to the above mentioned command I could not just skip the first
word on the command line.


I will push my bk tree soon and it will show up in next -mm.

It is not perfect in the sense that the last part of the build will get
redone (GEN .version and onwards). This is fixable but not worth it
right now.

So with current implmentation executing:

make

make KBUILD_NOCMDDEP=1 CROSS_COMPILE=i586-pc-linux-gnu-

will result in only a few files being rebuild - and not the whole
kernel as before.

	Sam

Re: kbuild: Implicit dependence on the C compiler

[H. Peter Anvin]

Sam Ravnborg wrote:
> 
> make KBUILD_NOCMDDEP=1
> will do what you want - at least I have it in my tree now.
> I could not just ignore 'gcc' - but had to ignore the full commandline.
> 
> This is due to more complex commands like:
> rm -f file; $(LD) ...
> 

That's probably the right behaviour actually.

> Within the Makefile.lib when I check KBUILD_NOCMDDEP there is no
> knowledge of the actual command being executed. And an implmentation
> that just filtered out $(CC) was too ugly.
> And due to the above mentioned command I could not just skip the first
> word on the command line.
> 
> 
> I will push my bk tree soon and it will show up in next -mm.
> 
> It is not perfect in the sense that the last part of the build will get
> redone (GEN .version and onwards). This is fixable but not worth it
> right now.
> 
> So with current implmentation executing:
> 
> make
> 
> make KBUILD_NOCMDDEP=1 CROSS_COMPILE=i586-pc-linux-gnu-
> 
> will result in only a few files being rebuild - and not the whole
> kernel as before.

That's fair.  You got what you asked for.

	-hpa

Re: kbuild: Implicit dependence on the C compiler

[Bodo Eggert]

Sam Ravnborg <sam@ravnborg.org> wrote:

> 1) Unconditionally execute make install assuming vmlinux is up-to-date.
>    make modules_install run unconditionally, so this is already know
>    practice

(o) Vote for this.

IMO, a make install should NEVER run the compiler.

The reason is: I'm deliberaely compiling as a user on the fastest system
before I copy or mount the FS to/on some other box and do a make install.
The other box will be _very_ slow while compiling or missing some of the
needed components (e.g. gcc).

Re: kbuild: Implicit dependence on the C compiler

[linux-os]

On Wed, 19 Jan 2005, Bodo Eggert wrote:

> Sam Ravnborg <sam@ravnborg.org> wrote:
>
>> 1) Unconditionally execute make install assuming vmlinux is up-to-date.
>>    make modules_install run unconditionally, so this is already know
>>    practice
>
> (o) Vote for this.
>
> IMO, a make install should NEVER run the compiler.
>
> The reason is: I'm deliberaely compiling as a user on the fastest system
> before I copy or mount the FS to/on some other box and do a make install.
> The other box will be _very_ slow while compiling or missing some of the
> needed components (e.g. gcc).
>

You must never execute `make install` or `make modules_install` without
the explicit action of the operator! To do so could (will) result
in an un-bootable system. I can't imagine what somebody would be
thinking to propose an automatic install. Whoever proposed this
must have lots of time and little knowledge. They are going to
be reinstalling everything from the distribution CD as a hobby.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: kbuild: Implicit dependence on the C compiler

[Sytse Wielinga]

On Wed, Jan 19, 2005 at 11:35:55AM -0500, linux-os wrote:
> >Sam Ravnborg <sam@ravnborg.org> wrote:
> >>1) Unconditionally execute make install assuming vmlinux is up-to-date.
> >>   make modules_install run unconditionally, so this is already know
> >>   practice
> 
> You must never execute `make install` or `make modules_install` without
> the explicit action of the operator! To do so could (will) result
> in an un-bootable system. I can't imagine what somebody would be
> thinking to propose an automatic install. Whoever proposed this
> must have lots of time and little knowledge. They are going to
> be reinstalling everything from the distribution CD as a hobby.

What I think Sam meant was, to let make install assume that vmlinux is
up-to-date, IOW, just to remove the dependency of install on vmlinux.

Err, to put it in your words: I can't imagine what somebody would be thinking
to think he proposed an automatic install. :-P

	Sytse