Re: Various kbuild problems in 2.5.22

Re: Various kbuild problems in 2.5.22

[Adam J. Richter]

>On Tue, 18 Jun 2002, Adam J. Richter wrote:

>> 	I would like to note the following problems with the
>> kernel build process in 2.5.22, after applying the patch that
>> Kai Germaschewski posted that enabled modversions to work again.
>> All but the first one are spurious dependencies.

>> #define __ver_pcmcia_get_mem_page_Rsmp_3d2ded54 smp_ba03375b
>> #define pcmcia_get_mem_page_Rsmp_3d2ded54       _set_ver(pcmcia_get_mem_page_Rsmp_3d2ded54)

>Yes, the fix I posted was not complete. I submitted a corrected one 
>already.

>> 	2. "make bzImage" does not build a bzImage if any module fails
>> to compile.  Really, it should not attempt to buidl modules or even
>> descend into directories that contain only modules.  To build a bzImage,
>> I have to edit the Makefile and comment out "BUILD_MODULES:=1".

>That's intentional. If you don't want to build modules, use "make 
>KBUILD_MODULES= bzImage". Or you can just ignore errors using "make -k".

	No, "make -k" still will not build bzImage if a module
fails to compile.

	Also, I do not understand why this is "intentional."  Normally,
if one does a "make" of a file in a source tree, build problems with
unneeded files do not effect it.


>> 	3. make include/linux/modversios.h aborts if any .c file has
>> a #error or #include's a .h that is not present (for example, because
>> the .h is built by the process, as is the case with one scsi driver).

>The fact that it aborts is intentional.

	We have adopted a convention of putting #error into lots
of device drivers to encourage people to port them.  Linus has
also recently integrated chagnes to support compiling with "all
modules" and "all yes" configurations.  This change makes that
facility useless.

	I do not think it improves anyone's prioritization to
require everyone to either make custom kernel configurations or
give top priority to fixing random drivers ahead of whatever
else depends on their getting the new kernel to build.


>That it doesn't build the .h in that case is a bug. Which driver is it?

	53c700.  The generated header file is drivers/scsi/53c700_d.h.

>> 	4. "make -k modules" will not build perfectly buildable modules
>> in a directory that has a subdirectory where a compile error occurs.

>Well, I can fix that, I'll look into it.

	Great.  Thanks.

Adam J. Richter     __     ______________   575 Oroville Road
adam@yggdrasil.com     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America
                         "Free Software For The Rest Of Us."

Re: Various kbuild problems in 2.5.22

[Kai Germaschewski]

On Tue, 18 Jun 2002, Adam J. Richter wrote:

> 	No, "make -k" still will not build bzImage if a module
> fails to compile.
> 
> 	Also, I do not understand why this is "intentional."  Normally,
> if one does a "make" of a file in a source tree, build problems with
> unneeded files do not effect it.

Yes, but they are not unneeded files, otherwise we wouldn't even try to 
build them. The point is, the semantics of bzImage changed: It now means 
"build bzImage and modules". That's the common case. If you really only 
want bzImage and no modules, you have to tell make by using
"make KBUILD_MODULES= bzImage" (I could allow for phrasing the latter as
"make bzImage nomodules", but that's only cosmetical)

> >> 	3. make include/linux/modversios.h aborts if any .c file has
> >> a #error or #include's a .h that is not present (for example, because
> >> the .h is built by the process, as is the case with one scsi driver).
> 
> >The fact that it aborts is intentional.
> 
> 	We have adopted a convention of putting #error into lots
> of device drivers to encourage people to port them.  Linus has
> also recently integrated chagnes to support compiling with "all
> modules" and "all yes" configurations.  This change makes that
> facility useless.

IMO these options (all yes/mod)  are there to find files which don't
compile, be it for an explicit #error or other reasons - and they serve
this purpose, they now flag those files already at "make dep" time if
you're using modversions.

Of course I could go ahead if I get an error during module version
generation, but then I'd get the exact same error later when compiling. So
I don't see the point. Module versions used to be fragile, exactly for
reasons like this. If this step goes wrong, just silently ignoring that
fact will get you in trouble later.

> 	I do not think it improves anyone's prioritization to
> require everyone to either make custom kernel configurations or
> give top priority to fixing random drivers ahead of whatever
> else depends on their getting the new kernel to build.

Well, normally people carry their .config along and adapt it as necessary. 
If a driver stopped building, if you don't need it, disable it in your 
.config, if you need it, just ignoring the failure won't help you either. 

Apart from that, "make modules_install" never worked in the case of 
failed builds, did it? - so it boils down to: you need a buildable .config 
to build and test a kernel.

> >That it doesn't build the .h in that case is a bug. Which driver is it?
> 
> 	53c700.  The generated header file is drivers/scsi/53c700_d.h.

Okay, I fixed that here.

> >> 	4. "make -k modules" will not build perfectly buildable modules
> >> in a directory that has a subdirectory where a compile error occurs.

Actually, I tried and don't see this happening.

--Kai

Re: Various kbuild problems in 2.5.22

[Gerd Knorr]

>  Apart from that, "make modules_install" never worked in the case of 
>  failed builds, did it? - so it boils down to: you need a buildable .config 
>  to build and test a kernel.

No.  2.4.x allows me to to "make modules_install" even if some modules
don't work.  My kernel build script compiles the modules with "make -k
modules", and if some non-essential modules doesn't build I can simply
ignore that and go ahead.  There is no need to disable that module
temporarely in .config (and risc to forget to reenable it later ...).

  Gerd

-- 
You can't please everybody.  And usually if you _try_ to please
everybody, the end result is one big mess.
				-- Linus Torvalds, 2002-04-20

Re: Various kbuild problems in 2.5.22

[Martin Dalecki]

Użytkownik Kai Germaschewski napisał:
> On Tue, 18 Jun 2002, Adam J. Richter wrote:
> 
> 
>>	No, "make -k" still will not build bzImage if a module
>>fails to compile.
>>
>>	Also, I do not understand why this is "intentional."  Normally,
>>if one does a "make" of a file in a source tree, build problems with
>>unneeded files do not effect it.
> 
> 
> Yes, but they are not unneeded files, otherwise we wouldn't even try to 
> build them. The point is, the semantics of bzImage changed: It now means 
> "build bzImage and modules". That's the common case. If you really only 

It is very inconenient if you are working only on some things
which don't affect any thing you are compiling as a module
during the edit/compile cycle or if you don't care to update
some modules you have just in case configured during developement.

How does one build the sole non modularized part of the kernel
nowadays? (Note: it is inconvenient now, but I'm not insisting.)

> want bzImage and no modules, you have to tell make by using
> "make KBUILD_MODULES= bzImage" (I could allow for phrasing the latter as
> "make bzImage nomodules", but that's only cosmetical)

Re: Various kbuild problems in 2.5.22

[Adam J. Richter]

	Before anyone notices, I want to clarify something I wrote:

| Russell King, for example, has complained on linux-kernel
| that the pressure to "make it compile" lead to bad fixes which are now
| more hidden because just the compiler symptoms have been fixed.

	I only meant to attribute the opinion before the word "which"
to Russell.

Adam J. Richter     __     ______________   575 Oroville Road
adam@yggdrasil.com     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America
                         "Free Software For The Rest Of Us."

Re: Various kbuild problems in 2.5.22

[Adam J. Richter]

>>> = Adam Richter
>>  = Sam Ravnborg

>>>         If I want to the kernel to build to continue even when a module
>>>  fails to compile, I should be able to do that by just using "-k".  Not
>>>  being able to build include/linux/modversions.h prevents me from doing
>>>  that.
>>>  From a conceptual point I disagree here. I would like make to
>> avoid completion in case an error is flagged.
>> My prediction is that the new behaviour may result in more errors being
>> corrected, due to the incentitive to do it.

	For most drivers, the result of these changes will simply be
that more device drivers are compiled out and they get less attention.
For example, Alan Cox has said that he does not want anyone trying to
fix the i2o drivers unless they have hardware to test the potential
bugs that he is worried about, especially 64-bit and big endian
issues.  Russell King, for example, has complained on linux-kernel
that the pressure to "make it compile" lead to bad fixes which are now
more hidden because just the compiler symptoms have been fixed.

       Even if you do succeed at getting people to over-prioritize
making certain files compile, it will be at the expense of whatever
else they might have done with that time, such as working on quality
of code that they are more specialized to handle, security,
implementing facilities that would subsequently have made the driver
adjustments much easier, smaller and cleaner.

>>Today you ignore it
>> and hardly cannot spot it in all the noise generated during the build
>> process.

	Baloney.  If you did "make modules" without "-k" before, the
build stopped at the first error, and I'm not asking for that to change.


> = Kai Germaschewski
>Let me second this. In particular, there is no way to reliably generate
>module versions when the affected files cannot even be preprocessed.

	It worked fine before.  I am not concerned about the versioning
of symbols in files that do not compile, as attempts to link to them
will fail anyhow.  I may have to rebuild modversions.h if I fixed a
file that exports symbols, but that can often be true if the file
compiled before.

Adam J. Richter     __     ______________   575 Oroville Road
adam@yggdrasil.com     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America
                         "Free Software For The Rest Of Us."

Re: Various kbuild problems in 2.5.22

[Adam J. Richter]

Kai Germaschewski writes:

>Well, let's say I agree that the kind of semantics change regarding
>building modules at the same time isn't the nicest. So I propose the
>following:

>make bzImage -> compile built-in, build bzImage
>make modules -> compile modules
>make bzImage modules -> build bzImage + modules (I found a way to do so
>                        without having to descend twice)

>make,
>make all     -> compile vmlinux + modules as a general default,
>                on i386 build bzImage + modules.
>               (other archs can change the behavior as they wish)

        Great.  That's fine with me.

Adam J. Richter     __     ______________   575 Oroville Road
adam@yggdrasil.com     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America
                         "Free Software For The Rest Of Us."

Re: Various kbuild problems in 2.5.22

[Kai Germaschewski]

On Tue, 18 Jun 2002, Adam J. Richter wrote:

> >Well, let's say I agree that the kind of semantics change regarding
> >building modules at the same time isn't the nicest. So I propose the
> >following:
> 
> >make bzImage -> compile built-in, build bzImage
> >make modules -> compile modules
> >make bzImage modules -> build bzImage + modules (I found a way to do so
> >                        without having to descend twice)
> 
> >make,
> >make all     -> compile vmlinux + modules as a general default,
> >                on i386 build bzImage + modules.
> >               (other archs can change the behavior as they wish)
> 
>         Great.  That's fine with me.

Okay. This (and some of the other small fixes, like 53c700 modversions 
generation) didn't make it into 2.5.23, but should be in the next release 
- couldn't keep up with Linus' pace ;)

--Kai

Re: Various kbuild problems in 2.5.22

[Adam J. Richter]

On Tue, 18 Jun 2002, Kai Germaschewski wrote:
>On Tue, 18 Jun 2002, Adam J. Richter wrote:

>> 	No, "make -k" still will not build bzImage if a module
>> fails to compile.
>> 
>> 	Also, I do not understand why this is "intentional."  Normally,
>> if one does a "make" of a file in a source tree, build problems with
>> unneeded files do not effect it.

>Yes, but they are not unneeded files, otherwise we wouldn't even try to 
>build them. The point is, the semantics of bzImage changed: It now means 
>"build bzImage and modules".  That's the common case. If you really only 
>want bzImage and no modules, you have to tell make by using
>"make KBUILD_MODULES= bzImage" (I could allow for phrasing the latter as
>"make bzImage nomodules", but that's only cosmetical)

	The standard for make is that if you name the target, it
builds the target.  If I want to make bzImage and modules, I should type
"make bzImage modules".

      I agree with making the common case easier to build, but I would
happy to have "make bzImage modules" be activatable by "make all" or
"make" (or both).


>> >> 	3. make include/linux/modversios.h aborts if any .c file has
>> >> a #error or #include's a .h that is not present (for example, because
>> >> the .h is built by the process, as is the case with one scsi driver).
>> 
>> >The fact that it aborts is intentional.
>> 
>> 	We have adopted a convention of putting #error into lots
>> of device drivers to encourage people to port them.  Linus has
>> also recently integrated chagnes to support compiling with "all
>> modules" and "all yes" configurations.  This change makes that
>> facility useless.

>IMO these options (all yes/mod)  are there to find files which don't
>compile, be it for an explicit #error or other reasons - and they serve
>this purpose, they now flag those files already at "make dep" time if
>you're using modversions.

>Of course I could go ahead if I get an error during module version
>generation, but then I'd get the exact same error later when compiling. So
>I don't see the point.

	The point is that people should be able to select "make
everything a module", build and install the kernel, and let the hot
plugging system worry about what modules to load.  If you believe in
custom kernel configuartions for each user, that is your choice, but
please do not impede those of us who just build everything.

       If I want to the kernel to build to continue even when a module
fails to compile, I should be able to do that by just using "-k".  Not
being able to build include/linux/modversions.h prevents me from doing
that.

	If the current behavior were the default, but "make -k
include/linux/modversions.h" would build modversions.h, that would be
acceptable, by the way.  As things stand under 2.5.22, I have to
comment out all of the #error calls to get the kernel to compile, and
that is particularly annoying because I like to go through the
#error's later and occasionally donate some of my time to trying to
fix a driver.


>Module versions used to be fragile, exactly for
>reasons like this. If this step goes wrong, just silently ignoring that
>fact will get you in trouble later.

      The error message is being printed, so it is not silent.
Building modversions.h will not get you into any trouble later on that
the subsequent compilation failuer will miss.


>> 	I do not think it improves anyone's prioritization to
>> require everyone to either make custom kernel configurations or
>> give top priority to fixing random drivers ahead of whatever
>> else depends on their getting the new kernel to build.

>Well, normally people carry their .config along and adapt it as necessary.

       Well, I like to have a single complete kernel build that
cover most hardware and let user level run time configuration
take care of the rest.  Given that most Linux users are not kernel
developers, getting the souce and recompiling will be a lot more
practical if more generic kernel configurations are not prevented
by your system.

>Apart from that, "make modules_install" never worked in the case of 
>failed builds, did it? - so it boils down to: you need a buildable .config 
>to build and test a kernel.

	Before, if some modules did not build, "make -k modules_install"
would still install all of the modules that did build.

>> >That it doesn't build the .h in that case is a bug. Which driver is it?
>> 
>> 	53c700.  The generated header file is drivers/scsi/53c700_d.h.

>Okay, I fixed that here.

>> >> 	4. "make -k modules" will not build perfectly buildable modules
>> >> in a directory that has a subdirectory where a compile error occurs.

>Actually, I tried and don't see this happening.

	For example, when I attempted to build a kernel with every
tristate set to "module" (except for romfs and the ramdisk), the
following sound drivers did not compile, because they needed
<linux/init.h>:

		sound/isa/ad1848/ad1848_lib.c.
		sound/pci/korg1212/korg1212.c,
		sound/pci/cs46xx/cs46xx.c

	With these modules not compiling, "make -k modules" did
not build sound/soundcore.o even though there was no real
dependency problem.

Adam J. Richter     __     ______________   575 Oroville Road
adam@yggdrasil.com     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America
                         "Free Software For The Rest Of Us."

Re: Various kbuild problems in 2.5.22

[Sam Ravnborg]

On Tue, Jun 18, 2002 at 10:10:59AM -0700, Adam J. Richter wrote:
> 
> 	The standard for make is that if you name the target, it
> builds the target.  If I want to make bzImage and modules, I should type
> "make bzImage modules".
As it is in 2.5.22 make bzImage compares to make installable in kbuild-2.5.
What about combining best of both worlds?

Let
make bzImage	-> Build bzImage
make modules	-> Build modules

And the new member of the family:
make kernel	-> Build selected binary and modules.

So "make kernel" is similar to kbuild-2.5 "make installable" a name 
that I dislike. Obviously "make kernel" requires support for selecting
the appropriate binary utilising make *config.

That would solve your concerns about the semantics - or?

> 
>       I agree with making the common case easier to build, but I would
> happy to have "make bzImage modules" be activatable by "make all" or
> "make" (or both).
"make all" I dislike.
"make all" is for me everything that can be maked, including doc's etc.
make with no argument should in my opinion be equal to "make kernel" -
so we agree there.

>        If I want to the kernel to build to continue even when a module
> fails to compile, I should be able to do that by just using "-k".  Not
> being able to build include/linux/modversions.h prevents me from doing
> that.
>From a conceptual point I disagree here. I would like make to
avoid completion in case an error is flagged.
My prediction is that the new behaviour may result in more errors being
corrected, due to the incentitive to do it. Today you ignore it
and hardly cannot spot it in all the noise generated during the build
process.
By the way - anyone having feedback on the "make KBUILD_VERBOSE=0"
mode. Why not make it default?

	Sam

Re: Various kbuild problems in 2.5.22

[Kai Germaschewski]

On Tue, 18 Jun 2002, Sam Ravnborg wrote:

> > 	The standard for make is that if you name the target, it
> > builds the target.  If I want to make bzImage and modules, I should type
> > "make bzImage modules".

Nitpick: 
[kai@chaos linux-2.5.make]$ make bzImage && ls bzImage
[..]
ls: bzImage: No such file or directory

So you that bzImage isn't a real target (arch/i386/boot/bzImage would be).

> As it is in 2.5.22 make bzImage compares to make installable in kbuild-2.5.
> What about combining best of both worlds?
> 
> Let
> make bzImage	-> Build bzImage
> make modules	-> Build modules
> 
> And the new member of the family:
> make kernel	-> Build selected binary and modules.
> 
> So "make kernel" is similar to kbuild-2.5 "make installable" a name 
> that I dislike. Obviously "make kernel" requires support for selecting
> the appropriate binary utilising make *config.

Well, let's say I agree that the kind of semantics change regarding 
building modules at the same time isn't the nicest. So I propose the 
following:

make bzImage -> compile built-in, build bzImage
make modules -> compile modules
make bzImage modules -> build bzImage + modules (I found a way to do so
                        without having to descend twice)

make,
make all     -> compile vmlinux + modules as a general default,
                on i386 build bzImage + modules.
	        (other archs can change the behavior as they wish)

> >        If I want to the kernel to build to continue even when a module
> > fails to compile, I should be able to do that by just using "-k".  Not
> > being able to build include/linux/modversions.h prevents me from doing
> > that.
> >From a conceptual point I disagree here. I would like make to
> avoid completion in case an error is flagged.
> My prediction is that the new behaviour may result in more errors being
> corrected, due to the incentitive to do it. Today you ignore it
> and hardly cannot spot it in all the noise generated during the build
> process.

Let me second this. In particular, there is no way to reliably generate
module versions when the affected files cannot even be preprocessed. So 
it's the right thing to error out. If someone dislikes that, they can
still turn MODVERSIONS of and proceed in their broken world.

> By the way - anyone having feedback on the "make KBUILD_VERBOSE=0"
> mode. Why not make it default?

I fear there won't be feedback unless it is made default, since nobody 
even notices otherwise.

Anyway, before making it default, I'm planning to call gcc from the 
topdir, so error/warning messages get prefixed with the complete
path to the source, which will allow a smart editor to automatically
jump to the place where the message occured.

(Currently it works due to make's 'Entering directory [...]' messages,
 which are disabled in quiet mode)

--Kai

Re: Various kbuild problems in 2.5.22

[Sam Ravnborg]

On Tue, Jun 18, 2002 at 02:28:02PM -0500, Kai Germaschewski wrote:

> Nitpick: 
> [kai@chaos linux-2.5.make]$ make bzImage && ls bzImage
> [..]
> ls: bzImage: No such file or directory
> 
> So you that bzImage isn't a real target (arch/i386/boot/bzImage would be).
Which actually annoy me a lot!
Maybe that just me, but I cannot see why I have to go down deep
in the architecture specific directory to locate the kernel.
But I have no brilliant idea how to solve it right away.

> make,
> make all     -> compile vmlinux + modules as a general default,
>                 on i386 build bzImage + modules.
> 	        (other archs can change the behavior as they wish)
I would like an easy way to do a full build, including supporting stuff
such as documentation. Think about doing regression on new kernels?

Today you have to specify a lot of different targets to accomplish
this - something that many people simply do not do.
Make it easy, and people will do it - and see more errors fixed earlier.

allyes, allmodules, allno were introduced for regression purpose.
And make all gives me the association that ALL is build, not only
the core kernel. Therefore I object against the make all semantic
that you suggest.
Better reserve that to a full build.

Furthermore I would advocate for a new target [yes one more!].

"make help"
make help shall generate a short list of available targets, something like:

<<<<<<<<<<<<
>make help
Configuration related targets:
  oldconfig	- bla bla
  menuconfig	- bla bla
  config	- bla bla
  configallyes  - ....
  configallno	- ....
Other generic targets:
  vmlinux	- The generic binary kernel
* modules	- Build all modules
# pdfdocs	- documentation
  all		- Build all targets marked with * and #
Architecture specific targets for current architecture (I386)
* bzImage	- Default compressed kernel (arch/i386/boot/bzImage)
  zImage	- Another target (arch/i386/boot/zImage)

Executing "make" or "make kernel" builds targets marked with '*'
<<<<<<<<<<

The architecture specific stuff should obviously be located in the
corresponding architecture specific makefiles and triggered by archhelp.

Comments?
I would be happy to do a patch if people support this.

	Sam

Various kbuild problems in 2.5.22

[Adam J. Richter]

	I would like to note the following problems with the
kernel build process in 2.5.22, after applying the patch that
Kai Germaschewski posted that enabled modversions to work again.
All but the first one are spurious dependencies.


	1. doing make modversions twice results in bad .ver files, with
lines like:

#define __ver_pcmcia_get_mem_page_Rsmp_3d2ded54 smp_ba03375b
#define pcmcia_get_mem_page_Rsmp_3d2ded54       _set_ver(pcmcia_get_mem_page_Rsmp_3d2ded54)

	...which should be:

#define __ver_pcmcia_get_mem_page smp_ba03375b
#define pcmcia_get_mem_page       _set_ver(pcmcia_get_mem_page)

	2. "make bzImage" does not build a bzImage if any module fails
to compile.  Really, it should not attempt to buidl modules or even
descend into directories that contain only modules.  To build a bzImage,
I have to edit the Makefile and comment out "BUILD_MODULES:=1".

	3. make include/linux/modversios.h aborts if any .c file has
a #error or #include's a .h that is not present (for example, because
the .h is built by the process, as is the case with one scsi driver).

	4. "make -k modules" will not build perfectly buildable modules
in a directory that has a subdirectory where a compile error occurs.

	All of this used to work a couple of kernel versions ago.

Adam J. Richter     __     ______________   575 Oroville Road
adam@yggdrasil.com     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America
                         "Free Software For The Rest Of Us."

Re: Various kbuild problems in 2.5.22

[Kai Germaschewski]

On Tue, 18 Jun 2002, Adam J. Richter wrote:

> 	I would like to note the following problems with the
> kernel build process in 2.5.22, after applying the patch that
> Kai Germaschewski posted that enabled modversions to work again.
> All but the first one are spurious dependencies.

> #define __ver_pcmcia_get_mem_page_Rsmp_3d2ded54 smp_ba03375b
> #define pcmcia_get_mem_page_Rsmp_3d2ded54       _set_ver(pcmcia_get_mem_page_Rsmp_3d2ded54)

Yes, the fix I posted was not complete. I submitted a corrected one 
already.

> 	2. "make bzImage" does not build a bzImage if any module fails
> to compile.  Really, it should not attempt to buidl modules or even
> descend into directories that contain only modules.  To build a bzImage,
> I have to edit the Makefile and comment out "BUILD_MODULES:=1".

That's intentional. If you don't want to build modules, use "make 
KBUILD_MODULES= bzImage". Or you can just ignore errors using "make -k".


> 	3. make include/linux/modversios.h aborts if any .c file has
> a #error or #include's a .h that is not present (for example, because
> the .h is built by the process, as is the case with one scsi driver).

The fact that it aborts is intentional. That it doesn't build the .h in 
that case is a bug. Which driver is it?

> 	4. "make -k modules" will not build perfectly buildable modules
> in a directory that has a subdirectory where a compile error occurs.

Well, I can fix that, I'll look into it.

--Kai