[PATCH 0/2] kbuild updates

[PATCH 0/2] kbuild updates

[Sam Ravnborg]

Hi Andrew, Linus.

Here follows two kbuild patches.

1) Add generic infrastructure for creating kernel packages.
   This moves make rpm support to scripts/package
   and add the deb-pkg target.
   The infrastructure were added because there is requests
   to add .tar.gz, .tar.gz2 support as well, and the functionality
   really did not belong in the top-level makefile.
   The implementation is simplified based on feedback on first
   patch, so one just have to set KBUILD_IMAGE in arch Makefiel
   to say what kernel image to include in the package.

2) Improved support for external modules.
   It has been debated what to name the symlink in /lib/modules/`uname -r`
   and where it should point.
   Now that there is a possibility to build the kernel with a separate output
   directory, there is a need to utilise this in the install.
   From now on build will point to the output directory, and source will point
   to the kernel source.
   A small MAkefile is created in the output directory allowing external modules
   to continue to build independent of the kernel being built with separate
   output directory, or with ouput and source mixed.

   If the kernel is build with source and output mixed there is no change
   in behaviour.
   If the kernel is build with separate source and output directories,
   all external modules that has not yet picked up on using the kbuild
   infrastructure will fail...
   No effort whatsoever will be done to keep external modules working if they
   do not use the kbuild infrastructure. There is no reason not to do so.

	Sam

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Sun, Jun 20, 2004 at 11:19:06PM +0200, Sam Ravnborg wrote:
> Hi Andrew, Linus.
> 
> Here follows two kbuild patches.

Available also at bkbits:
bk pull bk://linux-sam.bkbits.net/kbuild

	Sam

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

On Sun, 2004-06-20 at 23:19, Sam Ravnborg wrote:

Hiya,

> 2) Improved support for external modules.
>    It has been debated what to name the symlink in /lib/modules/`uname -r`
>    and where it should point.
>    Now that there is a possibility to build the kernel with a separate output
>    directory, there is a need to utilise this in the install.
>    From now on build will point to the output directory, and source will point
>    to the kernel source.

I know Sam's mta blocks my mail at least (lame isp), but for the rest,
please reconsider using this.  Many external modules, libs, etc use
/lib/modules/`uname -r`/build to locate the _source_, and this will
break them all.

Once again I do not argue the logic behind this, but please then rather
do it in 2.7, or just keep 'build', and make the output one 'output' or
'object' or something.

>  No effort whatsoever will be done to keep external modules working if
>  they do not use the kbuild infrastructure. There is no reason not to
>  do so.

Given, but to 'use' the kbuild infrastructure, you must still call it
via:

  make -C _path_to_sources M=`pwd`

and any external project that at least tries to automate things a bit
(because some things are not always distributed with vendor distro,
 or updated regularly by vendor distro, and most users know at least
 how to do 'make && make install') will break.

Lastly, if the 'build'/whatever symlinks is not an 'kbuild'
infrastructure (as Sam want to make it, and thus base his reasoning why
it is Ok to build it) for finding the source of the current running
kernel, then what is, why have it in the first place?


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]


> Given, but to 'use' the kbuild infrastructure, you must still call it
> via:
> 
>   make -C _path_to_sources M=`pwd`

I see no problem with requiring this though; requiring a correct
makefile is perfectly fine with me, and this is the only and documented
way for 2.6 already.
(And it's also the only way to build modules against Fedora Core 2
kernels by the way)

That's not my reservation to this approach.



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1108 bytes --]

On Sun, 2004-06-20 at 23:42, Arjan van de Ven wrote:
> > Given, but to 'use' the kbuild infrastructure, you must still call it
> > via:
> > 
> >   make -C _path_to_sources M=`pwd`
> 
> I see no problem with requiring this though; requiring a correct
> makefile is perfectly fine with me, and this is the only and documented
> way for 2.6 already.
> (And it's also the only way to build modules against Fedora Core 2
> kernels by the way)
> 

I did not mean I have a problem with that.  Say you take svgalib, and
you want the build system to automatically compile the kernel module,
you might do something like:

---
build_2_6_module:
	@make -C /lib/modules/`uname -r`/build M=`PWD`
---

will break with proposed patch ...

And the point I wanted to make was that AFIAK
'/lib/modules/`uname -r`/build' is an interface to figure
out where the _sources_ for the current running kernel are
located.  And apparently I am not the only one, as most if
not all external projects use this as the way (or one of
them at least) to determine this ... 


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Sun, Jun 20, 2004 at 11:30:34PM +0200, Martin Schlemmer wrote:
> 
> I know Sam's mta blocks my mail at least (lame isp), but for the rest,
> please reconsider using this.
Hmm, got your mail.

> Many external modules, libs, etc use
> /lib/modules/`uname -r`/build to locate the _source_, and this will
> break them all.
Examples please. What I have seen so far is modules that was not
adapted to use kbuild when being build.
If they fail to do so they are inherently broken.

If I get just one good example I will go for the object directory, but
what I have seen so far is whining - no examples.

	Sam

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1082 bytes --]

On Mon, 2004-06-21 at 00:03, Sam Ravnborg wrote:
> On Sun, Jun 20, 2004 at 11:30:34PM +0200, Martin Schlemmer wrote:
> > 
> > I know Sam's mta blocks my mail at least (lame isp), but for the rest,
> > please reconsider using this.
> Hmm, got your mail.
> 
> > Many external modules, libs, etc use
> > /lib/modules/`uname -r`/build to locate the _source_, and this will
> > break them all.
> Examples please. What I have seen so far is modules that was not
> adapted to use kbuild when being build.
> If they fail to do so they are inherently broken.
> 

Well, glibc use it for instance as an fall-through if you do not specify
it via ./configure arguments, or environment (yes, glibc should not use
it, etc, etc, no flames please =).  So as well does alsa-driver,
nvidia's drivers (gah, puke, yes, its got some binary-only stuff in
there ;), ati's drivers and a lot of other stuff (if you really need
them all I can try to find time to look for more).

I am not sure about ati's drivers and alsa, but nvidia uses kbuild.


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Mon, Jun 21, 2004 at 12:03:19AM +0200, Sam Ravnborg wrote:
> If I get just one good example I will go for the object directory, but
> what I have seen so far is whining - no examples.

Now I recall why I did not like the object directory.
I will break all modules using the kbuild infrastructure!

Why, because there is no way the to find the output directory except
specifying both directories.
One could do:
make -C /lib/modules/`uname -r`/source O=/lib/modules/`uname -r`/build M=`pwd`

So the currect choice is:
1) Break modules that actually dive into the src, grepping, including or whatever
2) Break all modules using kbuild infrastructure, including the above ones

I go for 1), introducing minimal breakage.

And please keep in mind. The breakage wil _only_ be visible when kernels are
shipped with separate output directory.
If kernels are shipped with no output files at all then one can just
compile the kernel. Seems to be the Fedora way. No breakage happens.

If kernels are shipped with mixed source and output then no breakage happens.

If kernels are shipped with separate source and output then we better break
as few modules as possible. And the introduced change actually minimize breakage.

So the patch will not change.

	Sam

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]

On Mon, 2004-06-21 at 00:18, Sam Ravnborg wrote:
> On Mon, Jun 21, 2004 at 12:03:19AM +0200, Sam Ravnborg wrote:
> > If I get just one good example I will go for the object directory, but
> > what I have seen so far is whining - no examples.
> 
> Now I recall why I did not like the object directory.
> I will break all modules using the kbuild infrastructure!
> 

Below do not really explain this - care to be more detailed?

> Why, because there is no way the to find the output directory except
> specifying both directories.
> One could do:
> make -C /lib/modules/`uname -r`/source O=/lib/modules/`uname -r`/build M=`pwd`
> 

Huh?  Explain to me how else you would do builds that have separate
output directory?  And what is the difference from above to:

make -C /lib/modules/`uname -r`/build O=/lib/modules/`uname -r`/object M=`pwd`

except that you will _not_ cause existing stuff to break?

> So the currect choice is:
> 1) Break modules that actually dive into the src, grepping, including or whatever
> 2) Break all modules using kbuild infrastructure, including the above ones
> 
> I go for 1), introducing minimal breakage.
> 
> And please keep in mind. The breakage wil _only_ be visible when kernels are
> shipped with separate output directory.

How is that?  In both cases the 'build' symlink do not point to the
source anymore.

> If kernels are shipped with no output files at all then one can just
> compile the kernel. Seems to be the Fedora way. No breakage happens.
> 
> If kernels are shipped with mixed source and output then no breakage happens.
> 
> If kernels are shipped with separate source and output then we better break
> as few modules as possible. And the introduced change actually minimize breakage.
> 
> So the patch will not change.
> 
> 	Sam
-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Andreas Gruenbacher]

On Sunday 20 June 2004 23:52, Martin Schlemmer wrote:
> On Sun, 2004-06-20 at 23:42, Arjan van de Ven wrote:
> > > Given, but to 'use' the kbuild infrastructure, you must still call it
> > > via:
> > >
> > >   make -C _path_to_sources M=`pwd`
> >
> > I see no problem with requiring this though; requiring a correct
> > makefile is perfectly fine with me, and this is the only and documented
> > way for 2.6 already.
> > (And it's also the only way to build modules against Fedora Core 2
> > kernels by the way)
>
> I did not mean I have a problem with that.  Say you take svgalib, and
> you want the build system to automatically compile the kernel module,
> you might do something like:
>
> ---
> build_2_6_module:
> 	@make -C /lib/modules/`uname -r`/build M=`PWD`
> ---
>
> will break with proposed patch ...

No it won't.

You always need to figure out $(objtree) to build external modules, with or 
without a separate output directory. Many modules don't need to know 
$(srctree) explicitly at all.

In case you want to do something depending on the sources/confguration, there 
are two ways:
  - follow the new source symlink,
  - let kbuild take you to $(srctree): When the makefile in the M directory
    is included, the current working directory is $(srctree). besides, all the
    usual variables like $(srctree), $(objtree), CONFIG_* variables, etc. are
    all available. That's a good time to check for features, etc.

> And the point I wanted to make was that AFIAK
> '/lib/modules/`uname -r`/build' is an interface to figure
> out where the _sources_ for the current running kernel are
> located.

That's a misconception. At the minimum, you want to be able to build the 
module. Directly messing with the sources is usually wrong. I know external 
module authors like to do that nevertheless; in a few cases it's actually 
useful. Most of the time it really is not. Most external modules have totally 
braindead/broken makefiles.

Regards,
-- 
Andreas Gruenbacher <agruen@suse.de>
SUSE Labs, SUSE LINUX AG

Re: [PATCH 0/2] kbuild updates

[Alistair John Strachan]

[snipped a few CC addresses]

On Sunday 20 June 2004 23:16, Martin Schlemmer wrote:
> On Mon, 2004-06-21 at 00:03, Sam Ravnborg wrote:
> > On Sun, Jun 20, 2004 at 11:30:34PM +0200, Martin Schlemmer wrote:
> > > I know Sam's mta blocks my mail at least (lame isp), but for the rest,
> > > please reconsider using this.
> >
> > Hmm, got your mail.
> >
> > > Many external modules, libs, etc use
> > > /lib/modules/`uname -r`/build to locate the _source_, and this will
> > > break them all.
> >
> > Examples please. What I have seen so far is modules that was not
> > adapted to use kbuild when being build.
> > If they fail to do so they are inherently broken.
>
> Well, glibc use it for instance as an fall-through if you do not specify
> it via ./configure arguments, or environment (yes, glibc should not use
> it, etc, etc, no flames please =).  So as well does alsa-driver,
> nvidia's drivers (gah, puke, yes, its got some binary-only stuff in
> there ;), ati's drivers and a lot of other stuff (if you really need
> them all I can try to find time to look for more).
>
> I am not sure about ati's drivers and alsa, but nvidia uses kbuild.
>
>
> Thanks,

Sam's point is that unless you ask KBUILD to put the kernel build in a 
separate directory to its sources (this is not the default 
behaviour), /lib/modules/`uname -r`/build will still point to the mixture of 
source and build data, therefore no breakage will occur.

I understand Sam's reasoning and I believe it is sensible to have the source 
and build output in separate directories within /lib/modules/`uname -r`. The 
drivers in question can easily be updated to support the exceptional case 
whereby users build kernels in a different directory to the source.

Sam, maybe if there was a way to easily detect whether a kernel had been build 
with or without a different output directory, it would be easier to have 
vendors take this change on board. For example, I imagine in the typical case 
whereby no change in build directory is made, you will have something like 
this:

/lib/modules/2.6.7/build -> /home/alistair/linux-2.6
/lib/modules/2.6.7/source -> /home/alistair/linux-2.6

Whereas when O is given, it will instead be like this:

/lib/modules/2.6.7/build -> /home/alistair/my-dir
/lib/modules/2.6.7/source -> /home/alistair/linux-2.6

I presume that checking for the existence of /lib/modules/`uname -r`/source 
will be enough.

#
# where's the kernel source?
#

if [ -d /lib/modules/`uname -r`/source ]; then
	# 2.6.8 and newer
	KERNDIR="/lib/modules/`uname -r`/source"
else
	# pre 2.6.8 kernels
	KERNDIR="/lib/modules/`uname -r`/build"
fi

Yeah?

-- 
Cheers,
Alistair.

personal:   alistair()devzero!co!uk
university: s0348365()sms!ed!ac!uk
student:    CS/AI Undergraduate
contact:    1F2 55 South Clerk Street,
            Edinburgh. EH8 9PP.

Re: [PATCH 0/2] kbuild updates

[Pascal Schmidt]

On Mon, 21 Jun 2004 00:00:12 +0200, you wrote in linux.kernel:

> ---
> build_2_6_module:
> 	@make -C /lib/modules/`uname -r`/build M=3D`PWD`
> ---
>
> will break with proposed patch ...

Huh? Sam wrote there will be a Makefile in the output directory.

-- 
Ciao,
Pascal

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 3828 bytes --]

On Mon, 2004-06-21 at 00:26, Andreas Gruenbacher wrote:
> On Sunday 20 June 2004 23:52, Martin Schlemmer wrote:
> > On Sun, 2004-06-20 at 23:42, Arjan van de Ven wrote:
> > > > Given, but to 'use' the kbuild infrastructure, you must still call it
> > > > via:
> > > >
> > > >   make -C _path_to_sources M=`pwd`
> > >
> > > I see no problem with requiring this though; requiring a correct
> > > makefile is perfectly fine with me, and this is the only and documented
> > > way for 2.6 already.
> > > (And it's also the only way to build modules against Fedora Core 2
> > > kernels by the way)
> >
> > I did not mean I have a problem with that.  Say you take svgalib, and
> > you want the build system to automatically compile the kernel module,
> > you might do something like:
> >
> > ---
> > build_2_6_module:
> > 	@make -C /lib/modules/`uname -r`/build M=`PWD`
> > ---
> >
> > will break with proposed patch ...
> 
> No it won't.
> 
> You always need to figure out $(objtree) to build external modules, with or 
> without a separate output directory. Many modules don't need to know 
> $(srctree) explicitly at all.
> 
> In case you want to do something depending on the sources/confguration, there 
> are two ways:
>   - follow the new source symlink,
>   - let kbuild take you to $(srctree): When the makefile in the M directory
>     is included, the current working directory is $(srctree). besides, all the
>     usual variables like $(srctree), $(objtree), CONFIG_* variables, etc. are
>     all available. That's a good time to check for features, etc.
> 

But my original concern (that the only way to figure where the source
are for the running kernel will be broken) is still valid.  And to do
all that you mentioned above, you still need to figure out _where_ the
kernel source are ...

> > And the point I wanted to make was that AFIAK
> > '/lib/modules/`uname -r`/build' is an interface to figure
> > out where the _sources_ for the current running kernel are
> > located.
> 
> That's a misconception. At the minimum, you want to be able to build the 
> module. Directly messing with the sources is usually wrong. I know external 
> module authors like to do that nevertheless; in a few cases it's actually 
> useful. Most of the time it really is not. Most external modules have totally 
> braindead/broken makefiles.
> 

I never said anything about messing with the source.  But anything that
needs access to the running kernel's headers need to know where the
sources are - and that could have been figured out by looking at the
'build' symlink.

Say you maintain an opensource (just to throw out the 'its closed
source, so screw them' arguments) external module that supports both
2.4 and 2.6, and easy way to implement it is to have:

makefile - make will first look for this, and then process it.
           in here you check what kernel is running (via uname -r
           or whatever), and then create the proper 'Makefile'
           symlink, and then call make with arguments to properly
           handle the external module build process for that
           version kernel.

Makefile-pre_M_flag - 100% valid kbuild Makefile for kernels that
                      do not support M=

Makefile-post_M_flag - 100% valid kbuild Makefile for kernels
                       supporting M=

Makefile-2_4 - 100% valid kbuild/whatever Makefile for 2.4 kernels
               (Ok, I am not so sure about how 2.4 handles things
               these days anymore ...)

now the clueless user just calls 'make && make install' and it should
work perfectly.

Or you create an install.sh that does things properly, or whatever,
but point remains that you need to know where the source are ...


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 3283 bytes --]

On Mon, 2004-06-21 at 00:26, Alistair John Strachan wrote:
> [snipped a few CC addresses]
> 
> On Sunday 20 June 2004 23:16, Martin Schlemmer wrote:
> > On Mon, 2004-06-21 at 00:03, Sam Ravnborg wrote:
> > > On Sun, Jun 20, 2004 at 11:30:34PM +0200, Martin Schlemmer wrote:
> > > > I know Sam's mta blocks my mail at least (lame isp), but for the rest,
> > > > please reconsider using this.
> > >
> > > Hmm, got your mail.
> > >
> > > > Many external modules, libs, etc use
> > > > /lib/modules/`uname -r`/build to locate the _source_, and this will
> > > > break them all.
> > >
> > > Examples please. What I have seen so far is modules that was not
> > > adapted to use kbuild when being build.
> > > If they fail to do so they are inherently broken.
> >
> > Well, glibc use it for instance as an fall-through if you do not specify
> > it via ./configure arguments, or environment (yes, glibc should not use
> > it, etc, etc, no flames please =).  So as well does alsa-driver,
> > nvidia's drivers (gah, puke, yes, its got some binary-only stuff in
> > there ;), ati's drivers and a lot of other stuff (if you really need
> > them all I can try to find time to look for more).
> >
> > I am not sure about ati's drivers and alsa, but nvidia uses kbuild.
> >
> >
> > Thanks,
> 
> Sam's point is that unless you ask KBUILD to put the kernel build in a 
> separate directory to its sources (this is not the default 
> behaviour), /lib/modules/`uname -r`/build will still point to the mixture of 
> source and build data, therefore no breakage will occur.
> 
> I understand Sam's reasoning and I believe it is sensible to have the source 
> and build output in separate directories within /lib/modules/`uname -r`. The 
> drivers in question can easily be updated to support the exceptional case 
> whereby users build kernels in a different directory to the source.
> 
> Sam, maybe if there was a way to easily detect whether a kernel had been build 
> with or without a different output directory, it would be easier to have 
> vendors take this change on board. For example, I imagine in the typical case 
> whereby no change in build directory is made, you will have something like 
> this:
> 
> /lib/modules/2.6.7/build -> /home/alistair/linux-2.6
> /lib/modules/2.6.7/source -> /home/alistair/linux-2.6
> 
> Whereas when O is given, it will instead be like this:
> 
> /lib/modules/2.6.7/build -> /home/alistair/my-dir
> /lib/modules/2.6.7/source -> /home/alistair/linux-2.6
> 
> I presume that checking for the existence of /lib/modules/`uname -r`/source 
> will be enough.
> 
> #
> # where's the kernel source?
> #
> 
> if [ -d /lib/modules/`uname -r`/source ]; then
> 	# 2.6.8 and newer
> 	KERNDIR="/lib/modules/`uname -r`/source"
> else
> 	# pre 2.6.8 kernels
> 	KERNDIR="/lib/modules/`uname -r`/build"
> fi
> 
> Yeah?

Yes, as said before, I can understand the name change.  The point is
more that the 'build' symlink will change in behavior in certain
circumstances, and because many projects already support 2.6, and
make use of the 'build' symlink, they will break.

If it was however done in 2.7, things will get supported well before
2.8 is out, or in general use ...

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Andreas Gruenbacher]

[CC trimmed]

On Monday 21 June 2004 00:39, Martin Schlemmer wrote:
> On Mon, 2004-06-21 at 00:26, Andreas Gruenbacher wrote:
> > On Sunday 20 June 2004 23:52, Martin Schlemmer wrote:
> > > On Sun, 2004-06-20 at 23:42, Arjan van de Ven wrote:
> > > > > Given, but to 'use' the kbuild infrastructure, you must still call
> > > > > it via:
> > > > >
> > > > >   make -C _path_to_sources M=`pwd`
> > > >
> > > > I see no problem with requiring this though; requiring a correct
> > > > makefile is perfectly fine with me, and this is the only and
> > > > documented way for 2.6 already.
> > > > (And it's also the only way to build modules against Fedora Core 2
> > > > kernels by the way)
> > >
> > > I did not mean I have a problem with that.  Say you take svgalib, and
> > > you want the build system to automatically compile the kernel module,
> > > you might do something like:
> > >
> > > ---
> > > build_2_6_module:
> > > 	@make -C /lib/modules/`uname -r`/build M=`PWD`
> > > ---
> > >
> > > will break with proposed patch ...
> >
> > No it won't.
> >
> > You always need to figure out $(objtree) to build external modules, with
> > or without a separate output directory. Many modules don't need to know
> > $(srctree) explicitly at all.
> >
> > In case you want to do something depending on the sources/confguration,
> > there are two ways:
> >   - follow the new source symlink,
> >   - let kbuild take you to $(srctree): When the makefile in the M
> > directory is included, the current working directory is $(srctree).
> > besides, all the usual variables like $(srctree), $(objtree), CONFIG_*
> > variables, etc. are all available. That's a good time to check for
> > features, etc.
>
> But my original concern (that the only way to figure where the source
> are for the running kernel will be broken) is still valid.

User-space stuff that needs access to kernel headers at build time is a 
problem. But for those programs, depending on the running kernel instead of 
simply looking in /usr/src/linux is an even bigger problem: What guarantees 
that the first time the program is run, the kernel will still be the same? So 
depending on the running kernel is definitely wrong. Depending 
on /usr/src/linux is also wrong; ideally those programs should look 
in /usr/include/{linux,asm}. Unfortunately these headers are not always 
recent enough, and so recently added definitions may be missing.

> And to do 
> all that you mentioned above, you still need to figure out _where_ the
> kernel source are ...

No. Use ``make -C /lib/modules/$(uname -r)/build M=$(pwd)''
[or ``make -C /lib/modules/$(uname -r)/build modules SUBDIRS=$(pwd)'']
and in Makefile, use conditionals to adapt to the configuration. 
Documentation/kbuild/makefiles.txt is your friend. Use
``make -C /lib/modules/$(uname -r)/build modules_install M=$(pwd)''
to install the modules.
Caution: the last command is dangerous; it will wipe away all other modules 
with kernels that don't have M= support.

In case you really think you must circumvent kbuild, check if the source 
symlink exists, etc.

> > > And the point I wanted to make was that AFIAK
> > > '/lib/modules/`uname -r`/build' is an interface to figure
> > > out where the _sources_ for the current running kernel are
> > > located.
> >
> > That's a misconception. At the minimum, you want to be able to build the
> > module. Directly messing with the sources is usually wrong. I know
> > external module authors like to do that nevertheless; in a few cases it's
> > actually useful. Most of the time it really is not. Most external modules
> > have totally braindead/broken makefiles.
>
> I never said anything about messing with the source.  But anything that
> needs access to the running kernel's headers need to know where the
> sources are - and that could have been figured out by looking at the
> 'build' symlink.

See above.

> Say you maintain an opensource (just to throw out the 'its closed
> source, so screw them' arguments) external module that supports both
> 2.4 and 2.6, and easy way to implement it is to have:
>
> makefile - make will first look for this, and then process it.
>            in here you check what kernel is running (via uname -r
>            or whatever), and then create the proper 'Makefile'
>            symlink, and then call make with arguments to properly
>            handle the external module build process for that
>            version kernel.

And what prevents you from doing exactly that? The only question is how to 
invoke kbuild, but that hasn't changed.

> Makefile-pre_M_flag - 100% valid kbuild Makefile for kernels that
>                       do not support M=
>
> Makefile-post_M_flag - 100% valid kbuild Makefile for kernels
>                        supporting M=

Right now I would collapse the pre/post Makefiles and use SUBDIRS instead. 
There is no easy and reliable test for M= support, and it's only cosmetic. 
Sam will probably disagree.

> Makefile-2_4 - 100% valid kbuild/whatever Makefile for 2.4 kernels
>                (Ok, I am not so sure about how 2.4 handles things
>                these days anymore ...)
>
> now the clueless user just calls 'make && make install' and it should
> work perfectly.
>
> Or you create an install.sh that does things properly, or whatever,
> but point remains that you need to know where the source are ...

Regards,
-- 
Andreas Gruenbacher <agruen@suse.de>
SUSE Labs, SUSE LINUX AG

Re: [PATCH 0/2] kbuild updates

[Hannu Savolainen]

On Mon, 21 Jun 2004, Sam Ravnborg wrote:

> > Many external modules, libs, etc use
> > /lib/modules/`uname -r`/build to locate the _source_, and this will
> > break them all.
> Examples please. What I have seen so far is modules that was not
> adapted to use kbuild when being build.
> If they fail to do so they are inherently broken.
For example our installation script expects that
/lib/modules/`uname -r`/build behaves like a symbolic link that points to
/usr/src/linux-`uname -r`. If /lib/modules/`uname -r`/build doesn't exist
then /usr/src/linux-`uname -r` will be attempted instead.

"make -C $KERNELDIR SUBDIRS=$MYDIR \
      CC=what_we_think_is_the_right_gcc_version" modules

This approach seems to work with the "stock" kernels as well as with all
the distribution kernels what we have tried so far (except the latest
kernel from SuSE that used different directory scheme).

It's relatively easy to use the dual directory scheme provided that the
object and source directories always have the same naming. However it's
possible that in the future there will be much more projects that need to
compile external modules. So it would be easier in general if the
(possible) dual directory scheme is "hidden" from the external projects.

It should be easy difficult to patch the Makefile in
/lib/modules/`uname -r`/build to locate the object directory without
requiring the caller to pass it as a parameter to the "make modules"
command.


There is one special problem that doesn't affect users who have compiled
their kernel themselves. Many distributions don't include gcc, make and
related tools in the default installation so large amount of customer
sites don't have them installed. In addition different gcc version may
have to be used to compile the kernel than the one that is the default gcc
command in the system. So before anything can be compiled the installation
scripts have to check also if the required tools are installed. If they
are not available then the customer needs to be instructed to install
them. Because the installation procedure depents on the distribution it's
very difficult to help the customer in any way. For this reason it would
be better if the re is some kind of unbrella script for building the
module. In the vanilla kernels it simply executes the right make command.
If the distribution vendors see it necessary they can improve the script
and add features like automatic installation of the required modules or
whatever else.

Does something like the following sound good?

sh /lib/modules/`uname -r`/build/make_module $MYSUBDIR CC=$CC

$MYSUBDIR is the absolute path to the directory where the module sources
are located.

$CC is optional parameter that the calling module can use if it thinks it
knows the right gcc version. If nothing is given then plain gcc is
assumed. If the script knows the right compiler then it overrides this
parameter.

In the minimalistic form the script is just

#!/bin/sh
make -C /lib/modules/`uname -r`/build SUBDIRS="$1" $2 modules

Best regards,

Hannu
-----
Hannu Savolainen (hannu@opensound.com)
http://www.opensound.com (Open Sound System (OSS))
http://www.compusonic.fi (Finnish OSS pages)
OH2GLH QTH: Karkkila, Finland LOC: KP20CM

Re: [PATCH 0/2] kbuild updates

[Andreas Gruenbacher]

On Monday 21 June 2004 02:29, Hannu Savolainen wrote:
> On Mon, 21 Jun 2004, Sam Ravnborg wrote:
> > > Many external modules, libs, etc use
> > > /lib/modules/`uname -r`/build to locate the _source_, and this will
> > > break them all.
> >
> > Examples please. What I have seen so far is modules that was not
> > adapted to use kbuild when being build.
> > If they fail to do so they are inherently broken.
>
> For example our installation script expects that
> /lib/modules/`uname -r`/build behaves like a symbolic link that points to
> /usr/src/linux-`uname -r`. If /lib/modules/`uname -r`/build doesn't exist
> then /usr/src/linux-`uname -r` will be attempted instead.
>
> "make -C $KERNELDIR SUBDIRS=$MYDIR \
>       CC=what_we_think_is_the_right_gcc_version" modules
>
> This approach seems to work with the "stock" kernels as well as with all
> the distribution kernels what we have tried so far (except the latest
> kernel from SuSE that used different directory scheme).

Did you actually try the above command with a SUSE kernel? I don't think so; 
one of the earlier postings in the other thread spiced with strong words 
tried to do other unusual things.

> It's relatively easy to use the dual directory scheme provided that the
> object and source directories always have the same naming. However it's
> possible that in the future there will be much more projects that need to
> compile external modules. So it would be easier in general if the
> (possible) dual directory scheme is "hidden" from the external projects.

This is already in place with Sam's patches.

> It should be easy difficult to patch the Makefile in
> /lib/modules/`uname -r`/build to locate the object directory without
> requiring the caller to pass it as a parameter to the "make modules"
> command.

There is one source tree for N object trees in general. Going from the N side 
to the 1 side makes a lot more sense than going the other way around.

> There is one special problem that doesn't affect users who have compiled
> their kernel themselves. Many distributions don't include gcc, make and
> related tools in the default installation so large amount of customer
> sites don't have them installed.

You will invoke make directly, so you should check if make and the kernel 
sources for the running kernel are available.

> In addition different gcc version may 
> have to be used to compile the kernel than the one that is the default gcc
> command in the system.

Then this should be defined in the Makefile in the kernel sources. Fixing 
problems for people who build their own kernels from vanilla sources without 
paying attention to such details will take you nowhere, it will only mess 
things up even worse.

> So before anything can be compiled the installation 
> scripts have to check also if the required tools are installed. If they
> are not available then the customer needs to be instructed to install
> them. Because the installation procedure depents on the distribution it's
> very difficult to help the customer in any way. For this reason it would
> be better if the re is some kind of unbrella script for building the
> module. In the vanilla kernels it simply executes the right make command.
> If the distribution vendors see it necessary they can improve the script
> and add features like automatic installation of the required modules or
> whatever else.

There could be a few basic checks in kbuild whether or not the required tools 
are available. Telling users who compile kernel modules on their own how to 
install packages sounds a bit much asked for my taste.

Regards,
-- 
Andreas Gruenbacher <agruen@suse.de>
SUSE Labs, SUSE LINUX AG

Re: [PATCH 0/2] kbuild updates

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 274 bytes --]

On Mon, Jun 21, 2004 at 03:29:24AM +0300, Hannu Savolainen wrote:
> Does something like the following sound good?
> 
> sh /lib/modules/`uname -r`/build/make_module $MYSUBDIR CC=$CC


no you shold not override the compiler; the makefiles should do that instead
automatically

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Hannu Savolainen]

On Mon, 21 Jun 2004, Arjan van de Ven wrote:

> On Mon, Jun 21, 2004 at 03:29:24AM +0300, Hannu Savolainen wrote:
> > Does something like the following sound good?
> >
> > sh /lib/modules/`uname -r`/build/make_module $MYSUBDIR CC=$CC
>
>
> no you shold not override the compiler; the makefiles should do that instead
> automatically
Right. That would be better.

Best regards,

Hannu
-----
Hannu Savolainen (hannu@opensound.com)
http://www.opensound.com (Open Sound System (OSS))
http://www.compusonic.fi (Finnish OSS pages)
OH2GLH QTH: Karkkila, Finland LOC: KP20CM

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1904 bytes --]

On Tue, 2004-06-22 at 00:33, Sam Ravnborg wrote:
> On Sun, Jun 20, 2004 at 11:26:54PM +0100, Alistair John Strachan wrote:

> > Sam's point is that unless you ask KBUILD to put the kernel build in a 
> > separate directory to its sources (this is not the default 
> > behaviour), /lib/modules/`uname -r`/build will still point to the mixture of 
> > source and build data, therefore no breakage will occur.
> 
> Correct!
> 

> > Sam, maybe if there was a way to easily detect whether a kernel had been build 
> > with or without a different output directory, it would be easier to have 
> > vendors take this change on board. For example, I imagine in the typical case 
> > whereby no change in build directory is made, you will have something like 
> > this:
> > 
> > /lib/modules/2.6.7/build -> /home/alistair/linux-2.6
> > /lib/modules/2.6.7/source -> /home/alistair/linux-2.6
> > 
> > Whereas when O is given, it will instead be like this:
> > 
> > /lib/modules/2.6.7/build -> /home/alistair/my-dir
> > /lib/modules/2.6.7/source -> /home/alistair/linux-2.6
> > 
> > I presume that checking for the existence of /lib/modules/`uname -r`/source 
> > will be enough.
> > 
> > #
> > # where's the kernel source?
> > #
> > 
> > if [ -d /lib/modules/`uname -r`/source ]; then
> > 	# 2.6.8 and newer
> > 	KERNDIR="/lib/modules/`uname -r`/source"
> > else
> > 	# pre 2.6.8 kernels
> > 	KERNDIR="/lib/modules/`uname -r`/build"
> > fi
> 
> Look ok.
> 

Yeah, understood, and already seen as the solution.

I guess I just wanted to try and prevent breakage to _existing_
projects, and the need for another hack to try and figure out
how to build for 2.6 (2.6 has been a bumpy ride in external
module building regards =).

But I guess it will not happen, so rather now then much later
when 2.6 is in much wider use ... :-)


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Mon, Jun 21, 2004 at 01:51:43AM +0200, Andreas Gruenbacher wrote:
> > But my original concern (that the only way to figure where the source
> > are for the running kernel will be broken) is still valid.
> 
> User-space stuff that needs access to kernel headers at build time is a 
> problem. But for those programs, depending on the running kernel instead of 
> simply looking in /usr/src/linux is an even bigger problem: What guarantees 
> that the first time the program is run, the kernel will still be the same? So 
> depending on the running kernel is definitely wrong. Depending 
> on /usr/src/linux is also wrong; ideally those programs should look 
> in /usr/include/{linux,asm}. Unfortunately these headers are not always 
> recent enough, and so recently added definitions may be missing.

But Martin has a point here.
How to figure out for example the number of arguments to remap_page_range.
One could do some grepping in the mm.h file, or one could try to compile
a minimal module calling this function.
If we go for the simple version by grepping we need to figure out where
to find the source. In the past this was simple - just follow the
build symlink. But now kernels may be shipped with separate source
and output directory exposing the weakness of this method.
A much more reliable way is to build a simple module.
If the module build succeeds that specific version
of remap_page_range is OK.

nvidia does something similar, but they take the false assumption
that the running kernel is always the one being build for.
So they call gcc direct.

Other modules uses the grep method - which will fail when the kernel
is build with separate output and source directories.

> 
> > Makefile-pre_M_flag - 100% valid kbuild Makefile for kernels that
> >                       do not support M=
> >
> > Makefile-post_M_flag - 100% valid kbuild Makefile for kernels
> >                        supporting M=
> 
> Right now I would collapse the pre/post Makefiles and use SUBDIRS instead. 
> There is no easy and reliable test for M= support, and it's only cosmetic. 
> Sam will probably disagree.

SUBDIRS were kept for backward compatibility - and I realise it will stay
for a long time. The implementation were kept straightforward so no problem.

	Sam

Re: [PATCH 0/2] kbuild updates

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 2460 bytes --]

On Tue, 2004-06-22 at 00:31, Sam Ravnborg wrote:
> On Mon, Jun 21, 2004 at 01:51:43AM +0200, Andreas Gruenbacher wrote:
> > > But my original concern (that the only way to figure where the source
> > > are for the running kernel will be broken) is still valid.
> > 
> > User-space stuff that needs access to kernel headers at build time is a 
> > problem. But for those programs, depending on the running kernel instead of 
> > simply looking in /usr/src/linux is an even bigger problem: What guarantees 
> > that the first time the program is run, the kernel will still be the same? So 
> > depending on the running kernel is definitely wrong. Depending 
> > on /usr/src/linux is also wrong; ideally those programs should look 
> > in /usr/include/{linux,asm}. Unfortunately these headers are not always 
> > recent enough, and so recently added definitions may be missing.
> 
> But Martin has a point here.
> How to figure out for example the number of arguments to remap_page_range.
> One could do some grepping in the mm.h file, or one could try to compile
> a minimal module calling this function.
> If we go for the simple version by grepping we need to figure out where
> to find the source. In the past this was simple - just follow the
> build symlink. But now kernels may be shipped with separate source
> and output directory exposing the weakness of this method.
> A much more reliable way is to build a simple module.
> If the module build succeeds that specific version
> of remap_page_range is OK.
> 
> nvidia does something similar, but they take the false assumption
> that the running kernel is always the one being build for.
> So they call gcc direct.
> 
> Other modules uses the grep method - which will fail when the kernel
> is build with separate output and source directories.
> 

Might it help if there was some silly 'external module example toolkit'
or whatever, that shows how to figure out with kbuild how to determine
srcdir, figure out how many args remap_page_range takes, etc?

I will admit that most fixes I do to external modules is maybe not
really because their build systems is 'buggy' as such, but more because
of short-sightedness and lack of understanding kbuild.  So above should
help.

Yeah, another project to maintain (in kernel if possible?).  I am
willing to try at some of the tests, but my docs skills is no where
good ... :/


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Sun, Jun 20, 2004 at 11:26:54PM +0100, Alistair John Strachan wrote:
> [snipped a few CC addresses]
> 
> On Sunday 20 June 2004 23:16, Martin Schlemmer wrote:
> > On Mon, 2004-06-21 at 00:03, Sam Ravnborg wrote:
> > > On Sun, Jun 20, 2004 at 11:30:34PM +0200, Martin Schlemmer wrote:
> > > > I know Sam's mta blocks my mail at least (lame isp), but for the rest,
> > > > please reconsider using this.
> > >
> > > Hmm, got your mail.
> > >
> > > > Many external modules, libs, etc use
> > > > /lib/modules/`uname -r`/build to locate the _source_, and this will
> > > > break them all.
> > >
> > > Examples please. What I have seen so far is modules that was not
> > > adapted to use kbuild when being build.
> > > If they fail to do so they are inherently broken.
> >
> > Well, glibc use it for instance as an fall-through if you do not specify
> > it via ./configure arguments, or environment (yes, glibc should not use
> > it, etc, etc, no flames please =).  So as well does alsa-driver,
> > nvidia's drivers (gah, puke, yes, its got some binary-only stuff in
> > there ;), ati's drivers and a lot of other stuff (if you really need
> > them all I can try to find time to look for more).
> >
> > I am not sure about ati's drivers and alsa, but nvidia uses kbuild.
> >
> >
> > Thanks,
> 
> Sam's point is that unless you ask KBUILD to put the kernel build in a 
> separate directory to its sources (this is not the default 
> behaviour), /lib/modules/`uname -r`/build will still point to the mixture of 
> source and build data, therefore no breakage will occur.

Correct!

> I understand Sam's reasoning and I believe it is sensible to have the source 
> and build output in separate directories within /lib/modules/`uname -r`. The 
> drivers in question can easily be updated to support the exceptional case 
> whereby users build kernels in a different directory to the source.
> 
> Sam, maybe if there was a way to easily detect whether a kernel had been build 
> with or without a different output directory, it would be easier to have 
> vendors take this change on board. For example, I imagine in the typical case 
> whereby no change in build directory is made, you will have something like 
> this:
> 
> /lib/modules/2.6.7/build -> /home/alistair/linux-2.6
> /lib/modules/2.6.7/source -> /home/alistair/linux-2.6
> 
> Whereas when O is given, it will instead be like this:
> 
> /lib/modules/2.6.7/build -> /home/alistair/my-dir
> /lib/modules/2.6.7/source -> /home/alistair/linux-2.6
> 
> I presume that checking for the existence of /lib/modules/`uname -r`/source 
> will be enough.
> 
> #
> # where's the kernel source?
> #
> 
> if [ -d /lib/modules/`uname -r`/source ]; then
> 	# 2.6.8 and newer
> 	KERNDIR="/lib/modules/`uname -r`/source"
> else
> 	# pre 2.6.8 kernels
> 	KERNDIR="/lib/modules/`uname -r`/build"
> fi

Look ok.

	Sam

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Mon, Jun 21, 2004 at 12:54:01AM +0200, Martin Schlemmer wrote:
> > 
> > #
> > # where's the kernel source?
> > #
> > 
> > if [ -d /lib/modules/`uname -r`/source ]; then
> > 	# 2.6.8 and newer
> > 	KERNDIR="/lib/modules/`uname -r`/source"
> > else
> > 	# pre 2.6.8 kernels
> > 	KERNDIR="/lib/modules/`uname -r`/build"
> > fi
> > 
> > Yeah?
> 
> Yes, as said before, I can understand the name change.  The point is
> more that the 'build' symlink will change in behavior in certain
> circumstances, and because many projects already support 2.6, and
> make use of the 'build' symlink, they will break.

But they were already broken.
And the patch actually helps here.

See the following table:



                                          Current kbuild                With patch
simple module, no grepping
kernel source + output mixed                  OK                           OK

simple module, no grepping
kernel source separate from output            FAIL *1                      OK


module grepping in src
kernel source + output mixed                  OK                           OK

module grepping in src
kernel source separate from output            FAIL *2                      FAIL *3




FAIL *1	Module cannot build because it fails to locate the compiled files used for kbuild.
        Also the kernel configuration is missing.

FAIL *2 Same as FAIL *1. Module cannot build because kbuild compiled files are missing and
        it cannot access kernel configuration.
        Also it fails to locate files when grepping.

FAIL *3 Grep will fail because it try to grep in files located under build/



The above table clearly shows that this patch fix building external modules in the
case where no grepping were preformed.
And the error were simplified in the last case.
So improvements all over.

What one should realise is that the patch makes it less painfull when kernels
are build using the separate output directory option.
And when that feature is not used - no change in behaviour occur.

	Sam
 

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Mon, Jun 21, 2004 at 12:25:18AM +0200, Martin Schlemmer wrote:
> On Mon, 2004-06-21 at 00:18, Sam Ravnborg wrote:
> > On Mon, Jun 21, 2004 at 12:03:19AM +0200, Sam Ravnborg wrote:
> > > If I get just one good example I will go for the object directory, but
> > > what I have seen so far is whining - no examples.
> > 
> > Now I recall why I did not like the object directory.
> > I will break all modules using the kbuild infrastructure!
> > 
> 
> Below do not really explain this - care to be more detailed?
> 
> > Why, because there is no way the to find the output directory except
> > specifying both directories.
> > One could do:
> > make -C /lib/modules/`uname -r`/source O=/lib/modules/`uname -r`/build M=`pwd`
> > 
> 
> Huh?  Explain to me how else you would do builds that have separate
> output directory?  And what is the difference from above to:
> 
> make -C /lib/modules/`uname -r`/build O=/lib/modules/`uname -r`/object M=`pwd`

Modules do not have O= specified.
And you did not address the issue with modules makefile grepping in src.

	Sam

Re: [PATCH 0/2] kbuild updates

[Andreas Gruenbacher]

On Tuesday 22 June 2004 00:31, Sam Ravnborg wrote:
> But Martin has a point here.

Yes.

> Other modules uses the grep method - which will fail when the kernel
> is build with separate output and source directories.

Is there anything fundamentally wrong with invoking the test script from 
within the module makefile, when really necessary? This doesn't work very 
well if you need the test results outside of the module build.

	----- Makefile -----
	test_result := $(shell ...)
	ifneq ($(test_result),)
	EXTRA_CFLAGS := -DSOMETHING
	endif
	...
	-------- 8< --------

Cheers,
-- 
Andreas Gruenbacher <agruen@suse.de>
SUSE Labs, SUSE LINUX AG

Re: [PATCH 0/2] kbuild updates

[Andreas Gruenbacher]

On Tuesday 22 June 2004 00:29, Martin Schlemmer wrote:
> I guess I just wanted to try and prevent breakage to _existing_
> projects, and the need for another hack to try and figure out
> how to build for 2.6 (2.6 has been a bumpy ride in external
> module building regards =).

We are *a lot* better now than we ever were before. Thanks, Sam.

Cheers,
-- 
Andreas Gruenbacher <agruen@suse.de>
SUSE Labs, SUSE LINUX AG

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Tue, Jun 22, 2004 at 12:50:09AM +0200, Andreas Gruenbacher wrote:
> On Tuesday 22 June 2004 00:31, Sam Ravnborg wrote:
> > But Martin has a point here.
> 
> Yes.
> 
> > Other modules uses the grep method - which will fail when the kernel
> > is build with separate output and source directories.
> 
> Is there anything fundamentally wrong with invoking the test script from 
> within the module makefile, when really necessary? This doesn't work very 
> well if you need the test results outside of the module build.
> 
> 	----- Makefile -----
> 	test_result := $(shell ...)
> 	ifneq ($(test_result),)
> 	EXTRA_CFLAGS := -DSOMETHING
> 	endif
> 	...
> 	-------- 8< --------

I like keeping the kbuild makefile clean as such - but otherwise not.
One has to secure that right CFLAGS etc is used though.

Also when make -j N is used one has to be very carefully with prerequisites - as always.

	Sam

Re: [PATCH 0/2] kbuild updates

[Peter Chubb]

>>>>> "Alistair" == Alistair John Strachan <s0348365@sms.ed.ac.uk> writes:


Alistair> Sam, maybe if there was a way to easily detect whether a
Alistair> kernel had been build with or without a different output
Alistair> directory, it would be easier to have vendors take this
Alistair> change on board. For example, I imagine in the typical case
Alistair> whereby no change in build directory is made, you will have
Alistair> something like this:


You can do this without any changes:

     if [ -d /lib/modules/`uname -r`/build/include2 ]; then
       kerndir=`cd  /lib/modules/`uname -r`/build/include2/asm; cd -P../..;/bin/pwd`
     else
       kerndir=`cd /lib/modules/`uname -r`/build; /bin/pwd`
    fi


But can the include2/asm symlink be made a relative one, please?  I frequently
build on one machine, then NFS-mount the build tree and run make
modules_install somewhere else; I always at present have to convert
that link to a relative symlink before doing so.

-- 
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
The technical we do immediately,  the political takes *forever*

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Tue, Jun 22, 2004 at 11:39:43AM +1000, Peter Chubb wrote:
> 
> But can the include2/asm symlink be made a relative one, please?  I frequently
> build on one machine, then NFS-mount the build tree and run make
> modules_install somewhere else; I always at present have to convert
> that link to a relative symlink before doing so.

Patch is welcome. I recall having trouble with it when introducing it. But that
can have been caused by other issues.

	Sam

Re: [PATCH 0/2] kbuild updates

[Jari Ruusu]

Sam Ravnborg wrote:
> On Mon, Jun 21, 2004 at 12:03:19AM +0200, Sam Ravnborg wrote:
> > If I get just one good example I will go for the object directory, but
> > what I have seen so far is whining - no examples.
> 
> Now I recall why I did not like the object directory.
> I will break all modules using the kbuild infrastructure!

No it does not. If 'export KBUILD_OUTPUT=/foo' command is used used before
kernel is built, it is not any more difficult to compile external modules
with that same env variable defined.

> Why, because there is no way the to find the output directory except
> specifying both directories.
> One could do:
> make -C /lib/modules/`uname -r`/source O=/lib/modules/`uname -r`/build M=`pwd`
> 
> So the currect choice is:
> 1) Break modules that actually dive into the src, grepping, including or whatever
> 2) Break all modules using kbuild infrastructure, including the above ones

or 3) Add the missing object symlink. It does not break anything.

If and when external module build scripts are updated to automatically
detect the /lib/modules/`uname -r`/object symlink, users can even unset the
KBUILD_OUTPUT= env variable and external modules will still compile ok.

> I go for 1), introducing minimal breakage.

You seem to be in some strange "I must destroy... I must destroy... I must
destroy..." mental state. There is a no-breakage alternative and you just
will not consider that at all.

> If kernels are shipped with separate source and output then we better break
> as few modules as possible. And the introduced change actually minimize breakage.

Nope. Solution #3 is the minimal breakage solution.

> So the patch will not change.

In which case Andrew and Linus should consider revoking your kbuild
maintainer status. I moved Andrew and Linus from CC list to TO list in hope
to get this small question answered:

Andrew, Linus,
How about choosing someone else less destructive to maintain kbuild?

> See the following table:
> 
>                                           Current kbuild                With patch
> simple module, no grepping
> kernel source + output mixed                  OK                           OK
> 
> simple module, no grepping
> kernel source separate from output            FAIL *1                      OK
> 
> module grepping in src
> kernel source + output mixed                  OK                           OK
> 
> module grepping in src
> kernel source separate from output            FAIL *2                      FAIL *3
> 
> FAIL *1 Module cannot build because it fails to locate the compiled files used for kbuild.
>         Also the kernel configuration is missing.

Same 'export KBUILD_OUTPUT=/foo' will work the same for kernel build and
external module build. Your "FAIL *1" is bogus.

> FAIL *2 Same as FAIL *1. Module cannot build because kbuild compiled files are missing and
>         it cannot access kernel configuration.
>         Also it fails to locate files when grepping.

Same 'export KBUILD_OUTPUT=/foo' will work the same for kernel build and
external module build. Your "FAIL *2" is also bogus.

> FAIL *3 Grep will fail because it try to grep in files located under build/

Yep.

-- 
Jari Ruusu  1024R/3A220F51 5B 4B F9 BB D3 3F 52 E9  DB 1D EB E3 24 0E A9 DD

Re: [PATCH 0/2] kbuild updates

[Andreas Gruenbacher]

[-- Attachment #1: Type: text/plain, Size: 1205 bytes --]

On Tuesday 22 June 2004 07:20, Sam Ravnborg wrote:
> On Tue, Jun 22, 2004 at 11:39:43AM +1000, Peter Chubb wrote:
> > But can the include2/asm symlink be made a relative one, please?  I
> > frequently build on one machine, then NFS-mount the build tree and run
> > make modules_install somewhere else; I always at present have to convert
> > that link to a relative symlink before doing so.
>
> Patch is welcome. I recall having trouble with it when introducing it. But
> that can have been caused by other issues.

The same issue exists with the KERNELSRC and KERNELOUTPUT paths in 
$(objtree)/Makefile: they would also better be relative to each other. In our 
case we have something like:

	KERNELSRC    := /usr/src/linux-2.6.5-7.79
	KERNELOUTPUT := /usr/src/linux-2.6.5-7.79-obj/i386/default

this would become:

	KERNELSRC    := ../../../linux-2.6.5-7.79
	KERNELOUTPUT := ../linux-2.6.5-7.79-obj/i386/default

I am currently regenerating $(objtree)/Makefile by hand by invoking mkmakefile 
with the appropriate parameters. The attached script could be used to 
automate this; it would work equally well for the asm symlink.

Regards,
-- 
Andreas Gruenbacher <agruen@suse.de>
SUSE Labs, SUSE LINUX AG

[-- Attachment #2: relpath --]
[-- Type: application/x-shellscript, Size: 580 bytes --]

Re: [PATCH 0/2] kbuild updates

[Andreas Gruenbacher]

On Tuesday 22 June 2004 07:29, Jari Ruusu wrote:
> Sam Ravnborg wrote:
> > On Mon, Jun 21, 2004 at 12:03:19AM +0200, Sam Ravnborg wrote:
> > > If I get just one good example I will go for the object directory, but
> > > what I have seen so far is whining - no examples.
> >
> > Now I recall why I did not like the object directory.
> > I will break all modules using the kbuild infrastructure!
>
> No it does not. If 'export KBUILD_OUTPUT=/foo' command is used before
> kernel is built, it is not any more difficult to compile external modules
> with that same env variable defined.

This clearly is not an option. We want the most trivial way of building 
external modules to continue working, no matter whether the kernel is using a 
separate output directory or not; with Sam's patch we get that:

	make -C /lib/modules/$(uname -r)/build M=$(pwd)

We also want to build modules for other configurations; this is as simple as 
passing another path in -C. For example, in the SUSE setup this would give 
you a module for an i386 bigsmp kernel:

	make -C /usr/scc/linux-obj/i386/bigsmp M=$(pwd)

The environment variable proposal is worthless: Where on earth should that 
environment variable come from?

> [cut the crap]
>
> In which case Andrew and Linus should consider revoking your kbuild
> maintainer status. I moved Andrew and Linus from CC list to TO list in hope
> to get this small question answered:
>
> Andrew, Linus,
> How about choosing someone else less destructive to maintain kbuild?

Jari, you have shown that you still didn't get it. Trying harder instead of 
badmouthing random people might help.

Regards,
-- 
Andreas Gruenbacher <agruen@suse.de>
SUSE Labs, SUSE LINUX AG

Re: [PATCH 0/2] kbuild updates

[Jari Ruusu]

Andreas Gruenbacher wrote:
> On Tuesday 22 June 2004 07:29, Jari Ruusu wrote:
> > Sam Ravnborg wrote:
> > > Now I recall why I did not like the object directory.
> > > I will break all modules using the kbuild infrastructure!
> >
> > No it does not. If 'export KBUILD_OUTPUT=/foo' command is used before
> > kernel is built, it is not any more difficult to compile external modules
> > with that same env variable defined.
> 
> This clearly is not an option. We want the most trivial way of building
> external modules to continue working, no matter whether the kernel is using a
> separate output directory or not; with Sam's patch we get that:
> 
>         make -C /lib/modules/$(uname -r)/build M=$(pwd)

Kernel already does that correctly, and follows KBUILD_OUTPUT env variable
that points to where user wants the object tree. No changes needed to
external module build scripts.

Maybe you SUSE guys just didn't realize that.
 
> We also want to build modules for other configurations; this is as simple as
> passing another path in -C. For example, in the SUSE setup this would give
> you a module for an i386 bigsmp kernel:
> 
>         make -C /usr/scc/linux-obj/i386/bigsmp M=$(pwd)

This is cool and desirable, but does not need or even justify
breaking/redirecting the 'build' symlink elsewhere.

> The environment variable proposal is worthless:

It is not a proposal. It has been in mainline since 2.5.x kernels and last
time I checked, it worked fine.

> Where on earth should that environment variable come from?

The person compiling kernel decides where he wants the object tree. He only
needs to set that once, before builing a kernel with separate object and
source trees. And same env variable works just fine when used with
externally compiled modules.

-- 
Jari Ruusu  1024R/3A220F51 5B 4B F9 BB D3 3F 52 E9  DB 1D EB E3 24 0E A9 DD

Re: [PATCH 0/2] kbuild updates

[Sam Ravnborg]

On Tue, Jun 22, 2004 at 08:29:55AM +0300, Jari Ruusu wrote:
> Sam Ravnborg wrote:
> > On Mon, Jun 21, 2004 at 12:03:19AM +0200, Sam Ravnborg wrote:
> > > If I get just one good example I will go for the object directory, but
> > > what I have seen so far is whining - no examples.
> > 
> > Now I recall why I did not like the object directory.
> > I will break all modules using the kbuild infrastructure!
> 
> No it does not. If 'export KBUILD_OUTPUT=/foo' command is used used before
> kernel is built, it is not any more difficult to compile external modules
> with that same env variable defined.
> 
> > Why, because there is no way the to find the output directory except
> > specifying both directories.
> > One could do:
> > make -C /lib/modules/`uname -r`/source O=/lib/modules/`uname -r`/build M=`pwd`
> > 
> > So the currect choice is:
> > 1) Break modules that actually dive into the src, grepping, including or whatever
> > 2) Break all modules using kbuild infrastructure, including the above ones
> 
> or 3) Add the missing object symlink. It does not break anything.

This is unfortunately not the case.
First a few facts.
The documented way to build external modules has for a while now been
to use the following command:

	make -C /lib/modules/`uname -r`/build SUBDIRS=`pwd` modules

Recently this was simplifed to

	make -C /lib/modules/`uname -r`/build M=`pwd`

As you pointed out in a previous post letting build point to the
output directory would break all existing drivers using the documented
approach, iff the kernel were compiled using a separate output directory.
So the breakage that was of a concern was how to build external
modules using the above command - but having the kernel build
using a separate output directory.
The was solved by adding a small makefile in the output directory.

So the situation with the patch posted is that all ordinary modules
compile independent of the kernel is compiled with separate output
directories or not.

The problem you point out is modules that to be backward compatible
needs access to the kernel source. This is solved in different ways:
a) Grepping the source direct
b) Compiling a small code fragment, and test if the .o file exists

Knowing your background you are most familiar with the a) approach.
Therefore I understand why you prefer to set an environment variable
and otherwise use your current makefile.
But doing so will break all 'ordinary' external modules.
The ordinary modules are adapted to use the above commands
to compile the module.
But if the build symlinks points to the source of the
kernel and the kernel uses separate directory for output files then
it is missing the possibility to:
- access .config
- use kbuild infrastructure
- include headers under asm symlink

All the abvoe are fatal.
The porposal to require the user to set KBUILD_OUTPUT before
compiling the module would fix this. But this is then a change
in the required command used to compile an 'ordinary' external module.
It would then look like:

	KBUILD_OUTPUT=/lib/modules/`uname -r`/object; make -C /lib/modules/`uname -r`/build M=`pwd`

This change is not acceptable.
The tradeoff is that external modules that needs access to the source
needs to be tweaked to support kernels build using separate output and
source directories.

So the final result is that external modules that needs access to
the source is broken iff the kernel uses separate output directories.


> > I go for 1), introducing minimal breakage.
> 
> You seem to be in some strange "I must destroy... I must destroy... I must
> destroy..." mental state. There is a no-breakage alternative and you just
> will not consider that at all.

Requiring user to set an environment variable in many common cases
are breakage.

	Sam