[patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Caveh Jalali]

in 2.6.31, it is no longer possible to build kernel modules for some platforms
(the ones that need include2) outside the kernel src tree if the
src tree was built using O= to specify an alternate directory for the
build output.

i think this broke with the following commit:

commit b8b572e1015f81b4e748417be2629dfe51ab99f9
Author: Stephen Rothwell <sfr@canb.auug.org.au>
Date:   Fri Aug 1 15:20:30 2008 +1000

   powerpc: Move include files to arch/powerpc/include/asm


when the platform specific include directories were shuffled around
without updating the
top level Makefile.  however, i think this FAIL is generic, and not
limited to powerpc.
here's a patch to the top level Makefile to teach it about the shuffled files:


--- /local/caveh/linux/linux-2.6.31/Makefile    2009-09-09
15:13:59.000000000 -0700
+++ Makefile    2009-09-16 12:23:05.000000000 -0700
@@ -958,9 +958,9 @@ ifneq ($(KBUILD_SRC),)
                echo "  in the '$(srctree)' directory.";\
                /bin/false; \
        fi;
-       $(Q)if [ ! -d include2 ]; then                                  \
-           mkdir -p include2;                                          \
-           ln -fsn $(srctree)/include/asm-$(SRCARCH) include2/asm;     \
+       $(Q)if [ ! -d include2 ]; then                                   \
+           mkdir -p include2;                                           \
+           ln -fsn $(srctree)/arch/$(SRCARCH)/include/asm include2/asm; \
        fi
 endif

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Américo Wang]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 2150 bytes --]

On Thu, Sep 17, 2009 at 10:30 AM, Caveh Jalali <ecaveh@gmail.com> wrote:> in 2.6.31, it is no longer possible to build kernel modules for some platforms> (the ones that need include2) outside the kernel src tree if the> src tree was built using O= to specify an alternate directory for the> build output.>> i think this broke with the following commit:>> commit b8b572e1015f81b4e748417be2629dfe51ab99f9> Author: Stephen Rothwell <sfr@canb.auug.org.au>> Date:   Fri Aug 1 15:20:30 2008 +1000

Only after 2.6.31? This commit was on 2008 Aug 1...
>>    powerpc: Move include files to arch/powerpc/include/asm>>> when the platform specific include directories were shuffled around> without updating the> top level Makefile.  however, i think this FAIL is generic, and not> limited to powerpc.> here's a patch to the top level Makefile to teach it about the shuffled files:>

You forgot your SOB line.
Anyway, it looks fine.
Acked-by: WANG Cong <xiyou.wangcong@gmail.com>
>> --- /local/caveh/linux/linux-2.6.31/Makefile    2009-09-09> 15:13:59.000000000 -0700> +++ Makefile    2009-09-16 12:23:05.000000000 -0700> @@ -958,9 +958,9 @@ ifneq ($(KBUILD_SRC),)>                 echo "  in the '$(srctree)' directory.";\>                 /bin/false; \>         fi;> -       $(Q)if [ ! -d include2 ]; then                                  \> -           mkdir -p include2;                                          \> -           ln -fsn $(srctree)/include/asm-$(SRCARCH) include2/asm;     \> +       $(Q)if [ ! -d include2 ]; then                                   \> +           mkdir -p include2;                                           \> +           ln -fsn $(srctree)/arch/$(SRCARCH)/include/asm include2/asm; \>         fi>  endifÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Caveh Jalali]

it's known good in 2.6.25, borked in 2.6.31.  i didn't track down
the kernel version number where the offending commit showed up.

anyway, here's my SOB:

Signed-off-by: Caveh Jalali <ecaveh@gmail.com>

On Wed, Sep 16, 2009 at 7:30 PM, Caveh Jalali <ecaveh@gmail.com> wrote:
> in 2.6.31, it is no longer possible to build kernel modules for some platforms
> (the ones that need include2) outside the kernel src tree if the
> src tree was built using O= to specify an alternate directory for the
> build output.
>
> i think this broke with the following commit:
>
> commit b8b572e1015f81b4e748417be2629dfe51ab99f9
> Author: Stephen Rothwell <sfr@canb.auug.org.au>
> Date:   Fri Aug 1 15:20:30 2008 +1000
>
>    powerpc: Move include files to arch/powerpc/include/asm
>
>
> when the platform specific include directories were shuffled around
> without updating the
> top level Makefile.  however, i think this FAIL is generic, and not
> limited to powerpc.
> here's a patch to the top level Makefile to teach it about the shuffled files:
>
>
> --- /local/caveh/linux/linux-2.6.31/Makefile    2009-09-09
> 15:13:59.000000000 -0700
> +++ Makefile    2009-09-16 12:23:05.000000000 -0700
> @@ -958,9 +958,9 @@ ifneq ($(KBUILD_SRC),)
>                 echo "  in the '$(srctree)' directory.";\
>                 /bin/false; \
>         fi;
> -       $(Q)if [ ! -d include2 ]; then                                  \
> -           mkdir -p include2;                                          \
> -           ln -fsn $(srctree)/include/asm-$(SRCARCH) include2/asm;     \
> +       $(Q)if [ ! -d include2 ]; then                                   \
> +           mkdir -p include2;                                           \
> +           ln -fsn $(srctree)/arch/$(SRCARCH)/include/asm include2/asm; \
>         fi
>  endif
>



-- 
00c
http://www.boincstats.com/signature/user_870433_project5.gif

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Arnd Bergmann]

On Thursday 17 September 2009, Caveh Jalali wrote:

> when the platform specific include directories were shuffled around
> without updating the
> top level Makefile.  however, i think this FAIL is generic, and not
> limited to powerpc.
> here's a patch to the top level Makefile to teach it about the shuffled files:
> 
> 
> --- /local/caveh/linux/linux-2.6.31/Makefile    2009-09-09
> 15:13:59.000000000 -0700
> +++ Makefile    2009-09-16 12:23:05.000000000 -0700
> @@ -958,9 +958,9 @@ ifneq ($(KBUILD_SRC),)
>                 echo "  in the '$(srctree)' directory.";\
>                 /bin/false; \
>         fi;
> -       $(Q)if [ ! -d include2 ]; then                                  \
> -           mkdir -p include2;                                          \
> -           ln -fsn $(srctree)/include/asm-$(SRCARCH) include2/asm;     \
> +       $(Q)if [ ! -d include2 ]; then                                   \
> +           mkdir -p include2;                                           \
> +           ln -fsn $(srctree)/arch/$(SRCARCH)/include/asm include2/asm; \
>         fi
>  endif

I think we should no longer have the include2 directory at all with new kernels.

LINUXINCLUDE already contains the right path in theory:

LINUXINCLUDE    := -Iinclude \
                   $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \
                   -I$(srctree)/arch/$(hdr-arch)/include               \
                   -include include/linux/autoconf.h

I'm not sure what goes wrong here. What is the gcc command line you see?

	Arnd <><

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Caveh Jalali]

sadly, some 3rd party src packages don't quite do the right thing with
their kernel include paths and still depend on the include2/asm symlink
to find the platform header files.

i see your point about include2 being somewhat redundant/legacy, however,
as long as we are creating it, we should have it point to the intended
location instead of being a dead link!

On Thu, Sep 17, 2009 at 4:45 AM, Arnd Bergmann <arnd@arndb.de> wrote:
> On Thursday 17 September 2009, Caveh Jalali wrote:
>
>> when the platform specific include directories were shuffled around
>> without updating the
>> top level Makefile.  however, i think this FAIL is generic, and not
>> limited to powerpc.
>> here's a patch to the top level Makefile to teach it about the shuffled files:
>>
>>
>> --- /local/caveh/linux/linux-2.6.31/Makefile    2009-09-09
>> 15:13:59.000000000 -0700
>> +++ Makefile    2009-09-16 12:23:05.000000000 -0700
>> @@ -958,9 +958,9 @@ ifneq ($(KBUILD_SRC),)
>>                 echo "  in the '$(srctree)' directory.";\
>>                 /bin/false; \
>>         fi;
>> -       $(Q)if [ ! -d include2 ]; then                                  \
>> -           mkdir -p include2;                                          \
>> -           ln -fsn $(srctree)/include/asm-$(SRCARCH) include2/asm;     \
>> +       $(Q)if [ ! -d include2 ]; then                                   \
>> +           mkdir -p include2;                                           \
>> +           ln -fsn $(srctree)/arch/$(SRCARCH)/include/asm include2/asm; \
>>         fi
>>  endif
>
> I think we should no longer have the include2 directory at all with new kernels.
>
> LINUXINCLUDE already contains the right path in theory:
>
> LINUXINCLUDE    := -Iinclude \
>                   $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \
>                   -I$(srctree)/arch/$(hdr-arch)/include               \
>                   -include include/linux/autoconf.h
>
> I'm not sure what goes wrong here. What is the gcc command line you see?
>
>        Arnd <><
>



-- 
00c
http://www.boincstats.com/signature/user_870433_project5.gif

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Américo Wang]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 1974 bytes --]

On Thu, Sep 17, 2009 at 7:45 PM, Arnd Bergmann <arnd@arndb.de> wrote:> On Thursday 17 September 2009, Caveh Jalali wrote:>>> when the platform specific include directories were shuffled around>> without updating the>> top level Makefile.  however, i think this FAIL is generic, and not>> limited to powerpc.>> here's a patch to the top level Makefile to teach it about the shuffled files:>>>>>> --- /local/caveh/linux/linux-2.6.31/Makefile    2009-09-09>> 15:13:59.000000000 -0700>> +++ Makefile    2009-09-16 12:23:05.000000000 -0700>> @@ -958,9 +958,9 @@ ifneq ($(KBUILD_SRC),)>>                 echo "  in the '$(srctree)' directory.";\>>                 /bin/false; \>>         fi;>> -       $(Q)if [ ! -d include2 ]; then                                  \>> -           mkdir -p include2;                                          \>> -           ln -fsn $(srctree)/include/asm-$(SRCARCH) include2/asm;     \>> +       $(Q)if [ ! -d include2 ]; then                                   \>> +           mkdir -p include2;                                           \>> +           ln -fsn $(srctree)/arch/$(SRCARCH)/include/asm include2/asm; \>>         fi>>  endif>> I think we should no longer have the include2 directory at all with new kernels.>> LINUXINCLUDE already contains the right path in theory:>> LINUXINCLUDE    := -Iinclude \>                   $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \>                   -I$(srctree)/arch/$(hdr-arch)/include               \>                   -include include/linux/autoconf.h
Hmm, at least in mmotm, we still have include2...
Maybe Sam is queueing the patches to remove include2?ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Sam Ravnborg]

On Fri, Sep 18, 2009 at 09:49:53AM +0800, Am?rico Wang wrote:
> On Thu, Sep 17, 2009 at 7:45 PM, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Thursday 17 September 2009, Caveh Jalali wrote:
> >
> >> when the platform specific include directories were shuffled around
> >> without updating the
> >> top level Makefile.  however, i think this FAIL is generic, and not
> >> limited to powerpc.
> >> here's a patch to the top level Makefile to teach it about the shuffled files:
> >>
> >>
> >> --- /local/caveh/linux/linux-2.6.31/Makefile    2009-09-09
> >> 15:13:59.000000000 -0700
> >> +++ Makefile    2009-09-16 12:23:05.000000000 -0700
> >> @@ -958,9 +958,9 @@ ifneq ($(KBUILD_SRC),)
> >>                 echo "  in the '$(srctree)' directory.";\
> >>                 /bin/false; \
> >>         fi;
> >> -       $(Q)if [ ! -d include2 ]; then                                  \
> >> -           mkdir -p include2;                                          \
> >> -           ln -fsn $(srctree)/include/asm-$(SRCARCH) include2/asm;     \
> >> +       $(Q)if [ ! -d include2 ]; then                                   \
> >> +           mkdir -p include2;                                           \
> >> +           ln -fsn $(srctree)/arch/$(SRCARCH)/include/asm include2/asm; \
> >>         fi
> >>  endif
> >
> > I think we should no longer have the include2 directory at all with new kernels.
> >
> > LINUXINCLUDE already contains the right path in theory:
> >
> > LINUXINCLUDE    := -Iinclude \
> >                   $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \
> >                   -I$(srctree)/arch/$(hdr-arch)/include               \
> >                   -include include/linux/autoconf.h
> 
> Hmm, at least in mmotm, we still have include2...
> 
> Maybe Sam is queueing the patches to remove include2?
I have them queued - yes.
But as they never hit -next they will wait another cycle.

	Sam

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Arnd Bergmann]

On Thursday 17 September 2009, Caveh Jalali wrote:
> sadly, some 3rd party src packages don't quite do the right thing with
> their kernel include paths and still depend on the include2/asm symlink
> to find the platform header files.
> 
> i see your point about include2 being somewhat redundant/legacy, however,
> as long as we are creating it, we should have it point to the intended
> location instead of being a dead link!

I guess we could just as well put something in there to notify such
programs of how to use the kernel Makefiles. External packages should
not try to guess the layout of the kernel source tree, they can only
get it wrong anyway.

There are apparently also user space packages trying to access 
/lib/modules/${krel}/build/arch/${arch}/include/, which is even
more wrong.

I'm thinking about adding something in <linux/kernel.h> to warn
about this when using the kernel headers without either '-D__KERNEL__'
or 'make headers_install'. Is there any valid case that this would
break? It should probably give a warning that includes a http URL
holding a description of what to do.

	Arnd <><

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Arnd Bergmann]

On Thursday 17 September 2009, Caveh Jalali wrote:
> sadly, some 3rd party src packages don't quite do the right thing with
> their kernel include paths and still depend on the include2/asm symlink
> to find the platform header files.
> 
> i see your point about include2 being somewhat redundant/legacy, however,
> as long as we are creating it, we should have it point to the intended
> location instead of being a dead link!

I guess we could just as well put something in there to notify such
programs of how to use the kernel Makefiles. External packages should
not try to guess the layout of the kernel source tree, they can only
get it wrong anyway.

There are apparently also user space packages trying to access 
/lib/modules/${krel}/build/arch/${arch}/include/, which is even
more wrong.

I'm thinking about adding something in <linux/kernel.h> to warn
about this when using the kernel headers without either '-D__KERNEL__'
or 'make headers_install'. Is there any valid case that this would
break? It should probably give a warning that includes a http URL
holding a description of what to do.

	Arnd <><

Re: [patch] Makefile: can't build modules outside the kernel tree if it was built using O=

[Sam Ravnborg]

On Wed, Sep 16, 2009 at 07:30:43PM -0700, Caveh Jalali wrote:
> in 2.6.31, it is no longer possible to build kernel modules for some platforms
> (the ones that need include2) outside the kernel src tree if the
> src tree was built using O= to specify an alternate directory for the
> build output.
> 
> i think this broke with the following commit:
> 
> commit b8b572e1015f81b4e748417be2629dfe51ab99f9
> Author: Stephen Rothwell <sfr@canb.auug.org.au>
> Date:   Fri Aug 1 15:20:30 2008 +1000
> 
>    powerpc: Move include files to arch/powerpc/include/asm
> 
> 
> when the platform specific include directories were shuffled around
> without updating the
> top level Makefile.  however, i think this FAIL is generic, and not
> limited to powerpc.
> here's a patch to the top level Makefile to teach it about the shuffled files:

This is no good. Nothing outside the kernel should in any way rely
on include2. And it will soon be gone anyway.

So we will not try to workaround something that is broken.

	Sam

[PATCH] warn about use of uninstalled kernel headers

[Arnd Bergmann]

On Friday 18 September 2009, Sam Ravnborg wrote:
> > > I think we should no longer have the include2 directory at all with new kernels.
> > >
> > > LINUXINCLUDE already contains the right path in theory:
> > >
> > > LINUXINCLUDE    := -Iinclude \
> > >                   $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \
> > >                   -I$(srctree)/arch/$(hdr-arch)/include               \
> > >                   -include include/linux/autoconf.h
> > 
> > Hmm, at least in mmotm, we still have include2...
> > 
> > Maybe Sam is queueing the patches to remove include2?
> I have them queued - yes.
> But as they never hit -next they will wait another cycle.

Variations of this are starting to turn into a FAQ. How about
printing a useful warning when someone tries to use the kernel
headers without installing them first?

	Arnd <><
---
User applications frequently hit problems when they try to use
the kernel headers directly, rather than the exported headers.

This adds an explicit warning for this case, and points to
a URL holding an explanation of why this is wrong and what
to do about it.

Signed-off-by: Arnd Bergmann <arnd@arndb.de>
---
 include/linux/kernel.h     |    6 ++++++
 scripts/headers_install.pl |    2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index d6320a3..f392d72 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -656,6 +656,12 @@ extern int do_sysinfo(struct sysinfo *info);
 
 #endif /* __KERNEL__ */
 
+#ifndef __EXPORTED_HEADERS__
+#ifndef __KERNEL__
+#warning Attempt to use kernel headers from user space, see http://kernelnewbies.org/KernelHeaders
+#endif /* __KERNEL__ */
+#endif /* __EXPORTED_HEADERS__ */
+
 #define SI_LOAD_SHIFT	16
 struct sysinfo {
 	long uptime;			/* Seconds since boot */
diff --git a/scripts/headers_install.pl b/scripts/headers_install.pl
index c6ae405..b89ca2c 100644
--- a/scripts/headers_install.pl
+++ b/scripts/headers_install.pl
@@ -20,7 +20,7 @@ use strict;
 
 my ($readdir, $installdir, $arch, @files) = @ARGV;
 
-my $unifdef = "scripts/unifdef -U__KERNEL__";
+my $unifdef = "scripts/unifdef -U__KERNEL__ -D__EXPORTED_HEADERS__";
 
 foreach my $file (@files) {
 	local *INFILE;

Re: [PATCH] warn about use of uninstalled kernel headers

[Sam Ravnborg]

On Mon, Sep 21, 2009 at 04:37:12PM +0200, Arnd Bergmann wrote:
> User applications frequently hit problems when they try to use
> the kernel headers directly, rather than the exported headers.
> 
> This adds an explicit warning for this case, and points to
> a URL holding an explanation of why this is wrong and what
> to do about it.

Like the idea.

> 
> Signed-off-by: Arnd Bergmann <arnd@arndb.de>
> ---
>  include/linux/kernel.h     |    6 ++++++
>  scripts/headers_install.pl |    2 +-
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/kernel.h b/include/linux/kernel.h
> index d6320a3..f392d72 100644
> --- a/include/linux/kernel.h
> +++ b/include/linux/kernel.h
> @@ -656,6 +656,12 @@ extern int do_sysinfo(struct sysinfo *info);
>  
>  #endif /* __KERNEL__ */
>  
> +#ifndef __EXPORTED_HEADERS__
But why do we need this "__EXPORTED_HEADERS__" thing?

?

	Sam

Re: [PATCH] warn about use of uninstalled kernel headers

[Arnd Bergmann]

On Tuesday 22 September 2009, Sam Ravnborg wrote:
> > +#ifndef __EXPORTED_HEADERS__
> But why do we need this "__EXPORTED_HEADERS__" thing?
> 
> ?

The problem is that the warning should only be for headers
that are not installed yet, but it needs to be stripped
in the installed version. Since we're already postprocessing
all files with unifdef, that seemed like the easiest way 
to strip out the #warning. Obviously, I couldn't use the
#ifdef __KERNEL__ logic, because that would either warn
give false positives when building kernel code or when
building user code from installed headers.

	Arnd <><

Re: [PATCH] warn about use of uninstalled kernel headers

[Sam Ravnborg]

On Mon, Sep 21, 2009 at 04:37:12PM +0200, Arnd Bergmann wrote:
> On Friday 18 September 2009, Sam Ravnborg wrote:
> > > > I think we should no longer have the include2 directory at all with new kernels.
> > > >
> > > > LINUXINCLUDE already contains the right path in theory:
> > > >
> > > > LINUXINCLUDE    := -Iinclude \
> > > >                   $(if $(KBUILD_SRC),-Iinclude2 -I$(srctree)/include) \
> > > >                   -I$(srctree)/arch/$(hdr-arch)/include               \
> > > >                   -include include/linux/autoconf.h
> > > 
> > > Hmm, at least in mmotm, we still have include2...
> > > 
> > > Maybe Sam is queueing the patches to remove include2?
> > I have them queued - yes.
> > But as they never hit -next they will wait another cycle.
> 
> Variations of this are starting to turn into a FAQ. How about
> printing a useful warning when someone tries to use the kernel
> headers without installing them first?

OK - applied.

	Sam