kernel headers not properly generated on platforms with a mach- and a plat- (i.e. mach-mx5, plat-mxc etc.)

kernel headers not properly generated on platforms with a mach- and a plat- (i.e. mach-mx5, plat-mxc etc.)

[Matt Sealey]

Hey guys,

Puzzling over this again, is there some deep dark reason that if I
create kernel headers (headers_install or headers_check) for MX51
processors, or anything with a mach- AND a plat-, that the
plat-foo/include/mach/* headers are not copied into the header tree?

As a quick example, build any external module (compat-wireless is a
good example) or libc against the headers alone and not the full
source and they will complain about asm/memory.h trying to include
mach/memory.h which does not exist. Doing a little hack to copy those
files across absolutely fixes it.

I know there is a weirdness here: the root arch/arm/Kconfig in my tree
includes plat-mxc/Kconfig which includes mach-mx5/Kconfig. This is how
the magic happens, but I can't find any magic that modifies the
include paths to pick includes from these directories in the first
place. They don't seem symlinked into the asm-arm/mach directory on a
kernel build.

My first thought is to modify the Debian packaging so it manually
hacks those files in to get it working, but my major concern is you
would have to do this for every platform and machine combination where
this occurs, and this is obviously a kernel problem not a packaging
problem.

Has anyone even noticed this before (it's quite hard to search for the
exact situation in freeform text in Google or ML archives, sigh)? Or
is there a well known solution everyone uses but just never pushes to
the distribution packagers or so? Is the solution to use the full
Linux source code (kind of annoying for consumers that need to put
aside 400MB of disk space for an extracted Linux source when 8MB of
headers would do) rather than just headers as we have been doing for
the past few weeks, and off and on for years?

-- 
Matt Sealey <matt@genesi-usa.com>
Product Development Analyst, Genesi USA, Inc.

kernel headers not properly generated on platforms with a mach- and a plat- (i.e. mach-mx5, plat-mxc etc.)

[Arnd Bergmann]

On Saturday 29 January 2011 01:57:19 Matt Sealey wrote:
> 
> Puzzling over this again, is there some deep dark reason that if I
> create kernel headers (headers_install or headers_check) for MX51
> processors, or anything with a mach- AND a plat-, that the
> plat-foo/include/mach/* headers are not copied into the header tree?

As far as I can tell, the mach/* files cannot possibly be exported,
because the user space headers are by definition machine independent.
Otherwise you would not be guaranteed that applications you build
can run on all systems.

> As a quick example, build any external module (compat-wireless is a
> good example) or libc against the headers alone and not the full
> source and they will complain about asm/memory.h trying to include
> mach/memory.h which does not exist. Doing a little hack to copy those
> files across absolutely fixes it.

memory.h is not exported by make headers_install.

external modules are not built against the exported headers
(linux-libc-dev*.dpkg), but are built against a sparse copy of the
kernel source tree (linux-headers-*.dpkg)

> I know there is a weirdness here: the root arch/arm/Kconfig in my tree
> includes plat-mxc/Kconfig which includes mach-mx5/Kconfig. This is how
> the magic happens, but I can't find any magic that modifies the
> include paths to pick includes from these directories in the first
> place. They don't seem symlinked into the asm-arm/mach directory on a
> kernel build.

look for 'machdirs' in arch/arm/Makefile

> Has anyone even noticed this before (it's quite hard to search for the
> exact situation in freeform text in Google or ML archives, sigh)? Or
> is there a well known solution everyone uses but just never pushes to
> the distribution packagers or so? Is the solution to use the full
> Linux source code (kind of annoying for consumers that need to put
> aside 400MB of disk space for an extracted Linux source when 8MB of
> headers would do) rather than just headers as we have been doing for
> the past few weeks, and off and on for years?

The problem has mostly gone away now: All the important drivers that
used to be external modules are now in the regular upstream kernel
or at least in drivers/staging. In the few cases where the license
prohibits this, you can still package the module to have a
build-dependency on the full kernel sources and put the small
burden of installing them on the packager, but not the end-user.

Of course, if you find a packaging problem with the linux-headers
package, submit a patch for that or open a bug report.

	Arnd

kernel headers not properly generated on platforms with a mach- and a plat- (i.e. mach-mx5, plat-mxc etc.)

[Russell King - ARM Linux]

On Fri, Jan 28, 2011 at 06:57:19PM -0600, Matt Sealey wrote:
> Puzzling over this again, is there some deep dark reason that if I
> create kernel headers (headers_install or headers_check) for MX51
> processors, or anything with a mach- AND a plat-, that the
> plat-foo/include/mach/* headers are not copied into the header tree?

headers_install does not install headers for building modules.  It
installs headers for use by userspace - specifically your libc.  These
headers purposely do not contain any platform specifics because those
are not part of the userspace API.  They also have various kernel
specific information stripped out of them.

They are completely unsuitable for building anything kernel-related.

> As a quick example, build any external module (compat-wireless is a
> good example) or libc against the headers alone and not the full
> source and they will complain about asm/memory.h trying to include
> mach/memory.h which does not exist. Doing a little hack to copy those
> files across absolutely fixes it.

asm/memory.h is not part of headers_install.

> I know there is a weirdness here: the root arch/arm/Kconfig in my tree
> includes plat-mxc/Kconfig which includes mach-mx5/Kconfig. This is how
> the magic happens, but I can't find any magic that modifies the
> include paths to pick includes from these directories in the first
> place. They don't seem symlinked into the asm-arm/mach directory on a
> kernel build.

We don't use symlinks anymore when dealing with header file directories.

You can only build modules against the kernel source.  Building external
modules is documented in Documentation/kbuild/modules.txt.  This requires
more than just the kernel headers - it requires a preconfigured kernel
tree, which if you're using modversions, has already been built.

kernel headers not properly generated on platforms with a mach- and a plat- (i.e. mach-mx5, plat-mxc etc.)

[Matt Sealey]

Okay so it's a Debian packaging problem then. To get dkms and similar
to work properly with just -headers there are a bunch of hacks (it
installs a pre-prepped kernel tree up to the point it's needed to get
that to work) and this would have to be handled.

Arnd, Thanks for the machdirs hint.

-- 
Matt Sealey <matt@genesi-usa.com>
Product Development Analyst, Genesi USA, Inc.



On Sat, Jan 29, 2011 at 3:43 AM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Fri, Jan 28, 2011 at 06:57:19PM -0600, Matt Sealey wrote:
>> Puzzling over this again, is there some deep dark reason that if I
>> create kernel headers (headers_install or headers_check) for MX51
>> processors, or anything with a mach- AND a plat-, that the
>> plat-foo/include/mach/* headers are not copied into the header tree?
>
> headers_install does not install headers for building modules. ?It
> installs headers for use by userspace - specifically your libc. ?These
> headers purposely do not contain any platform specifics because those
> are not part of the userspace API. ?They also have various kernel
> specific information stripped out of them.
>
> They are completely unsuitable for building anything kernel-related.
>
>> As a quick example, build any external module (compat-wireless is a
>> good example) or libc against the headers alone and not the full
>> source and they will complain about asm/memory.h trying to include
>> mach/memory.h which does not exist. Doing a little hack to copy those
>> files across absolutely fixes it.
>
> asm/memory.h is not part of headers_install.
>
>> I know there is a weirdness here: the root arch/arm/Kconfig in my tree
>> includes plat-mxc/Kconfig which includes mach-mx5/Kconfig. This is how
>> the magic happens, but I can't find any magic that modifies the
>> include paths to pick includes from these directories in the first
>> place. They don't seem symlinked into the asm-arm/mach directory on a
>> kernel build.
>
> We don't use symlinks anymore when dealing with header file directories.
>
> You can only build modules against the kernel source. ?Building external
> modules is documented in Documentation/kbuild/modules.txt. ?This requires
> more than just the kernel headers - it requires a preconfigured kernel
> tree, which if you're using modversions, has already been built.
>

kernel headers not properly generated on platforms with a mach- and a plat- (i.e. mach-mx5, plat-mxc etc.)

[Russell King - ARM Linux]

On Sat, Jan 29, 2011 at 08:57:54AM -0600, Matt Sealey wrote:
> Okay so it's a Debian packaging problem then. To get dkms and similar
> to work properly with just -headers there are a bunch of hacks (it
> installs a pre-prepped kernel tree up to the point it's needed to get
> that to work) and this would have to be handled.

What Fedora does is the kernel-headers package contains the userspace
headers (/usr/include/{asm,linux,...}) which correspond with the headers
which were used to build glibc.

The kernel package, containing the kernel binary and modules, contains
a link from /lib/modules/$version/build to the kernel development tree
in /usr/src/kernels/$version

The kernel-devel package contains a cut-down kernel tree, including the
Kconfigs, Makefiles, include files, scripts (and pre-built binaries
in scripts/) and a .config - in other words, a prepared kernel source
tree with most of the .c files missing.

I'm not sure what debian do, but they should be doing something similar.
They should not be stuffing the userspace headers package with any header
file which isn't exported by 'make headers_install'.

Hope that's of some use.