Simple header cleanups

Simple header cleanups

[David Woodhouse]

Andrew (or preferably Linus since these are fairly simple and
unintrusive bug fixes), please pull from my tree at 
git://git.infradead.org/hdrcleanup-2.6.git

(Gitweb at http://git.infradead.org/?p=hdrcleanup-2.6.git;a=summary)

This tree contains a number of simple fixes for kernel headers which are
currently exposing things to userspace that should not be present
outside #ifdef __KERNEL__.

Since ifdefs are horrid, these patches tend not to add new ones --
mostly it's a case of moving things inside existing ifdefs, rather than
adding new ones.

Most of the patches are small and self-contained, except for the one
which removes #include <linux/config.h> from everything under include/

The individual commits are as follows:

      Remove user-visible references to PAGE_SIZE in include/asm-powerpc/elf.h
      Include <linux/jiffies.h> from linux/acct.h only in kernel-private part.
      Don't include agp_backend.h in user-visible part of agpgart.h
      Use __KERNEL__ to hide kernel-private bits of linux/gameport.h
      Export only the appropriate GS_xxx flags to userspace from generic_serial.h
      Include various private files only from within __KERNEL__ in genhd.h
      Sanitise linux/i2c-algo-ite.h for userspace consumption
      Sanitise linux/i2c.h for userspace consumption
      Don't include <linux/device.h> from user-visible part of linux/ipmi.h
      Remove gratuitous inclusion of <linux/pci.h> from linux/isdn/tpam.h
      Sanitise linux/mman.h for userspace consumption
      Don't include private files from user-visible part of linux/ncp_fs.h
      Don't include <linux/list.h> from user-visible part of linux/msg.h
      Don't include <linux/stringify> from user-visible part of linux/net.h
      Don't include private headers from user-visible parts of include/linux/nfs*.h
      Don't include private headers from user-visible parts of linux/quota.h
      Don't include <linux/list.h> from user-visible part of reiserfs_xattr.h
      Partially sanitise linux/sched.h for userspace consumption
      Don't include <asm/atomic.h> from user-visible part of linux/sem.h
      Don't include private headers from user-visible part of linux/signal.h
      Move comment in mtd-abi.h to stop confusing unifdef
      Don't include <linux/spinlock.h> from user-visible part of linux/wanrouter.h
      Don't export CONFIG_COMPAT stuff in linux/usbdevice_fs.h to userspace
      Sanitise linux/sunrpc/debug.h for userspace consumption
      Don't include private headers from user-visible part of linux/smb_fs.h
      Don't include private headers from user-visible part of linux/ext2_fs.h
      Don't include private headers from user-visible part of linux/ext3_fs.h
      Don't include <linux/config.h> and <linux/linkage.h> from linux/socket.h
      Don't include linux/config.h from anywhere else in include/
      Sanitise linux/audit.h for userspace consumption, split elf-em.h from elf.h
      Sanitise linux/sched.h for userspace consumption

Excluding the one-line files which have just <linux/config.h> removed,
the diffstat looks like this:

 asm-m32r/mmu_context.h                       |    2
 asm-powerpc/elf.h                            |    7 --
 asm-ppc/page.h                               |    2
 asm-sparc/system.h                           |    2
 linux/acct.h                                 |    3
 linux/agpgart.h                              |    3
 linux/audit.h                                |    4 -
 linux/elf-em.h                               |   44 +++++++++++++
 linux/elf.h                                  |   59 -----------------
 linux/ext2_fs.h                              |    2
 linux/ext3_fs.h                              |    7 --
 linux/gameport.h                             |    6 +
 linux/generic_serial.h                       |    6 +
 linux/genhd.h                                |   12 +--
 linux/i2c-algo-ite.h                         |    7 +-
 linux/i2c.h                                  |    9 +-
 linux/ipmi.h                                 |    2
 linux/mman.h                                 |   12 ++-
 linux/msg.h                                  |    2
 linux/ncp_fs.h                               |    5 -
 linux/net.h                                  |    3
 linux/nfs.h                                  |    8 +-
 linux/nfs4.h                                 |    6 -
 linux/nfs_fs.h                               |   39 +++++------
 linux/quota.h                                |    4 -
 linux/reiserfs_xattr.h                       |    3
 linux/sched.h                                |   90 +++++++++++++--------------
 linux/sem.h                                  |    2
 linux/signal.h                               |    4 -
 linux/smb_fs.h                               |    4 -
 linux/socket.h                               |    2
 linux/sunrpc/debug.h                         |   24 +++----
 linux/usbdevice_fs.h                         |    2
 linux/wanrouter.h                            |    4 -
 mtd/mtd-abi.h                                |    5 -
 864 files changed, 191 insertions(+), 1034 deletions(-)



-- 
dwmw2

Re: Simple header cleanups

[Linus Torvalds]

On Thu, 27 Apr 2006, David Woodhouse wrote:
>
> Andrew (or preferably Linus since these are fairly simple and
> unintrusive bug fixes), please pull from my tree at 
> git://git.infradead.org/hdrcleanup-2.6.git

Hmm. Every time we've done this in the past, something has broken, so I'd 
actually _much_ rather wait until early in the 2.6.18 cycle than do it 
now.

Yeah, people shouldn't include kernel headers, but if they didn't, this 
patch wouldn't matter. And when they do, patches like this tends to show 
some strange app that depends on the current header layout.. Gaah.

		Linus

Re: Simple header cleanups

[Gerrit Huizenga]

On Wed, 26 Apr 2006 19:18:52 PDT, Linus Torvalds wrote:
> 
> 
> On Thu, 27 Apr 2006, David Woodhouse wrote:
> >
> > Andrew (or preferably Linus since these are fairly simple and
> > unintrusive bug fixes), please pull from my tree at 
> > git://git.infradead.org/hdrcleanup-2.6.git
> 
> Hmm. Every time we've done this in the past, something has broken, so I'd 
> actually _much_ rather wait until early in the 2.6.18 cycle than do it 
> now.
> 
> Yeah, people shouldn't include kernel headers, but if they didn't, this 
> patch wouldn't matter. And when they do, patches like this tends to show 
> some strange app that depends on the current header layout.. Gaah.
> 
> 		Linus

Except there are a few things you can't use in user space now, like
connectors, without including a kernel header file.  So, should apps
duplicate the structures and definitions in each and every application and
hopefully notice when the kernel API changes?  Seems painful.

Plus, do we really want Apache or other licensed software directly
including GPL header files?  Oh what a tangled web we weave when we suck
GPL headers into other applications.

Damned if you do, damned if you don't.

gerrit

Re: Simple header cleanups

[David Woodhouse]

On Wed, 2006-04-26 at 19:18 -0700, Linus Torvalds wrote:
> Hmm. Every time we've done this in the past, something has broken, so I'd 
> actually _much_ rather wait until early in the 2.6.18 cycle than do it 
> now.

OK. I've tried to be _very_ conservative with this set -- it mostly is
just a case of moving #includes from one part of a header file to
another so they aren't exposed to userspace. I figured the more
interesting stuff could come later.

But if you'd rather have it only in -mm only for now, that's fine.

> Yeah, people shouldn't include kernel headers, but if they didn't, this 
> patch wouldn't matter. And when they do, patches like this tends to show 
> some strange app that depends on the current header layout.. Gaah.

Well, yes, but we all know that people _have_ to include kernel headers.
We can't just bury our head in the sand and say "they mustn't do that".
The kernel headers contain all the juicy stuff like structure
definitions and ioctls which you _need_ in order to communicate with the
kernel.

The problem is that we don't actually have any _discipline_ about how we
throw our kernel headers over the wall. We never even _think_ about how
usable they are in userspace, or how what we're doing will affect
userspace.

That's why there are so many silly little cleanups which I had to make
just to be able to use some recent kernel headers sanely again in
userspace, instead of the ancient fork we'd taken in Fedora from a 2.4
kernel.

The answer to this lies in my other git tree, at
git://git.infradead.org/hdrinstall-2.6.git -- again in gitweb, at
http://git.infradead.org/?p=hdrinstall-2.6.git;a=summary

Based on an original implementation by Arnd Bergmann, this implements
new 'make headers_install' target for the kernel Makefiles, which
exports a selection of the contents of our include/ directories to
userspace, having run them through unifdef where appropriate.

This gives us a pristine set of headers which _are_ suitable for
userspace inclusion. I've been talking to maintainers of the
'kernel-headers' or similar packages in other distributions, and we seem
to have a consensus that this is going to be useful. All the
distributions can ship basically the _same_ set of headers, instead of
all doing their own thing to clean them up, and getting it inconsistent
across distros.

As well as achieving that laudable goal, the export step also allows us
(and in particular our janitor-types) to _read_ through the headers
which userspace gets, and start to clean them up a bit more, cleaning up
namespace pollution and removing things which are still visible but
which shouldn't be. It allows us to take a diff of the _exported_
headers between one release and the next, and see the user-visible
changes in isolation.

-- 
dwmw2

Re: Simple header cleanups

[Linus Torvalds]

On Wed, 26 Apr 2006, Gerrit Huizenga wrote:
> 
> Damned if you do, damned if you don't.

Right. Which is why I'm ok with the cleanups, I'm just _not_ ok with them 
happening this late in the game (ie I'm hoping to get 2.6.17 out in a 
couple of weeks..)

		Linus

Re: Simple header cleanups

[Linus Torvalds]

On Thu, 27 Apr 2006, David Woodhouse wrote:
> 
> Well, yes, but we all know that people _have_ to include kernel headers.
> We can't just bury our head in the sand and say "they mustn't do that".
> The kernel headers contain all the juicy stuff like structure
> definitions and ioctls which you _need_ in order to communicate with the
> kernel.

.. and that's generally the job of library maintainers.

I disagree violently with the notion that any normal system should _ever_ 
have the regular headers have anything to do with "what kernel source is 
installed right now". It is untenable to expect kernel developers to have 
to worry about user-level header problems.

> The problem is that we don't actually have any _discipline_ about how we
> throw our kernel headers over the wall. We never even _think_ about how
> usable they are in userspace, or how what we're doing will affect
> userspace.

AND WE SHOULD NOT HAVE TO!

Linkages are bad. We _will_ break kernel headers for user space, and 
that's just a undeniable fact.

If this is work to try to make kernel headers generally palatable to user 
space, I'm not going to apply it at all. Not now, not early in the 2.6.18 
sequence, not EVER. Because that's not a goal I believe in for a moment.

In contrast, if this is work to make it eventually _easier_ for _library_ 
people to decide to upgrade their kernel-related information, I'm ok with 
it. But that "target audience" part really is very very important. The 
target audience should NOT be applications including kernel headers.

The target audience should be distributions and library maintainers.

			Linus

Re: Simple header cleanups

[David Woodhouse]

On Wed, 2006-04-26 at 19:27 -0700, Gerrit Huizenga wrote:
> Plus, do we really want Apache or other licensed software directly
> including GPL header files?  Oh what a tangled web we weave when we
> suck GPL headers into other applications. 

libstdc++ is GPL too. Why's that not a problem? Because it has an
exception explicitly allowing you to use it _without_ becoming covered
by the GPL, in certain circumstances. A bit like this one:

   NOTE! This copyright does *not* cover user programs that use kernel
 services by normal system calls - this is merely considered normal use
 of the kernel, and does *not* fall under the heading of "derived work".

Of course, the libstdc++ exception was written a bit more carefully than
ours, but ours seems to be perfectly sufficient.

http://gcc.gnu.org/onlinedocs/libstdc++/17_intro/license.html

-- 
dwmw2

Re: Simple header cleanups

[David Woodhouse]

On Wed, 2006-04-26 at 19:59 -0700, Linus Torvalds wrote:
> Linkages are bad. We _will_ break kernel headers for user space, and 
> that's just a undeniable fact.

Agreed. And distributions and library maintainers _will_ fix them. Are
we to deny those people the tools which will help them to keep track of
our breakage and submit patches to fix it?

The current situation is that they all go off and do their own thing to
work around it, instead of submitting those fixes. We need to do better
than that.

> If this is work to try to make kernel headers generally palatable to user 
> space, I'm not going to apply it at all. Not now, not early in the 2.6.18 
> sequence, not EVER. Because that's not a goal I believe in for a moment.

Kernel headers are never going to be generally palatable to user space.
That's always been clear, and it's a large part of the reason why we
wanted the 'export' step -- so that a _subset_ of headers could be
copied out for use by userspace, rather than a complete tarball as has
been used in the part.

> In contrast, if this is work to make it eventually _easier_ for _library_ 
> people to decide to upgrade their kernel-related information, I'm ok with 
> it. But that "target audience" part really is very very important. The 
> target audience should NOT be applications including kernel headers.
> 
> The target audience should be distributions and library maintainers. 

And those are the people I've been talking to over the last week or so.

All the distributions have their own hacks and patches and scripts which
mangle the kernel headers. Some from a vaguely current copy, some from a
copy which was forked years ago and kept vaguely up to date with
syscalls and new ioctls by hand. The results differ wildly from
distribution to distribution, and the end result for the _user_ is a
steaming inconsistent pile of crap.

What I'm trying to do here is centralise that process and let people
work on it together -- and yes, it still needs to go through
distributions and library maintainers. The point is not to go back to
making symlinks from /usr/include/{linux,asm} to the kernel directories
again -- that would be a retrograde step. The point is that distro and
library maintainers can work from what is provided by the kernel to
achieve a consistent and useful result.

That _is_ needed. It's a quality of implementation issue.

-- 
dwmw2

Re: Simple header cleanups

[Linus Torvalds]

On Thu, 27 Apr 2006, David Woodhouse wrote:
> 
> Agreed. And distributions and library maintainers _will_ fix them. Are
> we to deny those people the tools which will help them to keep track of
> our breakage and submit patches to fix it?

No. As mentioned, as long as the target audience is distributions and 
library maintainers, I definitely think we should do help them as much as 
possible. Our problems have historically been "random people" who have 
/usr/include/linux being the symlink to "kernel source of the day", which 
is an unsupportable situation.

(And yes, for a short while back in the early nineties, that symlink was 
even the proper thing to do. But exactly because it's unsupportable, it 
pretty quickly got to be a "don't do that then", but I still occasionally 
hear from people who use bad distributions).

		Linus

Re: Simple header cleanups

[David Woodhouse]

On Wed, 2006-04-26 at 20:31 -0700, Linus Torvalds wrote:
> No. As mentioned, as long as the target audience is distributions and 
> library maintainers, I definitely think we should do help them as much as 
> possible. Our problems have historically been "random people" who have 
> /usr/include/linux being the symlink to "kernel source of the day", which 
> is an unsupportable situation. 

Right. I'm not interested in helping random people who want the kernel
source of the day; my point here is to just help the distributions get
their collective act together w.r.t. the kernel headers which we _have_
to ship in some form or other.

In particular, my reason for getting started on it was because for my
sins I own the glibc-kernheaders package in Fedora, and it's a complete
pain in my wossname.

We _do_ need to update periodically. I'd been doing it by cherry-picking
stuff like new ioctls and syscalls manually, and it was horrid. The
answer was obviously to have some way to at least partially automate the
process of updating from the real kernel... and it's fairly obvious how
we got from that requirement to the second of the git trees I'm showing
you now, having consulted the people who maintain equivalent packages in
other distributions along the way.

-- 
dwmw2

Re: Simple header cleanups

[Gerrit Huizenga]

On Thu, 27 Apr 2006 04:00:30 BST, David Woodhouse wrote:
> On Wed, 2006-04-26 at 19:27 -0700, Gerrit Huizenga wrote:
> > Plus, do we really want Apache or other licensed software directly
> > including GPL header files?  Oh what a tangled web we weave when we
> > suck GPL headers into other applications. 
> 
> libstdc++ is GPL too. Why's that not a problem? Because it has an
> exception explicitly allowing you to use it _without_ becoming covered
> by the GPL, in certain circumstances. A bit like this one:
> 
>    NOTE! This copyright does *not* cover user programs that use kernel
>  services by normal system calls - this is merely considered normal use
>  of the kernel, and does *not* fall under the heading of "derived work".
 
 This is not in conflict with the GPL in any way - just a clarification
 of the address boundary between running applications and the kernel
 address space.  In other words, if running any application on a GPL
 kernel virally infected those applications, very few people would ever
 use a GPL'd kernel.  Same is true with a GPL application - it clearly
 could not infect the kernel it was running on.  I believe this is
 nothing but a clarification of an interpretation of the GPL as it
 stands today (well, GPL and copyright law).

 This does not say that you can #include files from any source, including
 the kernel, which are released/licensed under the GPL.  The GPL
 specifically locks out any changes to the GPL so even if you wanted
 to provide an exception.  Keep in mind that this is inclusion into
 a source from which you build a binary and then redistribute that
 binary, such as in an rpm.  That is different from the libstdc++
 exception below, which is a *runtime* exception, not a compile time
 exception.

> Of course, the libstdc++ exception was written a bit more carefully than
> ours, but ours seems to be perfectly sufficient.
> 
> http://gcc.gnu.org/onlinedocs/libstdc++/17_intro/license.html

 The little suggestion of speaking to a lawyer (since I am not one,
 have never played one on TV, and would never admit to speaking to
 one) is a good suggestion.  However, note that you can build a
 free software *library* with this code because of this exception
 (which is purportedly replicated in every related header file)
 but this does not mean that you could include those files directly
 in an application.  Also, this is not the exception that is currently
 associated with the kernel header files so I'm not sure that I
 see a way to retroactively apply it.

 I think the technically correct "legal" approach is to create a
 set of header files which contain only user level APIs, release those
 under the LGPL as part of the kernel tree (I don't know if they need
 to be dual licensed LGPL/GPL but I think not) and maintain them as
 though they are the published, official API to the kernel.  Typically,
 structure definition, trival #define's and the like are not usually
 viewed as something that can be copyrighted (my believe, confirm with
 someone who actually knows something about copyright law).  Those
 new header files are not subject to copyright at least and I'm
 not sure that they would necessarily count as a "derived" work.

 This stuff is ugly.

gerrit

Re: Simple header cleanups

[Denis Vlasenko]

On Thursday 27 April 2006 06:31, Linus Torvalds wrote:
> 
> On Thu, 27 Apr 2006, David Woodhouse wrote:
> > 
> > Agreed. And distributions and library maintainers _will_ fix them. Are
> > we to deny those people the tools which will help them to keep track of
> > our breakage and submit patches to fix it?
> 
> No. As mentioned, as long as the target audience is distributions and 
> library maintainers, I definitely think we should do help them as much as 
> possible. Our problems have historically been "random people" who have 
> /usr/include/linux being the symlink to "kernel source of the day", which 
> is an unsupportable situation.

Maybe we should have a script which processes kernel's include/linux/*
files and produces sanitized set of headers (by deleting
"#ifdef __KERNEL__" blocks, etc), which will be treated at
*the* official kernel<->userspace API and will be used by glibc etc?

Such "kernel header sanitizator" script is to be maintained
and distributed as part of kernel tree.

glibc people will then know where to look/what part of kernel to patch
if they will stumble on incorrect sanitized header. I.e., they will need
to either fix original kernel header so that translation of it becomes ok
(for example, a missing #ifdef __KERNEL) or by modifying
sanitizator script.
--
vda

Re: Simple header cleanups

[Arjan van de Ven]

On Thu, 2006-04-27 at 10:10 +0300, Denis Vlasenko wrote:
> On Thursday 27 April 2006 06:31, Linus Torvalds wrote:
> > 
> > On Thu, 27 Apr 2006, David Woodhouse wrote:
> > > 
> > > Agreed. And distributions and library maintainers _will_ fix them. Are
> > > we to deny those people the tools which will help them to keep track of
> > > our breakage and submit patches to fix it?
> > 
> > No. As mentioned, as long as the target audience is distributions and 
> > library maintainers, I definitely think we should do help them as much as 
> > possible. Our problems have historically been "random people" who have 
> > /usr/include/linux being the symlink to "kernel source of the day", which 
> > is an unsupportable situation.
> 
> Maybe we should have a script which processes kernel's include/linux/*
> files and produces sanitized set of headers (by deleting
> "#ifdef __KERNEL__" blocks, etc), which will be treated at
> *the* official kernel<->userspace API and will be used by glibc etc?
> 
> Such "kernel header sanitizator" script is to be maintained
> and distributed as part of kernel tree.

this is what David is doing btw ;)

Re: Simple header cleanups

[Krzysztof Halasa]

Denis Vlasenko <vda@ilport.com.ua> writes:

> Maybe we should have a script which processes kernel's include/linux/*
> files and produces sanitized set of headers (by deleting
> "#ifdef __KERNEL__" blocks, etc), which will be treated at
> *the* official kernel<->userspace API and will be used by glibc etc?

It would really be better to create "ABI/API" header set in the
kernel at last. Glibc people, Linux-only kernel utils and of course
the rest of the kernel could just #include the API files (a copy
probably).

We _do_ have a stable userspace ABI and API, and contrary to the popular
belief the header set would be quite small (probably an order of magnitude
smaller than the "sanitized headers" package). What we need to export is
the actual API (including things like ioctl, netlink, SG_IO etc), most
things should be kept private to the kernel (and never copied anywhere
etc).

I realize this is completely uninteresting job. Perhaps someone wants
to pay a bit for having done it?

Why do we need, for example, /usr/include/linux/sc26198.h (FC5's
glibc-kernheaders-3.0-5.2)?

 *      sc26198.h  -- SC26198 UART hardware info.
...
/*
 *      Global register definitions. These registers are global to each 26198
 *      device, not specific ports on it.
 */
#define TSTR            0x0d
#define GCCR            0x0f
#define ICR             0x1b
-- 
Krzysztof Halasa

Re: Simple header cleanups

[Arnd Bergmann]

On Thursday 27 April 2006 10:49, Krzysztof Halasa wrote:
> We _do_ have a stable userspace ABI and API, and contrary to the popular
> belief the header set would be quite small (probably an order of magnitude
> smaller than the "sanitized headers" package). What we need to export is
> the actual API (including things like ioctl, netlink, SG_IO etc), most
> things should be kept private to the kernel (and never copied anywhere
> etc).

Yes, that is also what the 'make headers_install' target in David's tree
does (not the one proposed for inclusion here). It installs only headers
that are meaningful to glibc and the few other traditional users and strips
out the parts that are not inside __KERNEL__.

	Arnd <><

Re: Simple header cleanups

[David Woodhouse]

On Wed, 2006-04-26 at 19:18 -0700, Linus Torvalds wrote:
> On Thu, 27 Apr 2006, David Woodhouse wrote:
> >
> > Andrew (or preferably Linus since these are fairly simple and
> > unintrusive bug fixes), please pull from my tree at 
> > git://git.infradead.org/hdrcleanup-2.6.git
> 
> Hmm. Every time we've done this in the past, something has broken, so
> I'd actually _much_ rather wait until early in the 2.6.18 cycle than 
> do it now.

For -mm then please Andrew? Trees are
git://git.infradead.org/hdrcleanup-2.6.git for the header cleanups
and git://git.infradead.org/hdrinstall-2.6.git for the 'make
headers_install'.

This has been compile-tested on ppc, ppc64, x86_64, i686, ia64, s390 and
s390x. The exported headers have been used as the basis for a
glibc-kernheaders package and then to build glibc on the same set of
architectures.

Btw, I note there's no rbtree-git in your 2.6.17-rc2-mm1. Anything I can
do to remedy that? Did I just imagine that you had included it in a
previous -mm?

-- 
dwmw2

Re: Simple header cleanups

[Adrian Bunk]

On Wed, Apr 26, 2006 at 08:31:00PM -0700, Linus Torvalds wrote:
> 
> On Thu, 27 Apr 2006, David Woodhouse wrote:
> > 
> > Agreed. And distributions and library maintainers _will_ fix them. Are
> > we to deny those people the tools which will help them to keep track of
> > our breakage and submit patches to fix it?
> 
> No. As mentioned, as long as the target audience is distributions and 
> library maintainers, I definitely think we should do help them as much as 
> possible. Our problems have historically been "random people" who have 
> /usr/include/linux being the symlink to "kernel source of the day", which 
> is an unsupportable situation.
>...


A definition of the kernel <-> userspace ABI is required.

Currently, each distribution does it slightly different.

Currently we might accidentially touch the userspace ABI of the kernel 
without even noticing it.


I'd like to do the following:

Create an include/kabi/linux/ with the following properties:
- the goal is that include/kabi/linux an be copied verbatim to
  /usr/include/ (by distributions and library maintainers, normal users
  should use the copy installed by their distribution in /usr/include/)
- kernel code outside include/ wouldn't notice, since for each ABI 
  header include/kabi/linux/foo.h there's a header include/linux/foo.h 
  that does #include <kabi/linux/foo.h>
- moving stuff to kabi/linux can happen incrementially
- once the move is complete, we can announce this as the official ABI 
  headers
- moving to the kabi/ headers shouldn't cause compile time or
  runtime breakages for userspace software [1]


I'd be willing to work on this, but if you have any problems with this 
approach, I'd like to understand them


> 		Linus


cu
Adrian

[1] with the exception of header abuses like the MySQL asm/atomic.h usage

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Simple header cleanups

[Linus Torvalds]

On Thu, 27 Apr 2006, Adrian Bunk wrote:
> 
> A definition of the kernel <-> userspace ABI is required.

Well, we can get certain hints by just looking at every single type that 
is used as a __user pointer. That should give us a lot of the type 
information.

The other big piece ends up being argument values passed in to system 
calls, most notably ioctl numbers, but there are certainly others too.

And then there are the system call numbers themselves, and their calling 
conventions (fairly small part).

> Create an include/kabi/linux/ with the following properties:

I do hate your naming.

Why is that "linux" there? We're not going to have FreeBSD kabi files. And 
what about the (pretty common) architecture-specific ones?

The dependency chain is also quite often nontrivial. The ABI's all end up 
depending on the basic types, and often on each other (eg the ioctl 
numbers depend on the sizes of all the structures, which in turn depend on 
the architecture-specific structure layout and low-level types).

So it's _not_ usually possible to just do one file that does one thing, 
because they do actually have linkages.

And the linkages can be nasty, because they can easily be linkages that 
POSIX - and other standards - forbid them from being visible (you cannot 
expose certain typenames if they weren't _explicitly_ included, regardless 
of whether you need the type defines).

This is one reason why we shouldn't even _plan_ on having header files 
that can just be _directly_ used by the C libraries etc, even if it's just 
a "small" kernel ABI header.

Selling it as that kind of idea will inevitably mean that we then get 
blamed for not knowing magic rule #579 for SuS v2.1.6 subsection 8(a).

And if we say "you can use these headers unmodified", that _is_ what we're 
going to get blamed for. I'm so _not_ interested in having to care or 
worry.

So I seriously think we should aim for making it _easier_ for system 
libraries to get the information, but we should at the same time make it 
clear that we make it easier for them to get the basic info, BUT WE DO NOT 
CARE ABOUT THE RANDOM USER STANDARD OF THE DAY.

Have you looked in /usr/include lately? Have you _looked_ at the "expose 
BSD names" vs "GNU extended source" vs "strict POSIX" vs 
"_XOPEN_SOURCE==600" bs "_USE_MISC" vs a million random and strange 
things?

The day I see somebody adding crap like that to the kernel headers is the 
day I pull the plug on any "KABI" interfaces. 

And don't tell me this has got nothing to do with the kernel constants. Go 
look in something like /usr/include/bits/fcntl.h, and cry. See how it's 
using _exactly_ the kernel constants, but it has added all the random 
standard-of-the-day #ifdef (whether real standards, or the "GNU standards" 
or just "legacy BSD-like" etc).

And THAT is why I don't think the simplistic "kabi" directory approach 
that people have brought up many times over many years is actually 
realistic. People don't realize that glibcs makes "struct flock" actually 
look different in user space depending on whether "__USE_FILE_OFFSET64" is 
defined or not.

You just haven't seen just how NASTY those user-space headers are. They 
can't use _any_ kernel headers directly, because even when they want a 
_raw_ kernel data structure, they actually end up doing things differently 
in the _middle_ of that data structure. 

Really.

So we should try to help those system libc people perhaps _find_ the 
values and structures they need, but no, I will _never_ allow the kernel 
headers to be used directly. And it doesn't _matter_ if they've been moved 
to a "kabi" subdirectory. That's not the issue. The issue is that user 
space does insane things that aren't acceptable in kernel space.

			Linus

Re: Simple header cleanups

[David Woodhouse]

On Thu, 2006-04-27 at 15:00 -0700, Linus Torvalds wrote:
> This is one reason why we shouldn't even _plan_ on having header files 
> that can just be _directly_ used by the C libraries etc, even if it's just 
> a "small" kernel ABI header.
> 
> Selling it as that kind of idea will inevitably mean that we then get 
> blamed for not knowing magic rule #579 for SuS v2.1.6 subsection 8(a).
> 
> And if we say "you can use these headers unmodified", that _is_ what we're 
> going to get blamed for. I'm so _not_ interested in having to care or 
> worry. 

OK... I wasn't going to go here, because I don't want a potentially
contentious discussion to get in the way of the other cleanups which
I've been doing. We made that mistake at least year's Kernel Summit, and
we've made no progress since then despite having an almost unanimous
consensus on where we need to go.

But since Adrian's drawn you onto the topic, and since you've given me
an insight into what your objection is, I'll respond...

I don't think anyone imagines that a potential 'kabi' directory would be
used to specify anything like a _library_ interface, or that random
userspace programs would use stuff like 'struct stat' directly from it.
That would be like going back to the dark old days of libc5, and makes
me shudder as much as it does you.

Glibc has all that crap to define the interface between it and _its_
users. Its version of stuff like 'struct stat' may indeed vary according
to __USE_FILEOFFSET64 and the random standard of the day. We certainly
don't want to get involved with that kind of mess, I agree.

Our headers describe only _our_ interface to userspace, which often
bears no relation to glibc's interface to its users. We don't need to
care about our headers being usable in general by userspace
applications; only that they're useful for glibc. (With the exception of
stuff which defines ioctls, sockopts, etc.; obviously those are used by
system libraries and utilities rather than just glibc, but still -- it's
not just "random cross-platform POSIX userspace".)

Nobody _wants_ the hellish things you seem concerned about. All we want
to do is _think_ a little bit about avoiding gratuitous namespace
pollution in the headers -- for example, we should use only types like
__u32 instead of uint32_t and u32, we should refrain from exposing
private helper inlines, etc.

That's precisely what my cleanups have been fixing. It's not about
conforming to POSIX and other random standards so that our headers are
'clean' for direct inclusion as part of standard headers. This is _our_
interface, and our definition of it. It's just a case of being a little
bit disciplined about it rather than polluting it with internal crap.

Distinct from the cleanups, my existing 'make headers_install' takes the
limited set of headers which actually contain user-visible stuff, and
runs them through unifdef to get a consistent set of headers which all
distributions can use. The cleanups are just what was necessary to get
the resulting exported headers to the point where I can build glibc
against them on ppc, ppc64, i386, x86_64, s390, s390x and ia64. That's a
massive step forward, and it's all I really want to concentrate on right
now rather than talking about any potential 'next step'. What I'm doing
now stands on its own and is necessary.

However, there does seem to be a consensus that ifdefs are best avoided,
and that we should avoid them by splitting private stuff into one file,
public stuff into another, then including the latter from the former. I
haven't done that in the hdrcleanup-2.6.git tree; I've mostly just moved
stuff inside _existing_ '#ifdef __KERNEL__' guards within the same file.
But I _have_ done it in the past for include/linux/mtd/ (and put the
user-visible bits into linux/mtd/) and it's _much_ nicer that way.

_If_ we do that kind of thing, then it also makes some sense for the
user-visible files to be in one directory, and the private files to be
in another directory. That's just good practice.

That's all Adrian and others are talking about when they talk of a 'kabi
directory', I believe -- not your dystopian vision of trying to make our
headers conform to random standards-du-jour, but just treating them the
the same as we try to do in the actual C files through out the kernel --
avoiding ifdefs by splitting stuff into separate files, and grouping
those files into appropriate directories.

I think they're right, but I'm not arguing that point right now because
it's secondary to my task at hand, which is identifying the private vs.
public stuff in the first place and reducing the pollution so that our
headers are actually usable to build glibc again. In particular, I don't
want that discussion to get in the way of what I'm doing right now.

-- 
dwmw2

Re: Simple header cleanups

[Adrian Bunk]

On Thu, Apr 27, 2006 at 03:00:16PM -0700, Linus Torvalds wrote:
> 
> On Thu, 27 Apr 2006, Adrian Bunk wrote:
> > 
> > A definition of the kernel <-> userspace ABI is required.
> 
> Well, we can get certain hints by just looking at every single type that 
> is used as a __user pointer. That should give us a lot of the type 
> information.
> 
> The other big piece ends up being argument values passed in to system 
> calls, most notably ioctl numbers, but there are certainly others too.
> 
> And then there are the system call numbers themselves, and their calling 
> conventions (fairly small part).

Currently, it's sometimes non-trivial to figure out what is part of the 
ABI and what is not.

E.g. if you want to change a struct, how do you know whether it's part 
of the userspace ABI?

> > Create an include/kabi/linux/ with the following properties:
> 
> I do hate your naming.
> 
> Why is that "linux" there? We're not going to have FreeBSD kabi files. And 
> what about the (pretty common) architecture-specific ones?

Agreed.

But let's discuss the naming after this discussion.

> The dependency chain is also quite often nontrivial. The ABI's all end up 
> depending on the basic types, and often on each other (eg the ioctl 
> numbers depend on the sizes of all the structures, which in turn depend on 
> the architecture-specific structure layout and low-level types).
> 
> So it's _not_ usually possible to just do one file that does one thing, 
> because they do actually have linkages.

Sure.

> And the linkages can be nasty, because they can easily be linkages that 
> POSIX - and other standards - forbid them from being visible (you cannot 
> expose certain typenames if they weren't _explicitly_ included, regardless 
> of whether you need the type defines).
> 
> This is one reason why we shouldn't even _plan_ on having header files 
> that can just be _directly_ used by the C libraries etc, even if it's just 
> a "small" kernel ABI header.
> 
> Selling it as that kind of idea will inevitably mean that we then get 
> blamed for not knowing magic rule #579 for SuS v2.1.6 subsection 8(a).
> 
> And if we say "you can use these headers unmodified", that _is_ what we're 
> going to get blamed for. I'm so _not_ interested in having to care or 
> worry.
> 
> So I seriously think we should aim for making it _easier_ for system 
> libraries to get the information, but we should at the same time make it 
> clear that we make it easier for them to get the basic info, BUT WE DO NOT 
> CARE ABOUT THE RANDOM USER STANDARD OF THE DAY.
> 
> Have you looked in /usr/include lately? Have you _looked_ at the "expose 
> BSD names" vs "GNU extended source" vs "strict POSIX" vs 
> "_XOPEN_SOURCE==600" bs "_USE_MISC" vs a million random and strange 
> things?
> 
> The day I see somebody adding crap like that to the kernel headers is the 
> day I pull the plug on any "KABI" interfaces. 
> 
> And don't tell me this has got nothing to do with the kernel constants. Go 
> look in something like /usr/include/bits/fcntl.h, and cry. See how it's 
> using _exactly_ the kernel constants, but it has added all the random 
> standard-of-the-day #ifdef (whether real standards, or the "GNU standards" 
> or just "legacy BSD-like" etc).
> 
> And THAT is why I don't think the simplistic "kabi" directory approach 
> that people have brought up many times over many years is actually 
> realistic. People don't realize that glibcs makes "struct flock" actually 
> look different in user space depending on whether "__USE_FILE_OFFSET64" is 
> defined or not.
> 
> You just haven't seen just how NASTY those user-space headers are. They 
> can't use _any_ kernel headers directly, because even when they want a 
> _raw_ kernel data structure, they actually end up doing things differently 
> in the _middle_ of that data structure. 
> 
> Really.
> 
> So we should try to help those system libc people perhaps _find_ the 
> values and structures they need, but no, I will _never_ allow the kernel 
> headers to be used directly. And it doesn't _matter_ if they've been moved 
> to a "kabi" subdirectory. That's not the issue. The issue is that user 
> space does insane things that aren't acceptable in kernel space.

I do still not get your point.

The ABI headers will be used by C libraries.

And by some programs doing low-level Linux specific things - but these 
are exceptions.

Normal userspace programs will simply not care about the contents of the 
kernel ABI headers - the libc will present them pretty headers adhering 
to all past, current and future standards.

This works today and it will continue to work.

> 			Linus

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Simple header cleanups

[Linus Torvalds]

On Fri, 28 Apr 2006, Adrian Bunk wrote:
> 
> I do still not get your point.
> 
> The ABI headers will be used by C libraries.
> 
> And by some programs doing low-level Linux specific things - but these 
> are exceptions.
> 
> Normal userspace programs will simply not care about the contents of the 
> kernel ABI headers - the libc will present them pretty headers adhering 
> to all past, current and future standards.

As long as that is clear - that the kernel ABI headers are _never_ seen by 
normal programs, even indirectly #include'd from the standard include 
headers.

That means that any library implementation still needs to basically 
_duplicate_ all the kernel ABI structures etc. 

Last I heard, a lot of people wanted to avoid that duplication, and 
actually wanted the kabi headers exactly because they wanted just _one_ 
place for these things.

And I suspect a lot of people still think the kabi headers would be 
exactly that: things that get #include'd indirectly from the normal header 
files.

My point would be that we can't do that. Ever. 

As long as people realize that any kabi headers would only ever be used by 
system libraries _internally_ to build themselves (or strange system 
tools, of course), then I'm happy. I just get the feeling that people 
don't always realize that, and they really want to see it as some kind of 
"/usr/include/bits" kind of thing.

			Linus

Re: Simple header cleanups

[David Woodhouse]

On Thu, 2006-04-27 at 17:01 -0700, Linus Torvalds wrote:
> As long as people realize that any kabi headers would only ever be used by 
> system libraries _internally_ to build themselves (or strange system 
> tools, of course), then I'm happy. I just get the feeling that people 
> don't always realize that, and they really want to see it as some kind of 
> "/usr/include/bits" kind of thing.

Absolutely. There's no justification for kernel headers being treated
like that -- and one of my goals is to _reduce_ the amount they get
abused in that fashion. Certainly nobody wants to allow the abuse to
increase.

And of _course_ there are people who don't fully grasp the situation.
We've been entirely inconsistent about it with the "Never use kernel
headers" mantra, when we _know_ it actually has to be done in certain
special circumstances.

That's _why_ I want a 'make headers_install' which copies out (and
unifdefs) a strict subset of headers which were chosen by hand -- it
makes it very clear what is permitted, and what is not. Traditionally,
distributions have been far too liberal about what they've exposed, and
it's led to people doing insanely broken things like using
<asm/atomic.h> even though it isn't even actually atomic on i386 in
userspace. That needs to stop.

There is a small set of headers which we _need_ to export, for the
benefit of glibc, gdb and some Linux-specific tools which use ioctls and
sockopts. Nothing else should be exposed; random userspace apps
shouldn't _want_ to include kernel headers.

The point of what I'm doing is to be strict about what we export, to
_discourage_ the abuse of stuff which shouldn't really have been visible
in the first place. And to be _consistent_ across Linux distributions
about what is available and what is not. (With the vast majority being
'NOT', because that's just sane.)

That's what git://git.infradead.org/hdrinstall-2.6.git is about.


As a separate quality of implementation issue, those headers which we
_do_ need to export should be exported in a form which is actually
useful, and which doesn't require each distributor to maintain their own
fork by hand or use evil sed tricks like this one the Gentoo maintainer
showed me...

headers___fix() {
	# Voodoo to partially fix broken upstream headers.
	# Issues with this function should go to plasmaroo.
	sed -i \
		-e "s/\([ "$'\t'"]\)\(u\|s\)\(8\|16\|32\|64\)\([ "$'\t'"]\)/\1__\2\3\4/g;" \
		-e 's/ \(u\|s\)\(8\|16\|32\|64\)$/ __\1\2/g' \
		-e 's/\([(, ]\)\(u\|s\)64\([, )]\)/\1__\264\3/g' \
		-e "s/^\(u\|s\)\(8\|16\|32\|64\)\([ "$'\t'"]\)/__\1\2\3/g;" \
		-e "s/ inline / __inline__ /g" \
		"$@"

That's what git://git.infradead.org/hdrcleanup-2.6.git is about. It's
just simple fixes.


What Adrian brings up is a _third_ issue, one which is fairly
uncontentious in the context of C files: that ifdefs are generally bad
and should be avoided. Instead of using ifdefs, it's nicer to split
stuff into separate files. Also, files should be grouped together into
directories as appropriate for their purpose. 

In the specific case of kernel headers, the consensus seems to be that
we should tend to avoid __KERNEL__ by making separate 'public' files for
those bits which _have_ to be public -- we can include those public
files from other, entirely-private, files. And there also seems to be
agreement that those 'public' files should live in a directory together
where we can immediately see when we're changing the ABI, etc. For want
of a better name, people have referred to this as a 'kabi directory'.

I've been avoiding this third issue since it seems to cause you a lot of
concern -- perhaps because we'd managed to miscommunicate the intent,
and we shouldn't have said "public" when we do really mean it's only for
Linux-specific libraries and tools, not 'public' for general
applications. I've also been avoiding it because with my new 'make
headers_install' it isn't _quite_ so much of an issue as it once was --
although I do still agree with the general consensus that it's probably
a good idea anyway.

-- 
dwmw2

Re: Simple header cleanups

[Jan Engelhardt]

>
>Last I heard, a lot of people wanted to avoid that duplication, and 
>actually wanted the kabi headers exactly because they wanted just _one_ 
>place for these things.
>
Sounds like they want it BSD-style. Do they realize that?
New release, new headers, making it necessary to recompile every app, 
because a struct could have changed. That'd seriously impact 
compatibility.


Jan Engelhardt
-- 

Re: Simple header cleanups

[David Woodhouse]

On Fri, 2006-04-28 at 17:32 +0200, Jan Engelhardt wrote:
> Sounds like they want it BSD-style. Do they realize that?
> New release, new headers, making it necessary to recompile every app, 
> because a struct could have changed. That'd seriously impact 
> compatibility.

Utter crap.

We don't _change_ any of the structs which would be exposed in such
files (i.e. the structs which should be outside the #ifdef __KERNEL__ at
the moment, because that would mean we break userspace binary
compatibility from kernel to kernel.

We absolutely do _NOT_ want to go there. We're talking about cleaning up
the existing mess, not starting a crack habit.

-- 
dwmw2

Re: Simple header cleanups

[Arjan van de Ven]

On Fri, 2006-04-28 at 16:37 +0100, David Woodhouse wrote:
> On Fri, 2006-04-28 at 17:32 +0200, Jan Engelhardt wrote:
> > Sounds like they want it BSD-style. Do they realize that?
> > New release, new headers, making it necessary to recompile every app, 
> > because a struct could have changed. That'd seriously impact 
> > compatibility.
> 
> Utter crap.
> 
> We don't _change_ any of the structs which would be exposed in such
> files (i.e. the structs which should be outside the #ifdef __KERNEL__ at
> the moment, because that would mean we break userspace binary
> compatibility from kernel to kernel.
> 
> We absolutely do _NOT_ want to go there. We're talking about cleaning up
> the existing mess, not starting a crack habit.
> 

btw one advantage of having the user visible structures in their own
headers is that it becomes immediately more obvious that it IS a shared
struct and any reviewer can then scream about the ABI break.,,,

Re: Simple header cleanups

[Kyle Moffett]

On Apr 28, 2006, at 11:37:03, David Woodhouse wrote:
> On Fri, 2006-04-28 at 17:32 +0200, Jan Engelhardt wrote:
>> Sounds like they want it BSD-style. Do they realize that?  New  
>> release, new headers, making it necessary to recompile every app,  
>> because a struct could have changed. That'd seriously impact  
>> compatibility.
>
> Utter crap.
>
> We don't _change_ any of the structs which would be exposed in such  
> files (i.e. the structs which should be outside the #ifdef  
> __KERNEL__ at the moment, because that would mean we break  
> userspace binary compatibility from kernel to kernel.
>
> We absolutely do _NOT_ want to go there. We're talking about  
> cleaning up the existing mess, not starting a crack habit.

Example:  Say the proposed utsname virtualization gets a new clone  
flag (CLONE_NEWUTSNAME).  Old headers would obviously not include the  
CLONE_NEWUTSNAME define.  If you want to compile a glibc that uses  
CLONE_NEWUTSNAME, you need to build it against a set of the "kabi  
headers" from a version of the kernel that includes the feature.  The  
resulting glibc will _run_ on any kernel; if you call clone 
(CLONE_NEWUTSNAME) on an older kernel it would return EINVAL but work  
otherwise.  A glibc that does *not* use CLONE_NEWUTSNAME would build  
an identical binary with older versions and newer versions.  This is  
_exactly_ the way things work now, except there is no outside llh  
project, it's all stored in the kernel tree.

Cheers,
Kyle Moffett

Re: Simple header cleanups

[Krzysztof Halasa]

Kyle Moffett <mrmacman_g4@mac.com> writes:

> This is
> _exactly_ the way things work now, except there is no outside llh
> project, it's all stored in the kernel tree.

Precisely. I also think we don't have to worry about namespace
contamination and non-compliance to random "standards". It is the
userland _using_ the kernel headers at its discretion, we don't force
anyone to include anything. If something includes, say, kabi/hdlc.h
(just to make sure I know the details) then it has to be prepared
for hdlc_device struct to be defined, even if System 5 3/4 R 1.2.3.4
says nothing about it.

If glibc (as opposed to kernel utils) wants to use the kabi headers
(and I presume it should) then it has to build its own files based
on ours and/or should do other necessary steps to prevent namespace
problems (such as carefully picking files to include instead of
#include <kabi/*>). It will be easier than now anyway and I think
we're all open for suggestions.

If it turns out that we need a separate kabi/*.h file for every kernel
API structure or a set of #defines then let it be.
-- 
Krzysztof Halasa

Re: Simple header cleanups

[Peter Chubb]

>>>>> "Linus" == Linus Torvalds <torvalds@osdl.org> writes:

Linus> On Thu, 27 Apr 2006, Adrian Bunk wrote:
>>  A definition of the kernel <-> userspace ABI is required.


Linus> This is one reason why we shouldn't even _plan_ on having
Linus> header files that can just be _directly_ used by the C
Linus> libraries etc, even if it's just a "small" kernel ABI header.

I think I disagree with you here.  It should be possible to have Linux
kernel abi header files that are directly usable, and shared between
kernel and user space, precisely because they are _Linux_ kernel ABI
headers, not POSIX, not SUS, not XOPEN_SOURCE.  The consumers of the
header files shouldn't expect to be able necessarily to do 
       #include <stdio.h>
       $include <linux/kabi/xxx.h>
and have it work (although I think we should avoid breaking things if
it's easy to do so), because that's mixing interface definitions --
libc vs raw Linux.

Originally (back in edition 6 days) /usr/include/sys was precisely the
kind of thing that's being proposed, in that it contained system call
numbers, and the shapes of structures shared between user and kernel
space.  Over time, that directory became a compatibility layer for
POSIX-like systems, and has been implemented in terms of whatever the
OS provides.  So now we need something new.

-- 
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au           ERTOS within National ICT Australia

Re: Simple header cleanups

[David Woodhouse]

On Sun, 2006-04-30 at 19:38 +1000, Peter Chubb wrote:
> So now we need something new.

No we don't. We need only what we've already got. Our headers are
perfectly sufficient in principle -- we're already _relatively_ careful
not to pollute them with random kernel-private type and other stuff.
It's just that we need to be a little more competent at that, and that's
why we need the set of cleanups I've made and a tool to make life
easier. There's nothing particularly exciting to see here; there's no
need for any more fundamental change in the way things work.

Eliminating ifdefs and moving stuff into appropriate directories might
make it even easier still, if we do end up going that far. Certainly
it's the kind of cleanup which is _welcomed_ in C files, although
headers seem to be a touchy subject so I'm avoiding that question for
now.

But if we keep talking like this about crap like random userland
namespaces then I have a feeling Linus is just going to shut down and
not even take the normal, uncontentious cleanup patches which I was
careful to limit myself to because I didn't want to have wasted my time.

-- 
dwmw2

Re: Simple header cleanups

[Peter Chubb]

>>>>> "David" == David Woodhouse <dwmw2@infradead.org> writes:

David> On Sun, 2006-04-30 at 19:38 +1000, Peter Chubb wrote:
>> So now we need something new.

David> No we don't. 

I meant, we can no longer use /usr/include/sys.  In the early days of
linux, we used /usr/include/{linux,asm}, but that was stopped for good
reasons.  FWIW, I think your cleanups are a step in the right direction.

-- 
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au           ERTOS within National ICT Australia

Re: Simple header cleanups

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.64.0604271656390.3701@g5.osdl.org>
By author:    Linus Torvalds <torvalds@osdl.org>
In newsgroup: linux.dev.kernel
> > 
> > Normal userspace programs will simply not care about the contents of the 
> > kernel ABI headers - the libc will present them pretty headers adhering 
> > to all past, current and future standards.
> 
> As long as that is clear - that the kernel ABI headers are _never_ seen by 
> normal programs, even indirectly #include'd from the standard include 
> headers.
> 

For some of them, there isn't really any reason not to; in particular
when it comes to things like ioctl constants and structures that
describe what is functionally an RPC interface between the user-space
application and the kernel -- the library couldn't realistically
intervene even if it wanted to.

This is also by orders of magnitude the fastest evolving part of the
kernel ABI.

When it comes to more serious stuff like "struct stat" it should be
pretty obvious any "struct stat" the kernel exports is going to be
wrong.  klibc, which cares about size and not ABI stability, could
export it directly, but it still wants to be able to *choose* which
"struct stat" to use -- on most platform, it's what the kernel calls
"struct stat64", except with a 32-bit dev_t instead of a 64-bit
unsigned long long for st_[r]dev, plus of course appropriate padding.

One solution would of course be to have a purely declarative language
for that stuff.  A C program won't use it because a C program *can't*
use it without further processing, and the declarative language
processor is the ideal place for such stuff.  In that scenario even
the kernel would have to use a processed version of the ABI
declarations.

An intermediate solution might be to take sparse, and something like
it, and use it as the declarative language processor.  Then the kernel
could still use them as C, but the library would have an easy way to
process them.  The risk is that something might not.

What David has done is yet one step down from this in ambition level,
but at least he's gotten the ball rolling.

	-hpa