Header files and the kernel ABI

Header files and the kernel ABI

[H. Peter Anvin]

OK... I have had a thought grinding in my head for a while, and wanted
to throw it out for everyone to think about...

In the libc4/libc5 days, we attempted to use kernel headers in user
space.  This was a total mess, not the least because the kernel
headers tended to pull in a lot of other kernel headers, and the
datatypes were unsuitable to have spead all across userspace.

In glibc, the official rule is "don't use kernel headers."  This
causes problems, because certain aspects of the kernel ABI is only
available through the include files, and reproducing them by hand is
tedious and error-prone.

I'm in the process of writing a very tiny libc for initramfs, and will
likely have to deal with how to use the kernel ABI as well.

It seems to me that a reasonable solution for how to do this is not
for user space to use kernel headers, but for user space and the
kernel to share a set of common ABI description files[1].  These files
should be highly stylized, and only describe things visible to user
space.  Furthermore, if they introduce types, they should use the
already-established __kernel_ namespace, and of course __s* and __u*
could be used for specific types.

This means that we would be able to get rid of #if(n)def __KERNEL__ in
the main kernel header files, because there would be a separation by
file location -- something in the main kernel include files could
include the ABI description files, but the opposite should never be
true.

I would like to propose that these files be set up in the #include
namespace as <linux/abi/*>, with <linux/abi/arch/*> for any
architecture-specific support files (I do believe, however, that those
files should only be included by files in the linux/abi/ root.  This
probably would be a symlink to ../asm/abi in the kernel sources,
unless we change the kernel include layout.)  The linux/ namespace is
universally reserved for the kernel, and unlike <abi/*> I don't know
of any potential conflicts.  I was considered <kabi/*>, but it seems
cleaner to use existing namespace.

If people think this is an idea, I will try to set up the
infrastructure as part of my work on klibc, although I'm definitely
not going to be able to migrate every portion of every include file
that needs to be migrated all by myself.

Thoughts?

	-hpa



[1] I'm assuming here they are C include files, just because it's a
common language to everyone; however, it would be possible to create
an "ABI description language" which would compile to C headers as well
as perhaps other formats (assembly language support files?), ...)
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Header files and the kernel ABI

[Andreas Dilger]

On Jul 24, 2002  23:28 -0700, H. Peter Anvin wrote:
> It seems to me that a reasonable solution for how to do this is not
> for user space to use kernel headers, but for user space and the
> kernel to share a set of common ABI description files[1].  These files
> should be highly stylized, and only describe things visible to user
> space.  Furthermore, if they introduce types, they should use the
> already-established __kernel_ namespace, and of course __s* and __u*
> could be used for specific types.
> 
> I would like to propose that these files be set up in the #include
> namespace as <linux/abi/*>, with <linux/abi/arch/*> for any
> architecture-specific support files (I do believe, however, that those
> files should only be included by files in the linux/abi/ root.  This
> probably would be a symlink to ../asm/abi in the kernel sources,
> unless we change the kernel include layout.)  The linux/ namespace is
> universally reserved for the kernel, and unlike <abi/*> I don't know
> of any potential conflicts.  I was considered <kabi/*>, but it seems
> cleaner to use existing namespace.

I had thought on this briefly in the past, and my take would be for these
ABI definition files to live directly in /usr/include/linux for user space
(just as glibc puts its own sanitized copy of the kernel headers there)
and the appropriate ABI headers are included as needed from the kernel.

The kernel side would be something like <linux/scsi.h> includes
<linux/abi/scsi.h> or whatever, but in the future this can be included
directly as needed throughout the kernel.  The existing kernel
<linux/*.h> headers would also have extra kernel-specific data in them.

The same could be done with the user-space headers, but I think that
is missing the point that the linux/abi/*.h headers should define _all_
of the abi, so we may as well just use that directly.

Essentially "all" this would mean is that we take the existing headers,
remove everything which is inside #ifdef __KERNEL__ (and all of the
other kernel-specific non-abi headers that are included) and we are
done.  The kernel header now holds only things that were inside the
#ifdef __KERNEL__ (or should have been), and #include <linux/abi/foo.h>.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Header files and the kernel ABI

[H. Peter Anvin]

Andreas Dilger wrote:
> 
> I had thought on this briefly in the past, and my take would be for these
> ABI definition files to live directly in /usr/include/linux for user space
> (just as glibc puts its own sanitized copy of the kernel headers there)
> and the appropriate ABI headers are included as needed from the kernel.
> 

Given no legacy, I would agree with this, but I think it would imply bad 
legacy both on the kernel and the libc (it's not just glibc, either) side.

> The kernel side would be something like <linux/scsi.h> includes
> <linux/abi/scsi.h> or whatever, but in the future this can be included
> directly as needed throughout the kernel.  The existing kernel
> <linux/*.h> headers would also have extra kernel-specific data in them.
> 
> The same could be done with the user-space headers, but I think that
> is missing the point that the linux/abi/*.h headers should define _all_
> of the abi, so we may as well just use that directly.

Except now the paths are gratuitously different between kernel 
programming and non-kernel programming, and we create a much harder 
migration problem.  I'd rather leave the linux/* namespace to the 
user-space libc to do whatever backwards compatibility cruft they may 
consider necessary, for example, <linux/io.h> might #include <sys/io.h> 
since some user space apps bogusly included the former name.  Leaving 
that namespace available for backwards compatibility hacks avoids those 
kinds of problems.

> Essentially "all" this would mean is that we take the existing headers,
> remove everything which is inside #ifdef __KERNEL__ (and all of the
> other kernel-specific non-abi headers that are included) and we are
> done.  The kernel header now holds only things that were inside the
> #ifdef __KERNEL__ (or should have been), and #include <linux/abi/foo.h>.

Exactly.

	-hpa

Re: Header files and the kernel ABI

[Andreas Dilger]

On Jul 25, 2002  00:07 -0700, H. Peter Anvin wrote:
> Andreas Dilger wrote:
> >The kernel side would be something like <linux/scsi.h> includes
> ><linux/abi/scsi.h> or whatever, but in the future this can be included
> >directly as needed throughout the kernel.  The existing kernel
> ><linux/*.h> headers would also have extra kernel-specific data in them.
> >
> >The same could be done with the user-space headers, but I think that
> >is missing the point that the linux/abi/*.h headers should define _all_
> >of the abi, so we may as well just use that directly.
> 
> Except now the paths are gratuitously different between kernel 
> programming and non-kernel programming, and we create a much harder 
> migration problem.  I'd rather leave the linux/* namespace to the 
> user-space libc to do whatever backwards compatibility cruft they may 
> consider necessary, for example, <linux/io.h> might #include <sys/io.h> 
> since some user space apps bogusly included the former name.  Leaving 
> that namespace available for backwards compatibility hacks avoids those 
> kinds of problems.

OK, but essentially then <linux/io.h> will be mostly a hollow shell
which includes <linux/abi/io.h> and maybe a couple other files
(e.g. <linux/abi/types.h> or similar).  If so, then great.

That brings up the question - how do you tie a particular
<linux/abi/*.h> to a particular kernel?  Should there be a bunch of
directories <linux/abi-2.4/*.h> and/or <linux/abi-2.4.12/*.h> and/or
<linux/abi-`uname -r`/*.h> or what?  While there are efforts to keep
the ABI constant for major stable releases, this is not always true,
so abi-2.4 will certainly not be enough.  Maybe linux/abi is a symlink
to the abi directory of currently running kernel?

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Header files and the kernel ABI

[Oliver Xymoron]

On Thu, 25 Jul 2002, Andreas Dilger wrote:

> That brings up the question - how do you tie a particular
> <linux/abi/*.h> to a particular kernel?  Should there be a bunch of
> directories <linux/abi-2.4/*.h> and/or <linux/abi-2.4.12/*.h> and/or
> <linux/abi-`uname -r`/*.h> or what?  While there are efforts to keep
> the ABI constant for major stable releases, this is not always true,
> so abi-2.4 will certainly not be enough.  Maybe linux/abi is a symlink
> to the abi directory of currently running kernel?

Ideally, the ABI layer would be maintained and packaged separately from
both the kernel and glibc to avoid gratuitous changes from either side.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Header files and the kernel ABI

[H. Peter Anvin]

Oliver Xymoron wrote:
> 
> Ideally, the ABI layer would be maintained and packaged separately from
> both the kernel and glibc to avoid gratuitous changes from either side.
> 

I disagree.  The ABI is a product of the kernel and should be attached 
to it.  It is *not* a product of glibc -- glibc is a consumer of it, as 
are any other libcs.

	-hpa

Re: Header files and the kernel ABI

[Erik Andersen]

On Thu Jul 25, 2002 at 09:31:23AM -0700, H. Peter Anvin wrote:
> Oliver Xymoron wrote:
> >
> >Ideally, the ABI layer would be maintained and packaged separately from
> >both the kernel and glibc to avoid gratuitous changes from either side.
> >
> 
> I disagree.  The ABI is a product of the kernel and should be attached 
> to it.  It is *not* a product of glibc -- glibc is a consumer of it, as 
> are any other libcs.

Agreed.  I maintain a libc and I certainly do not want to 
have to maintain the kernel ABI of the day headers.  That
is clearly a job for the kernel.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: Header files and the kernel ABI

[Oliver Xymoron]

On Thu, 25 Jul 2002, Erik Andersen wrote:

> On Thu Jul 25, 2002 at 09:31:23AM -0700, H. Peter Anvin wrote:
> > Oliver Xymoron wrote:
> > >
> > >Ideally, the ABI layer would be maintained and packaged separately from
> > >both the kernel and glibc to avoid gratuitous changes from either side.
> > >
> >
> > I disagree.  The ABI is a product of the kernel and should be attached
> > to it.  It is *not* a product of glibc -- glibc is a consumer of it, as
> > are any other libcs.
>
> Agreed.  I maintain a libc and I certainly do not want to
> have to maintain the kernel ABI of the day headers.  That
> is clearly a job for the kernel.

The idea of maintaining them separately is that people won't be able to
touch the ABI without explicitly going through a gatekeeper whose job is
to minimize breakage. Linus usually catches ABI changes but not always.

I explicitly did _not_ suggest making it the job of libc maintainers. And
the whole point of the exercise is to avoid ABI of the day anyway. The ABI
should change less frequently than the kernel or libc. It's more analogous
to something like modutils.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Header files and the kernel ABI

[Eric W. Biederman]

Oliver Xymoron <oxymoron@waste.org> writes:
> 
> The idea of maintaining them separately is that people won't be able to
> touch the ABI without explicitly going through a gatekeeper whose job is
> to minimize breakage. Linus usually catches ABI changes but not always.
> 
> I explicitly did _not_ suggest making it the job of libc maintainers. And
> the whole point of the exercise is to avoid ABI of the day anyway. The ABI
> should change less frequently than the kernel or libc. It's more analogous
> to something like modutils.

Except for ioctls.  Until we can get those under control the abi headers
need to remain part of the kernel.  Gatekeeping on the ioctls is something
we need.

And even if the code is part of the kernel, Linus can still delegate
the work of verifying it he wants.

Eric

Re: Header files and the kernel ABI

[H. Peter Anvin]

Andreas Dilger wrote:
> 
> That brings up the question - how do you tie a particular
> <linux/abi/*.h> to a particular kernel?  Should there be a bunch of
> directories <linux/abi-2.4/*.h> and/or <linux/abi-2.4.12/*.h> and/or
> <linux/abi-`uname -r`/*.h> or what?  While there are efforts to keep
> the ABI constant for major stable releases, this is not always true,
> so abi-2.4 will certainly not be enough.  Maybe linux/abi is a symlink
> to the abi directory of currently running kernel?
> 

Well... I guess that depends on what kind of changes we make.  In 
general, I belive linux/abi should be cumulative, i.e. it should 
describe "sys_stat" as well as "sys_oldstat" or whatever it is called. 
ABI changes are hard to deal with regardless; you never really know what 
you're breaking, and you probably have to deal with it on a case-by-case 
basis.  People should at least understand that they're breaking the ABI, 
which I'm not sure they currently are.

In general I believe linux/abi should come from the current kernel, but 
for obvious reasons that doesn't mean it's the kernel that's running 
when the application is actually being executed.  This sort of things 
apply to all ABI changes, inherently, which is why 
non-backwards-compatible ABI changes must be avoided.  Ultimately, 
though, it's up to the person who changes it to do the appropriate thing.

	-hpa

Re: Header files and the kernel ABI

[DervishD]

    Hi HPA :)

>I would like to propose that these files be set up in the #include
>namespace as <linux/abi/*>, with <linux/abi/arch/*> for any
>architecture-specific support files (I do believe, however, that those
>files should only be included by files in the linux/abi/ root.  This
>probably would be a symlink to ../asm/abi in the kernel sources,
>unless we change the kernel include layout.)

    I think this is a *really* great idea. It will solve lots of
problems IMHO. Now the glibc copies some of the kernel headers, and
in the future it will support more, but standarizing an ABI set of
headers will be better (IMHO again). Go for it ;)

    Raúl

Re: Header files and the kernel ABI

[Brad Hards]

On Thu, 25 Jul 2002 16:28, H. Peter Anvin wrote:
> It seems to me that a reasonable solution for how to do this is not
> for user space to use kernel headers, but for user space and the
> kernel to share a set of common ABI description files[1].  These files
> should be highly stylized, and only describe things visible to user
> space.  Furthermore, if they introduce types, they should use the
> already-established __kernel_ namespace, and of course __s* and __u*
> could be used for specific types.
I like it (having just argued for it), except for the __s* and __u*.
The ABI definitions aren't for kernel programmers. They are for 
userspace programmers. So we should use standard types,
even if they are a bit ugly (and uint16_t isn't really much uglier
than __u16, and at least it doesn't carry connotations of
something that is meant to be internal, which is what the standard
double-underscore convention means). 

Please, let us agree that the ABI definition should use standard
types wherever possible.

Brad

-- 
http://conf.linux.org.au. 22-25Jan2003. Perth, Australia. Birds in Black.

Re: Header files and the kernel ABI

[DervishD]

    Hi Brad & HPA :)

>> already-established __kernel_ namespace, and of course __s* and __u*
>> could be used for specific types.
>I like it (having just argued for it), except for the __s* and __u*.
[...]
>Please, let us agree that the ABI definition should use standard
>types wherever possible.

    I think so. Moreover, any user-space developer should be familiar
with them and anyone may even use the standard ones by mistake, so
it's better to get them equal and be standard.

    Raúl

Re: Header files and the kernel ABI

[H. Peter Anvin]

Brad Hards wrote:
> 
> I like it (having just argued for it), except for the __s* and __u*.
> The ABI definitions aren't for kernel programmers. They are for 
> userspace programmers. So we should use standard types,
> even if they are a bit ugly (and uint16_t isn't really much uglier
> than __u16, and at least it doesn't carry connotations of
> something that is meant to be internal, which is what the standard
> double-underscore convention means). 
> 

Not quite -- it means they are implementation-specific (in this case, 
Linux-specific.)  The Linux __s* and __u* predates <stdint.h>; I 
certainly would like to migrate to <stdint.h> but I don't see it as a 
very high priority.

	-hpa

Re: Header files and the kernel ABI

[Erik Andersen]

On Thu Jul 25, 2002 at 09:17:40AM -0700, H. Peter Anvin wrote:
> Brad Hards wrote:
> >
> >I like it (having just argued for it), except for the __s* and __u*.
> >The ABI definitions aren't for kernel programmers. They are for 
> >userspace programmers. So we should use standard types,
> >even if they are a bit ugly (and uint16_t isn't really much uglier
> >than __u16, and at least it doesn't carry connotations of
> >something that is meant to be internal, which is what the standard
> >double-underscore convention means). 
> >
> 
> Not quite -- it means they are implementation-specific (in this case, 
> Linux-specific.)  The Linux __s* and __u* predates <stdint.h>; I 
> certainly would like to migrate to <stdint.h> but I don't see it as a 
> very high priority.

Using stdint.h types at least in the kernel ABI headers would be
a _huge_ improvement!  I hate having to cut-n-paste kernel structs 
into my user-spave code and then rework the types so user-space 
code can compile things.  

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Kernel ABI BoF at Linux-Kongress? [was: Header files and the kernel ABI]

[Brad Hards]

On Thu, 25 Jul 2002 16:28, H. Peter Anvin wrote:
> OK... I have had a thought grinding in my head for a while, and wanted
> to throw it out for everyone to think about...
>
> In the libc4/libc5 days, we attempted to use kernel headers in user
> space.  This was a total mess, not the least because the kernel
> headers tended to pull in a lot of other kernel headers, and the
> datatypes were unsuitable to have spead all across userspace.
>
> In glibc, the official rule is "don't use kernel headers."  This
> causes problems, because certain aspects of the kernel ABI is only
> available through the include files, and reproducing them by hand is
> tedious and error-prone.
>
> I'm in the process of writing a very tiny libc for initramfs, and will
> likely have to deal with how to use the kernel ABI as well.
Well the initial enthusiasm appears to have passed, and nothing
is happening.

Is there interest in getting together to resolve this issue?

I note that the next major conference appears to be
Linux Kongress (Koln/Cologne) in early September. Maybe
we can get some glibc / other-libraries people there, and
make some progress.

If there is some interest (post off-list if preferred), I'll contact the
organisers and try to get a BoF together for this.

I'll look at linux.conf.au as an additional opportunity when
time gets a bit closer.

-- 
http://conf.linux.org.au. 22-25Jan2003. Perth, Australia. Birds in Black.

Re: Header files and the kernel ABI

[dank]

Oliver Xymoron <oxymoron@waste.org> wrote:
> 
> On Thu, 25 Jul 2002, Erik Andersen wrote:
> 
> > On Thu Jul 25, 2002 at 09:31:23AM -0700, H. Peter Anvin wrote:
> > > Oliver Xymoron wrote:
> > > >
> > > >Ideally, the ABI layer would be maintained and packaged separately from
> > > >both the kernel and glibc to avoid gratuitous changes from either side.
> > >
> > > I disagree.  The ABI is a product of the kernel and should be attached
> > > to it.  It is *not* a product of glibc -- glibc is a consumer of it, as
> > > are any other libcs.
> >
> > Agreed.  I maintain a libc and I certainly do not want to
> > have to maintain the kernel ABI of the day headers.  That
> > is clearly a job for the kernel.
> 
> The idea of maintaining them separately is that people won't be able to
> touch the ABI without explicitly going through a gatekeeper whose job is
> to minimize breakage. Linus usually catches ABI changes but not always.
> 
> I explicitly did _not_ suggest making it the job of libc maintainers. And
> the whole point of the exercise is to avoid ABI of the day anyway. The ABI
> should change less frequently than the kernel or libc. It's more analogous
> to something like modutils.

IMHO it should live with the kernel, at least for now.  
The ABI .h files can live in a walled-off area that stays as stable as possible.
Anyone building glibc should be able to grab the ABI .h files from a
recent linux kernel source tarball without much effort.  (Maybe we'd
add a 'install headers_install' make target to install the ABI .h files
to make it obvious how to get them.)

Imagine what would happen if the base ABI .h files were maintained 
as part of a future Linux Standard Base, with the kernel only maintaining
.h files for extensions to the base ABI.  You'd need to install the ABI .h 
files before you could build the kernel!  That might be the right way to
go, but let's not propose it until the ABI .h files exist and are useful.

- Dan

Re: Header files and the kernel ABI

[jw schultz]

On Thu, Jul 25, 2002 at 02:13:45PM -0700, dank@kegel.com wrote:
> Oliver Xymoron <oxymoron@waste.org> wrote:
> > 
> > On Thu, 25 Jul 2002, Erik Andersen wrote:
> > 
> > > On Thu Jul 25, 2002 at 09:31:23AM -0700, H. Peter Anvin wrote:
> > > > Oliver Xymoron wrote:
> > > > >
> > > > >Ideally, the ABI layer would be maintained and packaged separately from
> > > > >both the kernel and glibc to avoid gratuitous changes from either side.
> > > >
> > > > I disagree.  The ABI is a product of the kernel and should be attached
> > > > to it.  It is *not* a product of glibc -- glibc is a consumer of it, as
> > > > are any other libcs.
> > >
> > > Agreed.  I maintain a libc and I certainly do not want to
> > > have to maintain the kernel ABI of the day headers.  That
> > > is clearly a job for the kernel.
> > 
> > The idea of maintaining them separately is that people won't be able to
> > touch the ABI without explicitly going through a gatekeeper whose job is
> > to minimize breakage. Linus usually catches ABI changes but not always.
> > 
> > I explicitly did _not_ suggest making it the job of libc maintainers. And
> > the whole point of the exercise is to avoid ABI of the day anyway. The ABI
> > should change less frequently than the kernel or libc. It's more analogous
> > to something like modutils.
> 
> IMHO it should live with the kernel, at least for now.  
> The ABI .h files can live in a walled-off area that stays as stable as possible.
> Anyone building glibc should be able to grab the ABI .h files from a
> recent linux kernel source tarball without much effort.  (Maybe we'd
> add a 'install headers_install' make target to install the ABI .h files
> to make it obvious how to get them.)
> 
> Imagine what would happen if the base ABI .h files were maintained 
> as part of a future Linux Standard Base, with the kernel only maintaining
> .h files for extensions to the base ABI.  You'd need to install the ABI .h 
> files before you could build the kernel!  That might be the right way to
> go, but let's not propose it until the ABI .h files exist and are useful.

I'd say it should be maintained with the kernel.  The lib
building process would include specifying the ABI headers
which would be copied to /usr/include as part of the libc
install.  I'd rather keep this in the hands of the kernel
maintainers than have a new standards committee slowing
progress.

Building the kernel would reference the ABI headers in
/usr/include so the kernel built would match the libs.
Backward compatibility could be supported with by build-time
checks.  In essence this would make the kernel ABI against
which the libc was built act as a set of implied kernel
build options.  The kernel build process could eventually
detect incompatabilities for ABI changes via version
assertions.  (what do you think Kieth?)

A CML option CONFIG_ABI would default to /usr/include/...
to allow building kernels for other systems or in
anticipation of a libc rebuild. (a shot in the foot)

This way kernel builds could leave out deprecated
interfaces as soon as the local libc stopped relying on
them.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt