Kernel header policy

Kernel header policy

[Marc Aurele La France]

It has been more than a week now...

+----------------------------------+-----------------------------------+
|  Marc Aurele La France           |  work:   1-780-492-9310           |
|  Academic Information and        |  fax:    1-780-492-1729           |
|    Communications Technologies   |  email:  tsi@ualberta.ca          |
|  352 General Services Building   +-----------------------------------+
|  University of Alberta           |                                   |
|  Edmonton, Alberta               |     Standard disclaimers apply    |
|  T6G 2H1                         |                                   |
|  CANADA                          |                                   |
+----------------------------------+-----------------------------------+
XFree86 developer and VP.  ATI driver and X server internals.

---------- Forwarded message ----------
Date: Sun, 3 Jul 2005 11:12:03 -0600 (MDT)
From: Marc Aurele La France <tsi@ualberta.ca>
To: Linus Torvalds
Subject: Kernel header policy

Hi, Linus.  It has been a while since we last talked.  I hope all is well
with you and your family.

I am contacting you to express my concern over a growing trend in kernel
development.  I am specifically referring to changes being made to kernel
headers that break compatibility at the userland level, where __KERNEL__
isn't #define'd.

The most common errors involve the introduction of references, in 
non-__KERNEL__ code, to symbols that ...

1) are only #define'd or typedef'ed in __KERNEL__-bracketed code;  and/or
2) are non-ANSIC C99'isms.

Recent examples include, but are not limited to, ...

a) <linux/fb.h> was temporarily broken during 2.6.9's development cycle.
    The problem here (fixed before 2.6.9's release) was an #include of
    <linux/list.h> not bracketed by __KERNEL__. i.e. "1)" above.

b) <linux/pci.h> has been broken since 2.5.62's development cycle and has
    yet to be fixed.  Here, the #include of <linux/mod_devicetable.h> needs
    to be bracked by __KERNEL__.  This is another occurrence of "1)".

c) <linux/joystick.h> is broken in both ways by 2.6.13-rc1.  This change
    introduces references to <asm/types.h> symbols BITS_PER_LONG (only
    #define'd under __KERNEL__), and __s64 (on 32-bit platforms, only
    typedef'ed under gcc without -ansi).

It is understandable that such errors do not usually show up during kernel
development.  Yet, an unfortunate consequence of the policy against
/usr/include/{linux,asm,...} symlinks is to delay detection of such errors
until someone attempts to rebuild an X server, or some other software,
with a glibc re-generated using the modified kernel headers.

It is certainly possible to report such problems individually and have
done so for most of them.  But, in practice, it has sometimes been
difficult to determine who to contact, let alone convince the relevant
party that there is a problem.  Furthermore, repetition over time turns
the reporting of such errors into an exercise in evangelisation, a
function I feel would be more effectively provided internally within the
kernel development community.

To that end, I would propose, as a possible technical solution, extending
the kernel build process to detect these errors during kernel development.

Your thoughts on this matter would be greatly appreciated.

Thanks.

Marc.

+----------------------------------+-----------------------------------+
|  Marc Aurele La France           |  work:   1-780-492-9310           |
|  Academic Information and        |  fax:    1-780-492-1729           |
|    Communications Technologies   |  email:  tsi@ualberta.ca          |
|  352 General Services Building   +-----------------------------------+
|  University of Alberta           |                                   |
|  Edmonton, Alberta               |     Standard disclaimers apply    |
|  T6G 2H1                         |                                   |
|  CANADA                          |                                   |
+----------------------------------+-----------------------------------+
XFree86 developer and VP.  ATI driver and X server internals.

Re: Kernel header policy

[Stephen Frost]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

* Marc Aurele La France (tsi@ualberta.ca) wrote:
> To that end, I would propose, as a possible technical solution, extending
> the kernel build process to detect these errors during kernel development.

Well, couple stupid comments:

#1: I'm not *entirely* sure Linus reads every mail to lkml..  Dunno if
    you also sent it to him directly, but if not then that might explain
    the lack of response...

#2: Have you got a patch that does this?  If not, you might want to
    write one up and just submit it..  If you're expecting someone else
    to write it, well, that might also be why you've not had much
    response... :)

    	Thanks,

		Stephen

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Kernel header policy

[Tomasz Torcz]

On Mon, Jul 11, 2005 at 03:37:47PM -0600, Marc Aurele La France wrote:
> I am contacting you to express my concern over a growing trend in kernel
> development.  I am specifically referring to changes being made to kernel
> headers that break compatibility at the userland level, where __KERNEL__
> isn't #define'd.

 You should CC person which maintains userspace kernel headers. I've
added Mariusz to CC list.

-- 
Tomasz Torcz                        To co nierealne - tutaj jest normalne.
zdzichu@irc.-nie.spam-.pl          Ziomale na życie mają tu patenty specjalne.

Re: Kernel header policy

[Horst von Brand]

Marc Aurele La France <tsi@ualberta.ca> wrote:
> It has been more than a week now...

> ---------- Forwarded message ----------
> Date: Sun, 3 Jul 2005 11:12:03 -0600 (MDT)
> From: Marc Aurele La France <tsi@ualberta.ca>
> To: Linus Torvalds
> Subject: Kernel header policy

[....]

> I am contacting you to express my concern over a growing trend in kernel
> development.  I am specifically referring to changes being made to kernel
> headers that break compatibility at the userland level, where __KERNEL__
> isn't #define'd.

The policy with respect to kernel headers is /very/ simple:

  T H E Y   A R E   N E V E R   U S E D   F R O M   U S E R L A N D.

This general policy makes all your points (trivially) moot.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: Kernel header policy

[Peter Staubach]

Horst von Brand wrote:

>>I am contacting you to express my concern over a growing trend in kernel
>>development.  I am specifically referring to changes being made to kernel
>>headers that break compatibility at the userland level, where __KERNEL__
>>isn't #define'd.
>>    
>>
>
>The policy with respect to kernel headers is /very/ simple:
>
>  T H E Y   A R E   N E V E R   U S E D   F R O M   U S E R L A N D.
>
>This general policy makes all your points (trivially) moot.
>

I must admit a little confusion here.  Clearly, kernel header files are
used at the user level.  The kernel and user level applications must share
definitions for a great many things.

Perhaps more precisely, the rule is that kernel header files should not be
#include'd directly from user level applications, but may be #include'd
indirectly through other header files as appropriate?

    Thanx...

       ps

Re: Kernel header policy

[Arjan van de Ven]

On Tue, 2005-07-12 at 09:26 -0400, Peter Staubach wrote:
> Horst von Brand wrote:
> 
> >>I am contacting you to express my concern over a growing trend in kernel
> >>development.  I am specifically referring to changes being made to kernel
> >>headers that break compatibility at the userland level, where __KERNEL__
> >>isn't #define'd.
> >>    
> >>
> >
> >The policy with respect to kernel headers is /very/ simple:
> >
> >  T H E Y   A R E   N E V E R   U S E D   F R O M   U S E R L A N D.
> >
> >This general policy makes all your points (trivially) moot.
> >
> 
> I must admit a little confusion here.  Clearly, kernel header files are
> used at the user level.  The kernel and user level applications must share
> definitions for a great many things.

you are incorrect or rather imprecise here. Userspace needs headers
which define the kernel<->Userspace ABI. That is not the same as "the"
kernel headers.

> 
> Perhaps more precisely, the rule is that kernel header files should not be
> #include'd directly from user level applications, but may be #include'd
> indirectly through other header files as appropriate?

actually the rule in linux is that you should use cleaned up ABI
defining headers. There's several sets to chose from even. Generally
those sets have their origins in the kernel but are stripped down to
just the userspace-abi elements. 
(eg no kernel specific things like spinlocks or inlines or ..)

Re: Kernel header policy

[Alistair John Strachan]

On Tuesday 12 Jul 2005 14:36, Arjan van de Ven wrote:
> On Tue, 2005-07-12 at 09:26 -0400, Peter Staubach wrote:
> > Horst von Brand wrote:
> > >>I am contacting you to express my concern over a growing trend in
> > >> kernel development.  I am specifically referring to changes being made
> > >> to kernel headers that break compatibility at the userland level,
> > >> where __KERNEL__ isn't #define'd.
> > >
> > >The policy with respect to kernel headers is /very/ simple:
> > >
> > >  T H E Y   A R E   N E V E R   U S E D   F R O M   U S E R L A N D.
> > >
> > >This general policy makes all your points (trivially) moot.
> >
> > I must admit a little confusion here.  Clearly, kernel header files are
> > used at the user level.  The kernel and user level applications must
> > share definitions for a great many things.
>
> you are incorrect or rather imprecise here. Userspace needs headers
> which define the kernel<->Userspace ABI. That is not the same as "the"
> kernel headers.
>
> > Perhaps more precisely, the rule is that kernel header files should not
> > be #include'd directly from user level applications, but may be
> > #include'd indirectly through other header files as appropriate?
>
> actually the rule in linux is that you should use cleaned up ABI
> defining headers. There's several sets to chose from even. Generally
> those sets have their origins in the kernel but are stripped down to
> just the userspace-abi elements.
> (eg no kernel specific things like spinlocks or inlines or ..)

One example of such "cleaned up headers" that are distro non-specific are the 
linux-libc-headers available from:

http://ep09.pld-linux.org/~mmazur/linux-libc-headers/

These are replacements for /usr/include/linux and /usr/include/asm (for your 
arch) and favour userspace, kept in sync with kernelspace manually. However, 
your glibc should (ideally) be compiled against these so that you don't get 
weird, incompatible data structures. I've never personally seen this happen, 
however.

If the "bugs" you were complaining about originally are still present in these 
libc-headers, you may have a legitimate issue and directing your questions to 
Mariusz Mazur is probably okay.

-- 
Cheers,
Alistair.

personal:   alistair()devzero!co!uk
university: s0348365()sms!ed!ac!uk
student:    CS/CSim Undergraduate
contact:    1F2 55 South Clerk Street,
            Edinburgh. EH8 9PP.

Re: Kernel header policy

[Jim Nance]

On Tue, Jul 12, 2005 at 09:26:53AM -0400, Peter Staubach wrote:

> I must admit a little confusion here.  Clearly, kernel header files are
> used at the user level.  The kernel and user level applications must share
> definitions for a great many things.

Perhaps a little history would help.  In the beginning, the kernel was
written with the intention that userland would be including the headers.
And libc did include the kernel headers.

This did provide an effective way to get new kernel features to show
up in userland, but it created all sorts of other problems.  Eventually
it was decided/decreed that userland would NOT include kernel headers.
Instead, libc would provide a set of headers which would either be
compatable, or would marshel data into the form the kernel wanted.

I don't remember exactly when this was done, but I believe it was
some time in the late 90s.  It's been this way a while now.

Thanks,

Jim

-- 
jlnance@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

Re: Kernel header policy

[Eric Piel]

12.07.2005 20:38, Jim Nance wrote/a écrit:
> 
> 
> Perhaps a little history would help.  In the beginning, the kernel was
> written with the intention that userland would be including the headers.
> And libc did include the kernel headers.
> 
> This did provide an effective way to get new kernel features to show
> up in userland, but it created all sorts of other problems.  Eventually
> it was decided/decreed that userland would NOT include kernel headers.
> Instead, libc would provide a set of headers which would either be
> compatable, or would marshel data into the form the kernel wanted.
>  

So does this mean that all the "#ifdef __KERNEL__" are useless or are 
they still used?

Eric

Re: Kernel header policy

[Tom Rini]

On Tue, Jul 12, 2005 at 09:08:53PM +0200, Eric Piel wrote:
> 12.07.2005 20:38, Jim Nance wrote/a écrit:
> >
> >
> >Perhaps a little history would help.  In the beginning, the kernel was
> >written with the intention that userland would be including the headers.
> >And libc did include the kernel headers.
> >
> >This did provide an effective way to get new kernel features to show
> >up in userland, but it created all sorts of other problems.  Eventually
> >it was decided/decreed that userland would NOT include kernel headers.
> >Instead, libc would provide a set of headers which would either be
> >compatable, or would marshel data into the form the kernel wanted.
> > 
> 
> So does this mean that all the "#ifdef __KERNEL__" are useless or are 
> they still used?

Because a large number of things aren't "fixed", __KERNEL__ is still
used so that nothing more breaks.

-- 
Tom Rini
http://gate.crashing.org/~trini/

Re: Kernel header policy

[Greg KH]

On Mon, Jul 11, 2005 at 03:37:47PM -0600, Marc Aurele La France wrote:
> I am contacting you to express my concern over a growing trend in kernel
> development.  I am specifically referring to changes being made to kernel
> headers that break compatibility at the userland level, where __KERNEL__
> isn't #define'd.

Kernel headers are not to be included by userspace programs.  See the
archives for details...

> b) <linux/pci.h> has been broken since 2.5.62's development cycle and has
>    yet to be fixed.  Here, the #include of <linux/mod_devicetable.h> needs
>    to be bracked by __KERNEL__.  This is another occurrence of "1)".

-ENOPATCH.

thanks,

greg k-h