[RFC] Splitting out kernel<=>userspace ABI headers

[RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

A while ago there was a big discussion about splitting out the
userspace-accessible portions of the kernel headers into a separate
directory, "kabi", "kernel-abi", "linux-abi", or a half-dozen other
suggestions.  Linus sprinkled a bit of holy-penguin-pee on the idea,
but nothing ever really happened after that.  I have some available
time at the moment, and I would be willing to undertake the task,
but I would like a bit of guidance first, both from Linus/akpm/etc,
and from the list in general, about a few initial issues I see from
my initial attempts to sort through the mess:

   1)  There are a couple header files upon which almost everything
else depends, among them {asm,linux}/{posix_,}types.h, which have
some significant duplications.  Many of the archs have weird sizes
for those types to preserve some backwards-compatibility ABI, but
nowhere does it explain if there are any type-size restrictions in
general.  I would propose that those headers be reorganized so that
there are sane defaults for all the types in kabi/types.h, and
archs that require different would #define exceptions in their
kabi/arch-foo/types.h.  This would allow new archs to start with a
sane standard ABI before it becomes set in stone.

   2)  There is a bunch of stuff that would be _really_ useful in
userspace programs as well, even though not kernel ABI, such as
list.h, atomic.h (with a few archs modified due to privilege
restrictions), etc.  If there is interest, I would attempt to split
off those headers into a kcore/kerncore/linuxcore/whatever inline
header collection included in the linux distribution and installed
as part of the kernel headers.

   3)  What names are preferable for the above?  My personal
preferences are "kabi" and "kcore", because those save the most
typing for the sucker trying to do all this (IE: me), although if
someone has good reasons otherwise, I'll listen.

I realize this project is only slightly short of massive, however I
do have a bunch of time and am willing to do the grunt work if
enlightened as to the community desires.  I have a few different
semi-patches almost ready, and I can probably finish up a couple
this weekend if I can figure out which way people want to go.  One
of the major challenges is that kernel files have historically kind
of indiscriminately included asm/foo.h when they really meant
linux/foo.h (See the types.h example), only to have it magically
work because some other header already included linux/types.h
anyways.  If arch/driver/etc maintainers are willing to take patches
to clean that up, I'll start with that and eventually get a decent
set of kabi/* headers.


Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things,  
because that
would also stop them from doing clever things.
   -- Doug Gwyn

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Erik Andersen]

On Thu Sep 01, 2005 at 11:00:16PM -0400, Kyle Moffett wrote:
> A while ago there was a big discussion about splitting out the
> userspace-accessible portions of the kernel headers into a separate
> directory, "kabi", "kernel-abi", "linux-abi", or a half-dozen other
> suggestions.  Linus sprinkled a bit of holy-penguin-pee on the idea,
> but nothing ever really happened after that.

Have you seen the linux-libc-headers:
    http://ep09.pld-linux.org/~mmazur/linux-libc-headers/
which, while not an official part of the kernel, do a pretty
good job...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 2, 2005, at 09:41:09, Erik Andersen wrote:
> On Thu Sep 01, 2005 at 11:00:16PM -0400, Kyle Moffett wrote:
>> A while ago there was a big discussion about splitting out the
>> userspace-accessible portions of the kernel headers into a separate
>> directory, "kabi", "kernel-abi", "linux-abi", or a half-dozen other
>> suggestions.  Linus sprinkled a bit of holy-penguin-pee on the idea,
>> but nothing ever really happened after that.
>
> Have you seen the linux-libc-headers:
>     http://ep09.pld-linux.org/~mmazur/linux-libc-headers/
> which, while not an official part of the kernel, do a pretty
> good job...

Well, the eventual goal of this project would be to eliminate the
need for linux-libc-headers by making that task trivial (IE: Just copy
the kcore/ and kabi/ (or whatever they get called) directories into
/usr/include.  There would probably be some compatibility headers
installed into /usr/include/linux until 2.8 is released or 2.7 is
forked for some major internal modification, but other than that, the
stuff shared by userspace and kernelspace would be only in kcore and
kabi, and eventually the linux/* stuff could remove all the __KERNEL__
ifdefs contained therein.  Right now linux-libc-headers is maintained
by one person at each kernel revision.  It would be much better if
that maintenance load could be undertaken instead by those who create
the code that uses those headers, the kernel developers themselves,
because they surely understand it better and are likely to be able to
do it more easily and accurately.

Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things,  
because that
would also stop them from doing clever things.
   -- Doug Gwyn

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Erik Andersen]

On Fri Sep 02, 2005 at 04:51:49PM -0400, Kyle Moffett wrote:
> On Sep 2, 2005, at 09:41:09, Erik Andersen wrote:
> >Have you seen the linux-libc-headers:
> >    http://ep09.pld-linux.org/~mmazur/linux-libc-headers/
> >which, while not an official part of the kernel, do a pretty
> >good job...
> 
> Well, the eventual goal of this project would be to eliminate the
> need for linux-libc-headers by making that task trivial (IE: Just copy
> the kcore/ and kabi/ (or whatever they get called) directories into
> /usr/include.

<uClibc maintainer hat on>
That would be wonderful.
</off>

It would be especially nice if everything targeting user space
were to use only all the nice standard ISO C99 types as defined
in include/stdint.h such as uint32_t and friends...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Followup to:  <20050902235833.GA28238@codepoet.org>
By author:    Erik Andersen <andersen@codepoet.org>
In newsgroup: linux.dev.kernel
> 
> <uClibc maintainer hat on>
> That would be wonderful.
> </off>
> 
> It would be especially nice if everything targeting user space
> were to use only all the nice standard ISO C99 types as defined
> in include/stdint.h such as uint32_t and friends...
> 

Absolutely not.  This would be a POSIX namespace violation; they
*must* use double-underscore types.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 2, 2005, at 20:07:58, H. Peter Anvin wrote:
> Followup to:  <20050902235833.GA28238@codepoet.org>
> By author:    Erik Andersen <andersen@codepoet.org>
> In newsgroup: linux.dev.kernel
>> <uClibc maintainer hat on>
>> That would be wonderful.
>> </off>
>>
>> It would be especially nice if everything targeting user space
>> were to use only all the nice standard ISO C99 types as defined
>> in include/stdint.h such as uint32_t and friends...
>
> Absolutely not.  This would be a POSIX namespace violation; they
> *must* use double-underscore types.

I would actually be more inclined to provide and use types like
_kabi_{s,u}{8,16,32,64}, etc.  Then the glibc/klibc/etc authors would
have the option of just doing "typedef _kabi_u32 uint32_t;" in their
header files.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+ 
++) E
W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5  
X R?
tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  !y?(-)
------END GEEK CODE BLOCK------

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Kyle Moffett wrote:
> On Sep 2, 2005, at 20:07:58, H. Peter Anvin wrote:
> 
>> Followup to:  <20050902235833.GA28238@codepoet.org>
>> By author:    Erik Andersen <andersen@codepoet.org>
>> In newsgroup: linux.dev.kernel
>>
>>> <uClibc maintainer hat on>
>>> That would be wonderful.
>>> </off>
>>>
>>> It would be especially nice if everything targeting user space
>>> were to use only all the nice standard ISO C99 types as defined
>>> in include/stdint.h such as uint32_t and friends...
>>
>>
>> Absolutely not.  This would be a POSIX namespace violation; they
>> *must* use double-underscore types.
> 
> 
> I would actually be more inclined to provide and use types like
> _kabi_{s,u}{8,16,32,64}, etc.  Then the glibc/klibc/etc authors would
> have the option of just doing "typedef _kabi_u32 uint32_t;" in their
> header files.
> 

They have to be *double-underscore*.

We have that.  They're called __[su]{8,16,32,64}.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 2, 2005, at 20:34:11, H. Peter Anvin wrote:
> Kyle Moffett wrote:
>> I would actually be more inclined to provide and use types like
>> _kabi_{s,u}{8,16,32,64}, etc.  Then the glibc/klibc/etc authors would
>> have the option of just doing "typedef _kabi_u32 uint32_t;" in their
>> header files.
>
> They have to be *double-underscore*.
>
> We have that.  They're called __[su]{8,16,32,64}.

I realize this completely.  The point of moving to kabi/* and kcore/*
would be to remove the dependence of userspace-accessible headers on
kernel-internal stuff.  As I see it, part of that means exporting a
reasonably clean and straightforward API from kabi/kcore, including a
decent namespace prefix.  The goal would be something that the kernel
headers could map to types useable in kernel code, that various *libc
in userspace could map to POSIX types, and that would have a nice
prefix to be namespace clean and avoid the risk of contamination.
Given this set of goals, I think that something like the below would
probably work and satisfy the needs of both *libc and the kernel:



/* kcore/types.h */
typedef unsigned char __kabi_u8;
typedef   signed char __kabi_s8;
typedef [...]



/* linux/types.h */
#include <kcore/types.h>

#ifndef __KERNEL__
# warning "Insert some kind of deprecation warning here
#endif

   /* These for compatibility only.  When the last ABI headers move
      to kcore or kabi, these should go in __KERNEL__ */
typedef __kabi_u8 __u8;
typedef __kabi_s8 __s8;
[...]

#ifdef __KERNEL__
typedef __kabi_u8 u8;
typedef __kabi_s8 s8;
#endif



/* stdint.h */
#include <kcore/types.h>
typedef __kabi_u8 uint8_t;
typedef __kabi_s8 int8_t;
[...]



Cheers,
Kyle Moffett

--
There is no way to make Linux robust with unreliable memory subsystems,
sorry.  It would be like trying to make a human more robust with an
unreliable O2 supply. Memory just has to work.
   -- Andi Kleen

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Erik Andersen]

On Sat Sep 03, 2005 at 12:07:58AM +0000, H. Peter Anvin wrote:
> Followup to:  <20050902235833.GA28238@codepoet.org>
> By author:    Erik Andersen <andersen@codepoet.org>
> In newsgroup: linux.dev.kernel
> > 
> > <uClibc maintainer hat on>
> > That would be wonderful.
> > </off>
> > 
> > It would be especially nice if everything targeting user space
> > were to use only all the nice standard ISO C99 types as defined
> > in include/stdint.h such as uint32_t and friends...
> > 
> 
> Absolutely not.  This would be a POSIX namespace violation; they
> *must* use double-underscore types.

I assume you are worried about the stuff under asm that ends up
being included by nearly every header file in the world.  Of
course asm must use double-underscore types.  But the thing is,
the vast majority of the kernel headers live under
linux/include/linux/ and do not use double-underscore types, they
use kernel specific, non-underscored types such as s8, u32, etc.
My copy of IEEE 1003.1 and my copy of ISO/IEC 9899:1999 both fail
to prohibit using the shiny new ISO C99 type for the various
#include <linux/*> header files, which is what I was suggesting.

The world would be so much nicer a place if user space were free
to #include linux/* header files rather than keeping a
per-project private copy of all kernel structs of interest.  And
where these kernel headers would #include stdint.h and define
their stucts in terms of ISO C99 types.  I see nothing at all in
the standards preventing such a change,

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Erik Andersen wrote:
> 
> I assume you are worried about the stuff under asm that ends up
> being included by nearly every header file in the world.  Of
> course asm must use double-underscore types.  But the thing is,
> the vast majority of the kernel headers live under
> linux/include/linux/ and do not use double-underscore types, they
> use kernel specific, non-underscored types such as s8, u32, etc.
> My copy of IEEE 1003.1 and my copy of ISO/IEC 9899:1999 both fail
> to prohibit using the shiny new ISO C99 type for the various
> #include <linux/*> header files, which is what I was suggesting.
> 
> The world would be so much nicer a place if user space were free
> to #include linux/* header files rather than keeping a
> per-project private copy of all kernel structs of interest.  And
> where these kernel headers would #include stdint.h and define
> their stucts in terms of ISO C99 types.  I see nothing at all in
> the standards preventing such a change,
> 

Exportable types need to be double-underscore types, because the header 
files in user space that would include them can generally not include 
<stdint.h>.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Erik Andersen]

On Fri Sep 02, 2005 at 10:22:20PM -0700, H. Peter Anvin wrote:
> Exportable types need to be double-underscore types, because the header 
> files in user space that would include them can generally not include 
> <stdint.h>.

I'm not talking about kernel headers that have to worry about
eventually being included in user space headers.  Those nearly
all live in include/asm.  I'm talking about the kernel headers
that define how userspace is supposed to interface with
particular kernel drivers or hardware.  Headers such as
linux/cdrom.h and linux/loop.h and linux/fb.h.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Erik Andersen wrote:
> On Fri Sep 02, 2005 at 10:22:20PM -0700, H. Peter Anvin wrote:
> 
>>Exportable types need to be double-underscore types, because the header 
>>files in user space that would include them can generally not include 
>><stdint.h>.
> 
> 
> I'm not talking about kernel headers that have to worry about
> eventually being included in user space headers.  Those nearly
> all live in include/asm.  I'm talking about the kernel headers
> that define how userspace is supposed to interface with
> particular kernel drivers or hardware.  Headers such as
> linux/cdrom.h and linux/loop.h and linux/fb.h.
> 

What are you going to do with them if you're not going to include them 
in userspace?!!!

If you're proposing one policy for linux/loop.h and one for sys/stat.h, 
I would have to classify that as insane.  Everything that gets exported 
to userspace should behave the same way, please.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Erik Andersen]

On Fri Sep 02, 2005 at 10:53:19PM -0700, H. Peter Anvin wrote:
> Erik Andersen wrote:
> >On Fri Sep 02, 2005 at 10:22:20PM -0700, H. Peter Anvin wrote:
> >
> >>Exportable types need to be double-underscore types, because the header 
> >>files in user space that would include them can generally not include 
> >><stdint.h>.
> >
> >
> >I'm not talking about kernel headers that have to worry about
> >eventually being included in user space headers.  Those nearly
> >all live in include/asm.  I'm talking about the kernel headers
> >that define how userspace is supposed to interface with
> >particular kernel drivers or hardware.  Headers such as
> >linux/cdrom.h and linux/loop.h and linux/fb.h.
> >
> 
> What are you going to do with them if you're not going to include them 
> in userspace?!!!
> 
> If you're proposing one policy for linux/loop.h and one for sys/stat.h, 
> I would have to classify that as insane.  Everything that gets exported 
> to userspace should behave the same way, please.

That is certainly not what I was proposing.  Why are you bringing
sys/stat.h into this?  The contents of sys/stat.h are entirely up
to SUSv3 and the C library to worry about.  Nobody has proposed
mucking with that.  I dunno about your C library, but mine
doesn't include linux/* header files (not even sys/stat.h).  And
I'd really like to fix uClibc to not use any asm/* either, since
much of it is entirely unsuitable for user space.

I am proposing a single consistant policy for all of linux/* such
that all linux/* headers that need integer types of a specific
size shall #include unistd.h and use ISO C99 types rather that
the ad-hoc kernel types they now use.

The policy has _long_ been that user space must never include
linux/* header files.  Since we are now proposing a project to
reverse this policy, the long standing policy making linux/*
verboten now leaves us completely free to do pretty much anything
with linux/*.  And what I want is for linux/* to use the shiny
ISO C99 types.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Erik Andersen wrote:
> 
> That is certainly not what I was proposing.  Why are you bringing
> sys/stat.h into this?  The contents of sys/stat.h are entirely up
> to SUSv3 and the C library to worry about.  Nobody has proposed
> mucking with that.  I dunno about your C library, but mine
> doesn't include linux/* header files (not even sys/stat.h).  And
> I'd really like to fix uClibc to not use any asm/* either, since
> much of it is entirely unsuitable for user space.
> 

I'm in particular commenting on the stat structure involved with the 
kernel interface.

> I am proposing a single consistant policy for all of linux/* such
> that all linux/* headers that need integer types of a specific
> size shall #include unistd.h and use ISO C99 types rather that
> the ad-hoc kernel types they now use.
> 
> The policy has _long_ been that user space must never include
> linux/* header files.  Since we are now proposing a project to
> reverse this policy, the long standing policy making linux/*
> verboten now leaves us completely free to do pretty much anything
> with linux/*.  And what I want is for linux/* to use the shiny
> ISO C99 types.

And I'm pointing out that that you're not only excluding a whole major 
portion of the kernel ABI from this mechanism if you do that, you're 
effectively requiring new mechanisms every time something is included 
into POSIX over time!  If that isn't insane, I don't know what is.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Erik Andersen wrote:
> 
> That is certainly not what I was proposing.  Why are you bringing
> sys/stat.h into this?  The contents of sys/stat.h are entirely up
> to SUSv3 and the C library to worry about.  Nobody has proposed
> mucking with that.  I dunno about your C library, but mine
> doesn't include linux/* header files (not even sys/stat.h).  And
> I'd really like to fix uClibc to not use any asm/* either, since
> much of it is entirely unsuitable for user space.
> 

That's the whole problem here, isn't it, so let's fix it the sane way 
instead of putting Descartes before Dehorse.

Anyway, to answer your implied question is: since I explicitly don't 
have to worry about forward ABI compatibility, I expose the kernel ABI 
raw.  Thus I want to be able to use the kernel ABI directly, including 
for things like struct stat.  It poses a particularly interesting 
problem, actually, because the real stat system call is called stat64 on 
most platforms.

Thus, an ABIzed <linux/abi/stat.h> or whatever it's called might export 
  "struct __kabi_stat" and "struct __kabi_stat64" with the expectation 
that the caller would "#define __kabi_stat64 stat" if that is the 
version they want.  A typedef isn't good enough for C, since you can't 
typedef struct tags.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 3, 2005, at 11:19:17, H. Peter Anvin wrote:
> Thus, an ABIzed <linux/abi/stat.h> or whatever it's called might  
> export
> "struct __kabi_stat" and "struct __kabi_stat64" with the  
> expectation that
> the caller would "#define __kabi_stat64 stat" if that is the  
> version they
> want.  A typedef isn't good enough for C, since you can't typedef  
> struct
> tags.

Didn't you mean "#define stat __kabi_stat64"?  Also, I can see that  
would
pose other issues as well say my app does "struct stat stat;"  Any error
messages would refer to a variable "__kabi_stat64" instead of the  
expected
"stat":

A userspace program:
struct stat stat;
stat.invalid = 1;

Preprocesses into:
struct __kabi_stat64 __kabi_stat64;
__kabi_stat64.invalid = 1;

And gives an error something like this for that line, confusing the
programmer:
Invalid member "invalid" for "__kabi_stat64"


As far as I can tell, this is not a solvable issue unless GCC can  
come up
with a way to either:
     typedef struct foo struct bar;
or
     struct bar { unnamed struct foo; };
the former being much nicer.  On the other hand, I think the following
should work, because the st_* names are within the C namespace and  
should
be much easier to redefine, although misuse of one of those names might
be a bit more catastrophic for the user app.

struct stat {
     struct __kabi_stat64 __stat64;
};
#define st_dev __stat64.st_dev
#define st_ino __stat64.st_ino
[...]

Then the userspace program could do this:
struct stat foo;
foo.st_ino = 0;

And it would be preprocessed into:
struct stat foo;foo.__stat64.st_ino = 0;

Cheers,
Kyle Moffett

--
Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best  
answer:

"Why do musicians compose symphonies and poets write poems? They do  
it because
life wouldn't have any meaning for them if they didn't. That's why I  
draw
cartoons. It's my life."
   -- Charles Shultz

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Followup to:  <B9E70F6F-CC0A-4053-AB34-A90836431358@mac.com>
By author:    Kyle Moffett <mrmacman_g4@mac.com>
In newsgroup: linux.dev.kernel
> 
> Didn't you mean "#define stat __kabi_stat64"?  Also, I can see that
> would pose other issues as well say my app does "struct stat stat;"
> Any error messages would refer to a variable "__kabi_stat64" instead
> of the expected "stat":
> 

No, I didn't.  That's *exactly* why I didn't mean that.

#define __kabi_stat64 stat
#include <linux/abi/stat.h>

That being said, I would personally like to see it possible to typedef
struct, union and enum tags.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 5, 2005, at 12:35:42, H. Peter Anvin wrote:
> Followup to:  <B9E70F6F-CC0A-4053-AB34-A90836431358@mac.com>
> By author:    Kyle Moffett <mrmacman_g4@mac.com>
> In newsgroup: linux.dev.kernel
>
>> Didn't you mean "#define stat __kabi_stat64"?  Also, I can see that
>> would pose other issues as well say my app does "struct stat stat;"
>> Any error messages would refer to a variable "__kabi_stat64" instead
>> of the expected "stat":
>
> No, I didn't.  That's *exactly* why I didn't mean that.
>
> #define __kabi_stat64 stat
> #include <linux/abi/stat.h>
>
> That being said, I would personally like to see it possible to typedef
> struct, union and enum tags.

_OH_!!! Forgive me for missing the point entirely!  I can see how  
that would
work very well.  Nice trick, BTW!  Very sneaky, needs significant  
explanatory
comments in whatever header file it ends up in lest others get  
confused in
the same fashion as I.  With all of that mess out of the way, I'll  
work on
getting a few initial RFC patches out the door, and then we can  
revisit this
discussion once there is something tangible to talk about.

Cheers,
Kyle Moffett

--
Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best  
answer:

"Why do musicians compose symphonies and poets write poems? They do  
it because
life wouldn't have any meaning for them if they didn't. That's why I  
draw
cartoons. It's my life."
   -- Charles Shultz

[RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Kyle Moffett]

On Sep 5, 2005, at 19:28:07, Kyle Moffett wrote:
> With all of that mess out of the way, I'll work on getting a few  
> initial RFC
> patches out the door, and then we can revisit this discussion once  
> there is
> something tangible to talk about.

Ugh.  Step one for my cleanup is to rename __ASSEMBLY__ to something  
defined
automatically by GCC (IE: __ASSEMBLER__).  And yes, I checked,  
__ASSEMBLER__
is defined by everything from old 2.95 to 4.0, even though it wasn't  
really
documented in anything older than 3.4.  This megapatch is basically a  
search
and replace of __ASSEMBLY__ with __ASSEMBLER__ over the whole kernel  
source,
except in Makefiles, where I just delete the -D__ASSEMBLY__  
argument.  If
this is generally acceptable, I'll break it up into small digestible  
pieces
and send to individual maintainers, unless someone wants to pass the  
whole
monster through their tree in one big lump.  This is a lot of code  
churn,
but it's a valid cleanup and will help me out as I try to make more  
of the
kernel headers easily digestible for userspace.

Ok, the patch itself is temporarily located here (Please be nice to my
desktop, it has a 650MB/day upload limit imposed by Virginia Tech  
that I'd
rather not go over) [patch is 308k]:

http://zeus.moffetthome.net/~kyle/rename-__ASSEMBLY__-to- 
__ASSEMBLER__.patch

And here's the diffstat [27k]

http://zeus.moffetthome.net/~kyle/rename-__ASSEMBLY__-to- 
__ASSEMBLER__.diffstat

Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things,  
because that
would also stop them from doing clever things.
   -- Doug Gwyn

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Kyle Moffett]

On Sep 5, 2005, at 21:29:27, Kyle Moffett wrote:
> On Sep 5, 2005, at 19:28:07, Kyle Moffett wrote:
>
>> With all of that mess out of the way, I'll work on getting a few  
>> initial RFC
>> patches out the door, and then we can revisit this discussion once  
>> there is
>> something tangible to talk about.
>
> If this is generally acceptable, I'll break it up into small  
> digestible
> pieces and send to individual maintainers...

Ok, here's the broken-out version.  I haven't yet found a good  
patchbomb script,
so until I do, I've got the broken-out parts on the web here:

http://zeus.moffetthome.net/~kyle/patches/convert-assembly-to-assembler/

The file 000-convert-assembly-to-assembler.diffstat in that directory  
has the
diffstat.

And once again, my 650MB upload limit applies, although so far I  
haven't even
come close, so it's not that big an issue, really...

PS:  If you have a nice patchbomb script with support for CC-ing  
individual
maintainers, numerical patch numbers, etc that you're willing to send  
me, I
would appreciate it a lot!

Cheers,
Kyle Moffett

--
I lost interest in "blade servers" when I found they didn't throw  
knives at
people who weren't supposed to be in your machine room.
   -- Anthony de Boer

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Andrew Morton]

This patch seems to have a rather low value-to-noise ratio.  Why
on earth do we want to do this?

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Kyle Moffett]

On Sep 10, 2005, at 04:45:43, Andrew Morton wrote:
> This patch seems to have a rather low value-to-noise ratio.  Why
> on earth do we want to do this?

When I started trying to split out the userspace<=>kernelspace ABI  
headers, I
found a number of things (such as __ASSEMBLY__) that would not operate
properly in userspace.  I did a bit of research and noticed that GCC  
had a
macro __ASSEMBLER__ defined everywhere we wanted __ASSEMBLY__ to be  
defined,
except probably more reliably (IE: We don't need to manually pass  
flags to
gas).  I figured that if I was going to change the linux-core headers, I
might as well change the rest.  If you don't think this is  
appropriate, I
would be interested to hear your opinion, although it might have  
saved me a
bunch of work if you had brought up your issues before I split the thing
into chunks. :-D

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so
simple that there are obviously no deficiencies. And the other way is  
to make
it so complicated that there are no obvious deficiencies.  The first  
method is
far more difficult.
   -- C.A.R. Hoare

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Andrew Morton]

Kyle Moffett <mrmacman_g4@mac.com> wrote:
>
> On Sep 10, 2005, at 04:45:43, Andrew Morton wrote:
> > This patch seems to have a rather low value-to-noise ratio.  Why
> > on earth do we want to do this?
> 
> When I started trying to split out the userspace<=>kernelspace ABI  
> headers, I
> found a number of things (such as __ASSEMBLY__) that would not operate
> properly in userspace.

Oh, OK.

What's the status of this userspace ABI project?  Is it a thing we want to
do?  How far along is it?  Have we worked out how it is to be done?  I
haven't been paying much attention to the discussion (self-defense reflex
;))

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Kyle Moffett]

On Sep 10, 2005, at 18:04:46, Andrew Morton wrote:
> Kyle Moffett <mrmacman_g4@mac.com> wrote:
>> When I started trying to split out the userspace<=>kernelspace ABI
>> headers, I found a number of things (such as __ASSEMBLY__) that
>> would not operate properly in userspace.
>
> Oh, OK.

[questions reordered to fit the answers better :-D]

> What's the status of this userspace ABI project?  How far along is it?

It's just getting started.  This is patch #1 :-D  Unfortunately, this
project will basically touch most or all of the header files, because
anything that userspace wants to use (IOCTLs, structs, etc) needs to
be put into a separate directory tree.  We've kind of tentatively
assigned "kabi" and "kcore" to that, but it's still really preliminary.

> Is it a thing we want to do?

The UML people really want this, because it would mean that UML could
completely ignore libc for most of its code and just use the ABI and
kcore headers.  Since these headers would have a specially defined
private namespace, they would be trivially useable from various libc
implementations as well.  (IE: everything begins with __kabi_ or
__kcore_, except for macros which begin with __KABI_ or __KCORE_).
The target for this is to also provide (in <kcore/*.h>) some optimized
inline routines for user programs to use, some LGPL and others GPL.
One example would be list.h, which I've copied and hacked on several
times for various other GPL projects I've done work on.

> Have we worked out how it is to be done?

Here's what we've got so far:

1)  At some point the arch/driver/etc maintainers (for anything that
interacts with userspace), need to start converting things on their
own (such as moving ioctl and struct declarations to a <kabi/*.h>
header file), because the people working on it certainly don't have
all the varieties of hardware and userspace programs that would be
affected by this change.

2)  The goal is to minimize changes to kernel code.  I'm not out to
rename "struct list_head", that would be silly!  Instead, the header
<linux/list.h>  would be basically reduced to this:

#ifndef  __LINUX_LIST_H
# define __LINUX_LIST_H 1
# ifdef __KERNEL__

#  define __kcore_list_item list_head
#  include <kcore/list.h>
#  define list_add(x,y) __kcore_list_add(x,y)

[...etc...]

# endif /* __KERNEL__ */
#endif /* not __LINUX_LIST_H */

3)  Another side effect of this project will be that we will have
the chance to clean up and merge some of the stuff currently in
the asm-* directories.  For example, the posix_types.h headers on
most of the architectures have the same sizes for each type, only
a few are different.  With this we have a chance to have the few
weird architectures do:
   #define __KABI_ARCH_TYPE_*_T __kabi_[su][0-9]+_t
Then all the rest just use the default.  This would make it much
easier and less error-prone to add a new architecture, because
you would have a really small set of structs, types, definitions,
etc in <kabi/arch-*/*.h> that are _required_ across all
architectures, and most of the stuff in asm-*/*.h would be header
files for code that exists only on a _single_ architecture.

Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things,  
because that
would also stop them from doing clever things.
   -- Doug Gwyn

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Andrew Morton]

Kyle Moffett <mrmacman_g4@mac.com> wrote:
>
> > Have we worked out how it is to be done?
> 
>  Here's what we've got so far:
> 
>  1)  At some point the arch/driver/etc maintainers (for anything that
>  interacts with userspace), need to start converting things on their
>  own (such as moving ioctl and struct declarations to a <kabi/*.h>
>  header file), because the people working on it certainly don't have
>  all the varieties of hardware and userspace programs that would be
>  affected by this change.

This will be very disruptive.

>  2)  The goal is to minimize changes to kernel code.  I'm not out to
>  rename "struct list_head", that would be silly!  Instead, the header
>  <linux/list.h>  would be basically reduced to this:
> 
>  #ifndef  __LINUX_LIST_H
>  # define __LINUX_LIST_H 1
>  # ifdef __KERNEL__
> 
>  #  define __kcore_list_item list_head
>  #  include <kcore/list.h>
>  #  define list_add(x,y) __kcore_list_add(x,y)
> 
>  [...etc...]
> 
>  # endif /* __KERNEL__ */
>  #endif /* not __LINUX_LIST_H */

I hope list.h was a poorly-chosen example, and that there are no plans to
actually do anything like the above to list.h.

Surely the only files which need to be altered are those which we can
reasonably expect userspace to actually include.

>  3)  Another side effect of this project will be that we will have
>  the chance to clean up and merge some of the stuff currently in
>  the asm-* directories.

I'd suggest that you avoid side-effects.  Unrelated cleanups are unrelated
- do it as a separate project.

I'm very dubious about the whole idea, frankly.

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Kyle Moffett]

On Sep 10, 2005, at 20:48:18, Andrew Morton wrote:
>>  1)  At some point the arch/driver/etc maintainers (for anything that
>>  interacts with userspace), need to start converting things on their
>>  own (such as moving ioctl and struct declarations to a <kabi/*.h>
>>  header file), because the people working on it certainly don't have
>>  all the varieties of hardware and userspace programs that would be
>>  affected by this change.
>
> This will be very disruptive.

I know, but currently it places a lot of unnecessary load on the
linux-libc-headers maintainer, and sometimes that doesn't get done
correctly.  For most of the above, all the maintainer will need to
do is move the ioctl and struct declarations necessary in userspace
from the header file in linux/ to a corresponding one in kabi/,
_maybe_ change a few macro names, and then verify that it all still
works correctly.

>>  2)  The goal is to minimize changes to kernel code.  I'm not out to
>>  rename "struct list_head", that would be silly!  Instead, the header
>>  <linux/list.h>  would be basically reduced to this:
>>
>>  #ifndef  __LINUX_LIST_H
>>  # define __LINUX_LIST_H 1
>>  # ifdef __KERNEL__
>>
>>  #  define __kcore_list_item list_head
>>  #  include <kcore/list.h>
>>  #  define list_add(x,y) __kcore_list_add(x,y)
>>
>>  [...etc...]
>>
>>  # endif /* __KERNEL__ */
>>  #endif /* not __LINUX_LIST_H */
>
> I hope list.h was a poorly-chosen example, and that there are no  
> plans to
> actually do anything like the above to list.h.

Yeah, that was actually a rather poorly chosen example, but it
does somewhat illustrate the way we would preserve compatibility.

> Surely the only files which need to be altered are those which we can
> reasonably expect userspace to actually include.

Yeah.  I would have given one of the types.h files, except those
are so massively chaotic and with all sorts of cross dependencies
that I haven't tinkered with them enough yet.  I'm working on it,  
though!

>>  3)  Another side effect of this project will be that we will have
>>  the chance to clean up and merge some of the stuff currently in
>>  the asm-* directories.
>
> I'd suggest that you avoid side-effects.  Unrelated cleanups are  
> unrelated
> - do it as a separate project.

My plan is to start with a bunch of cleanups that make it easier
to split the files, and then proceed to begin splitting the low
level files that are essential to everything else.  Once that's
done, I would move on to the other headers, the ones that have
dependencies on lots of headers.  I'll probably have to submit
cleanups to stuff like linux/kernel.h, linux/sched.h, etc, to
prevent include cycles.

> I'm very dubious about the whole idea, frankly.

This would make life a million times easier for the UML people,
the glibc people, the klibc people, and the linux-libc-headers
maintainer (IE: you don't need one, and he can go do something
more productive), etc.  If you want proof, several of the above
groups have previously voiced in support of this project in
this very thread.

This is definitely not a short-term project, and will really only
reach significant improvements for those groups by around
2.6.30 or so, but many feel it's a necessary change.

Cheers,
Kyle Moffett

--
I lost interest in "blade servers" when I found they didn't throw  
knives at
people who weren't supposed to be in your machine room.
   -- Anthony de Boer

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Paul Jackson]

Kyle wrote:
> This would make life a million times easier for the UML people,
> the glibc people, the klibc people, and the linux-libc-headers
> maintainer

Spraying wildly from the hip with my Uzi ...

If the several groups you list would all benefit from some particular
form of kernel headers that is not what we have now, then why don't
they pool together and have one person maintain such a header set,
keeping it current with the kernel.  It could even (Lord Linus willing)
be given a place to live in the kernel source: "the kernel-user API,
as seen from userland".

But trying to cram that header view into the same files as the kernel's
internal view of itself seems like a fools errand.

Heck, on a good day, you might even get an occassional kernel patch
that included the corresponding kernel-user API header change, rather
as happens now with the Documentation files.  And efforts to keep
stuff usable with C++ code could be welcomed on such headers, where
they are rejected for kernel internals.

Bottom line - leave all existing kernel files as they are, and add
a new subdirectory, for these new header files.  Agree amongst
yourselves (the above named groups) on a MAINTAINER, and start
crafting the headers you need, in the style to which you wish to
become accustomed.

Don't confuse theory and practice.  In theory, these new headers
present information already known to the kernel, so should not
be separate.  But in practice, the demands are sufficiently different
that it will be easier to maintain as a separate set of headers.

Better two simple answers than one convoluted answer.

... hopefully my stray bullets didn't harm any innocents.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[H. Peter Anvin]

Paul Jackson wrote:
> 
> Don't confuse theory and practice.  In theory, these new headers
> present information already known to the kernel, so should not
> be separate.  But in practice, the demands are sufficiently different
> that it will be easier to maintain as a separate set of headers.
> 

The only sane thing is to have a set of ABI headers with a clean, 
specific set of rules, which is included by the kernel private headers, 
as well as userspace.

	-hpa

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Paul Jackson]

hpa wrote:
> The only sane thing is to have a set of ABI headers with a clean, 
> specific set of rules, which is included by the kernel private headers, 
> as well as userspace.

Why must the ABI headers be included by both kernel and user headers to
be sane?

Hmmm ... I'm not sure I want to ask that, actually.  I have this feeling
from the tone of your assertion that you can explain to me why such a
header organization is the only one that fits your mental model of how
these things are structured, but that communication between us may
break down when you try to convince me that your mental model for this
is the only correct one.

Oh well ... we'll see where this goes.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Sam Ravnborg]

On Mon, Sep 12, 2005 at 08:47:56AM -0700, Paul Jackson wrote:
> hpa wrote:
> > The only sane thing is to have a set of ABI headers with a clean, 
> > specific set of rules, which is included by the kernel private headers, 
> > as well as userspace.
> 
> Why must the ABI headers be included by both kernel and user headers to
> be sane?

Thats the only way to keep them consistent.
Likewise we do not have seperate copies of the core headers for
individual subsystems.

If one have to edit in two places when adding a list of constants for
example you can be sure at one point in time the second place is
forgotten.
What does this separation where user space headers live a life outside
the kernel buy us? As kernel developers we are free to think on the user
psace side - thats nice sometimes. But hiding in a hole is not the
way we create the best interfaces. That has the kernel model shwed many
times. It requires publicity and frequent usage to be good. Two thinks
that is harder to get when we hide.

But seen form the userspace perspective this is horrible. 
They have to maintain a set of almost duplicated headers with 
various licenses, sync problems etc. for absolutely no gain.

	Sam

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Paul Jackson]

Sam wrote:
> ... for absolutely no gain.

I hear the usual and reasonable arguments on one side,
favoring consolidating the headers.

The final phrase "absolutely no gain" suggests a lack
of consideration for the usual and reasonable arguments
that no doubt exist on the other side.

The devil is in the details.

Good luck.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Kyle Moffett]

On Sep 12, 2005, at 17:14:26, Paul Jackson wrote:
> Sam wrote:
>> ... for absolutely no gain.
>
> I hear the usual and reasonable arguments on one side,
> favoring consolidating the headers.
>
> The final phrase "absolutely no gain" suggests a lack
> of consideration for the usual and reasonable arguments
> that no doubt exist on the other side.

I dunno about Sam, but I've looked at this a lot, and there really  
seem to
be no advantages to keeping them separate.  There's a lot of cruft in  
the
kernel headers, and in many cases a lack of organization.  By keeping  
the
user-accessible kernel-headers separate, you save work in the short term
by putting off cleaning up that cruft, but in the long term you have  
twice
the maintenance burden, in addition to the fact that you'll need to  
clean
up the cruft eventually anyways.  The user-space-accessible versions  
will
be identical to the kernel-space-accessible ones except without some
private kernel stuff.  Also, by having the separation _within_ the  
kernel
tree, we can help trigger more thought about kernel/user ABI for new
drivers, because it's much more obvious exactly what the userspace
interface is when it goes in.  It also makes it much easier to have a
template header file to show driver authors how to get their user/kernel
interfaces correct the first time.  If someone sees something about this
that I'm missing, then please tell me, but as far as I can tell, keeping
this project out of the kernel would give "absolutely no gain" over
doing it in kernel, and would in fact result in massive amounts of
duplicated effort ("You forgot to patch user-kernel-headers to add your
shiny new IOCTL!").  It also is dismissive to suckers^H^H^H^H^H^Hpeople
who are willing to invest effort in cleaning up kernel headers, removing
messy include cycles, breaking up monoliths like linux/sched.h, etc.

Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things,  
because that
would also stop them from doing clever things.
   -- Doug Gwyn

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[H. Peter Anvin]

Paul Jackson wrote:
> 
> Why must the ABI headers be included by both kernel and user headers to
> be sane?
> 

Version creep.

	-hpa

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Kyle Moffett]

On Sep 12, 2005, at 11:47:56, Paul Jackson wrote:
> hpa wrote:
>
>> The only sane thing is to have a set of ABI headers with a clean,
>> specific set of rules, which is included by the kernel private  
>> headers,
>> as well as userspace.
>>
>
> Why must the ABI headers be included by both kernel and user  
> headers to
> be sane?
>
> Hmmm ... I'm not sure I want to ask that, actually.  I have this  
> feeling
> from the tone of your assertion that you can explain to me why such a
> header organization is the only one that fits your mental model of how
> these things are structured, but that communication between us may
> break down when you try to convince me that your mental model for this
> is the only correct one.

If we acknowledge the fact that syncing the release dates of two  
projects
is basically futile, especially given that under your system the kernel
headers would not change much/at-all to make the user-headers project
easier, then any feature X that appears in a new release of the kernel
will not be accessible from userspace tools without ignoring the  
point of
the user-headers project all together and having separate headers.   
Given
this, as well as the maintenance burden for those who would need to
maintain the user-headers (which would be nearly nil if the current
kernel headers could be cleaned up to the point which they could be used
instead), this project is lots of messy work either way, but in the long
run, if included into the upstream kernel, it will result in much less
duplication of effort and much cleaner code.

Cheers,
Kyle Moffett

--
Premature optimization is the root of all evil in programming
   -- C.A.R. Hoare

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Paul Jackson]

Kyle wrote:
> it will result in much less
> duplication of effort and much cleaner code.

It is not a foregone conclusion that combining two jobs into
one is less effort - that depends on how well the two jobs
merge.

Your comments remind me a little bit of the reassurance that
one hear's from a CEO justifying big merger.

Sometimes there's enough synergy to pay for the merger, sometimes
not.  The devil is in the details.

The devil really is in the details.

Since there is nothing further I have to contribute to these
details, I will return to my regularly scheduled cpuset programming.

Take care.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Bill Davidsen]

Kyle Moffett wrote:
> On Sep 12, 2005, at 11:47:56, Paul Jackson wrote:
> 
>> hpa wrote:
>>
>>> The only sane thing is to have a set of ABI headers with a clean,
>>> specific set of rules, which is included by the kernel private  headers,
>>> as well as userspace.
>>>
>>
>> Why must the ABI headers be included by both kernel and user  headers to
>> be sane?
>>
>> Hmmm ... I'm not sure I want to ask that, actually.  I have this  feeling
>> from the tone of your assertion that you can explain to me why such a
>> header organization is the only one that fits your mental model of how
>> these things are structured, but that communication between us may
>> break down when you try to convince me that your mental model for this
>> is the only correct one.
> 
> 
> If we acknowledge the fact that syncing the release dates of two  projects
> is basically futile, especially given that under your system the kernel
> headers would not change much/at-all to make the user-headers project
> easier, then any feature X that appears in a new release of the kernel
> will not be accessible from userspace tools without ignoring the  point of
> the user-headers project all together and having separate headers.   Given
> this, as well as the maintenance burden for those who would need to
> maintain the user-headers (which would be nearly nil if the current
> kernel headers could be cleaned up to the point which they could be used
> instead), this project is lots of messy work either way, but in the long
> run, if included into the upstream kernel, it will result in much less
> duplication of effort and much cleaner code.

The issue, as I see it, is not that the nifty new ioctl doesn't become 
instantly available, although that's not a small benefit of having one 
and only one set of user headers. The real benefit is avoiding the case 
where some part of the API *changes* and some feature stops working.

This is obviously uncommon, but not unheard of.

I see the greatest benefit from just not having two sets of headers, I 
believe all that stuff I learned in CS classes about not having two 
copies of stuff and assuming that they're the same. It would be less 
work to clean up the headers once, and let the folks who now maintain 
the separate headers become the "kernel janitors" to keep it clean.

Not my job, but we have someone offering to do the first cut at it, and 
it seems a desirable end result.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: [RFC][MEGAPATCH] Change __ASSEMBLY__ to __ASSEMBLER__ (defined by GCC from 2.95 to current CVS)

[Jeremy Fitzhardinge]

Kyle Moffett wrote:

> This would make life a million times easier for the UML people,
> the glibc people, the klibc people, and the linux-libc-headers
> maintainer


Valgrind could definitely use this; it currently has its own private
kernel ABI definitions, which are a pain.

    J

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 3, 2005, at 00:28:59, Erik Andersen wrote:
>> Absolutely not.  This would be a POSIX namespace violation; they
>> *must* use double-underscore types.
>
> I assume you are worried about the stuff under asm that ends up
> being included by nearly every header file in the world.  Of
> course asm must use double-underscore types.  But the thing is,
> the vast majority of the kernel headers live under
> linux/include/linux/ and do not use double-underscore types, they
> use kernel specific, non-underscored types such as s8, u32, etc.
> My copy of IEEE 1003.1 and my copy of ISO/IEC 9899:1999 both fail
> to prohibit using the shiny new ISO C99 type for the various
> #include <linux/*> header files, which is what I was suggesting.

Anything in linux/* that is included by userspace should not
presume that stdint.h has already been included or include it on
its own, because the userspace program may have already made its
own definitions of uint32_t, or it may not want them defined at
all.

> The world would be so much nicer a place if user space were free
> to #include linux/* header files rather than keeping a
> per-project private copy of all kernel structs of interest.

Exactly!  This is why I want to create kcore/* and kabi/* that
define the appropriate types, then both userspace and the kernel
could use whatever types fit their fancy, defined in terms of the
__kcore_ and __kabi_ types, which could be _depended_ on to exist
because they are guaranteed not to conflict with other namespaces

Cheers,
Kyle Moffett

--
I have yet to see any problem, however complicated, which, when you  
looked at
it in the right way, did not become still more complicated.
   -- Poul Anderson

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Kyle Moffett wrote:
> 
>> The world would be so much nicer a place if user space were free
>> to #include linux/* header files rather than keeping a
>> per-project private copy of all kernel structs of interest.
> 
> Exactly!  This is why I want to create kcore/* and kabi/* that
> define the appropriate types, then both userspace and the kernel
> could use whatever types fit their fancy, defined in terms of the
> __kcore_ and __kabi_ types, which could be _depended_ on to exist
> because they are guaranteed not to conflict with other namespaces
> 

Agreed.  We should use well-defined namespaces that won't conflict. 
However, I think the __[us][0-9]+ namespace can be considered 
well-established.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 3, 2005, at 01:57:26, H. Peter Anvin wrote:
> Kyle Moffett wrote:
>
>>> The world would be so much nicer a place if user space were free
>>> to #include linux/* header files rather than keeping a
>>> per-project private copy of all kernel structs of interest.
>> Exactly!  This is why I want to create kcore/* and kabi/* that
>> define the appropriate types, then both userspace and the kernel
>> could use whatever types fit their fancy, defined in terms of the
>> __kcore_ and __kabi_ types, which could be _depended_ on to exist
>> because they are guaranteed not to conflict with other namespaces
>
> Agreed.  We should use well-defined namespaces that won't conflict.
> However, I think the __[us][0-9]+ namespace can be considered
> well-established.

True, however, IMNSHO it would be much better if the kcore/kabi stuff  
had
a _consistent_ namespace as well.  If every macro begins with "__KABI_"
and every type and function with "__kabi_" (With a few function-like  
macro
exceptions, of course), then it is trivial to see where it originally  
came
from and provides a standard naming scheme that external parties can  
kind
of rely upon.  It also means there are fewer exceptions to remember when
coding.  My thought for the __[us][0-9]+ types is that they should still
be defined in linux/types.h for compatibility (outside of __KERNEL__)  
and
based off the __kabi_* types.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so
simple that there are obviously no deficiencies. And the other way is  
to make
it so complicated that there are no obvious deficiencies.  The first  
method is
far more difficult.
   -- C.A.R. Hoare

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Denis Vlasenko]

On Saturday 03 September 2005 08:55, Kyle Moffett wrote:
> On Sep 3, 2005, at 00:28:59, Erik Andersen wrote:
> >> Absolutely not.  This would be a POSIX namespace violation; they
> >> *must* use double-underscore types.
> >
> > I assume you are worried about the stuff under asm that ends up
> > being included by nearly every header file in the world.  Of
> > course asm must use double-underscore types.  But the thing is,
> > the vast majority of the kernel headers live under
> > linux/include/linux/ and do not use double-underscore types, they
> > use kernel specific, non-underscored types such as s8, u32, etc.
> > My copy of IEEE 1003.1 and my copy of ISO/IEC 9899:1999 both fail
> > to prohibit using the shiny new ISO C99 type for the various
> > #include <linux/*> header files, which is what I was suggesting.
> 
> Anything in linux/* that is included by userspace should not
> presume that stdint.h has already been included or include it on
> its own, because the userspace program may have already made its
> own definitions of uint32_t, or it may not want them defined at
> all.

Is this an excercise in academia? Userspace app which defines
uint32_t to anything different than 'typedef <appropriate int type>'
deserves the punishment, and one which does have such typedef
instead of #include stdint.h will not notice.

All these u32, uint32_t, __u32 end up typedef'ing to same
integer type anyway...

Why should we care of such 'struct uint32_t { ... };' pervert cases?
If we will do, I suspect we will end up with __________u32's.
We already have a lot of __'s:

linux/cache.h:#define ____cacheline_aligned __attribute__((__aligned__(SMP_CACHE_BYTES)))
linux/rcupdate.h:                             typeof(p) _________p1 = p; \

*three underscores* line count over kernel's include/:

# grep -r ___ . | wc -l
10129

and "only" two underscores:

# grep -r __ . | wc -l
72216
--
vda

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 3, 2005, at 11:36:22, Denis Vlasenko wrote:
> Is this an exercise in academia? Userspace app which defines
> uint32_t to anything different than 'typedef <appropriate int type>'
> deserves the punishment, and one which does have such typedef
> instead of #include stdint.h will not notice.

That's not the issue.  Say I do this (which is perfectly valid on
most platforms):

typedef unsigned int uint32_t;
#include <linux/loop.h>

What exactly should happen?  If linux/loop.h includes stdint.h to get
uint32_t, then I'll get duplicate definition errors.  If it omits
stdint.h, then uint16_t won't be defined (because the userspace app
doesn't think that it needs it) and I'll get undefined type errors.
Either way, depending on the existence or nonexistence of the POSIX
types in userspace-accessible kernel headers is not viable.

> All these u32, uint32_t, __u32 end up typedef-ing to same
> integer type anyway...

The point is to provide a type that _isn't_ in some standard so that
_we_ can define its inclusion rules.  If the standards had gone and
defined "Userspace must include stdint.h or define _all_ types
appropriately", then we would not have had this issue, but many apps
in userspace would cease to compile on standards compliant platforms.

Cheers,
Kyle Moffett

--
Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best  
answer:

"Why do musicians compose symphonies and poets write poems? They do  
it because
life wouldn't have any meaning for them if they didn't. That's why I  
draw
cartoons. It's my life."
   -- Charles Shultz

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Denis Vlasenko]

On Saturday 03 September 2005 19:33, Kyle Moffett wrote:
> On Sep 3, 2005, at 11:36:22, Denis Vlasenko wrote:
> > Is this an exercise in academia? Userspace app which defines
> > uint32_t to anything different than 'typedef <appropriate int type>'
> > deserves the punishment, and one which does have such typedef
> > instead of #include stdint.h will not notice.
> 
> That's not the issue.  Say I do this (which is perfectly valid on
> most platforms):
> 
> typedef unsigned int uint32_t;
> #include <linux/loop.h>
> 
> What exactly should happen?  If linux/loop.h includes stdint.h to get
> uint32_t, then I'll get duplicate definition errors.  If it omits
> stdint.h, then uint16_t won't be defined (because the userspace app
> doesn't think that it needs it) and I'll get undefined type errors.

I vote for a second choice. Do not #include stdint.h from the loop.h
(i.e. loop.h assumes that it is included (or unit32_t typedef'ed),
but doesn't do it itself).
--
vda

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Bill Davidsen]

H. Peter Anvin wrote:
> Followup to:  <20050902235833.GA28238@codepoet.org>
> By author:    Erik Andersen <andersen@codepoet.org>
> In newsgroup: linux.dev.kernel
> 
>><uClibc maintainer hat on>
>>That would be wonderful.
>></off>
>>
>>It would be especially nice if everything targeting user space
>>were to use only all the nice standard ISO C99 types as defined
>>in include/stdint.h such as uint32_t and friends...
>>
> 
> 
> Absolutely not.  This would be a POSIX namespace violation; they
> *must* use double-underscore types.

Could you explain why you think it would be a violation to use POSIX 
types instead of defining our own? That's what the types are for, to 
avoid having everyone define some slightly conflicting types.

The kernel predates C99, sort of, and it would be a massive but valuable 
  task to figure out where a type is really, for instance, 32 bits 
rather than "size of default int" in length, etc, and use POSIX types 
where they are correct. Fewer things to maintain, and would make it 
clear when something is 32 bits by default and when it really must be 32 
bits.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Sam Ravnborg]

 
> The kernel predates C99, sort of, and it would be a massive but valuable 
>  task to figure out where a type is really, for instance, 32 bits 
> rather than "size of default int" in length, etc, and use POSIX types 
> where they are correct. Fewer things to maintain, and would make it 
> clear when something is 32 bits by default and when it really must be 32 
> bits.
This has been discussed several times on lkml.
Ask google...
In short - the kernel provide its own namespace, and here __u32 etc is
used. And starting to change that would be a noisy effort with no or
limited gain neither for the kernel nor userspace.

	Sam

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Sam Ravnborg wrote:
>  
> 
>>The kernel predates C99, sort of, and it would be a massive but valuable 
>> task to figure out where a type is really, for instance, 32 bits 
>>rather than "size of default int" in length, etc, and use POSIX types 
>>where they are correct. Fewer things to maintain, and would make it 
>>clear when something is 32 bits by default and when it really must be 32 
>>bits.
> 
> This has been discussed several times on lkml.
> Ask google...
> In short - the kernel provide its own namespace, and here __u32 etc is
> used. And starting to change that would be a noisy effort with no or
> limited gain neither for the kernel nor userspace.
> 

More specifically, replacing u32 with uint32_t as the kernel-internal 
type would be fine and arguably useful; in fact, a handful of drivers I 
believe already use it.

Replacing __u32 with uint32_t would be wrong, because it would require 
some user space headers that cannot include <stdint.h> to have to do so. 
  POSIX has *very* specific rules for namespace pollution, and most 
libcs have to care about that (klibc doesn't, but it's an exception, not 
the rule.)

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 14, 2005, at 09:46:04, Bill Davidsen wrote:
> H. Peter Anvin wrote:
>> Followup to:  <20050902235833.GA28238@codepoet.org>
>> By author:    Erik Andersen <andersen@codepoet.org>
>> In newsgroup: linux.dev.kernel
>>> <uClibc maintainer hat on>
>>> That would be wonderful.
>>> </off>
>>>
>>> It would be especially nice if everything targeting user space  
>>> were to use only all the nice standard ISO C99 types as defined  
>>> in include/stdint.h such as uint32_t and friends...
>> Absolutely not.  This would be a POSIX namespace violation; they  
>> *must* use double-underscore types.
>
> Could you explain why you think it would be a violation to use  
> POSIX types instead of defining our own? That's what the types are  
> for, to avoid having everyone define some slightly conflicting types.
>
> The kernel predates C99, sort of, and it would be a massive but  
> valuable  task to figure out where a type is really, for instance,  
> 32 bits rather than "size of default int" in length, etc, and use  
> POSIX types where they are correct. Fewer things to maintain, and  
> would make it clear when something is 32 bits by default and when  
> it really must be 32 bits.

Argh, it seems I'm going to be giving this example forever!  Here's  
why this won't work.  We want to have sys/stat.h do something like  
the following:

# define __kabi_stat64 stat
# include <kabi/stat.h>
/* Now we expect struct stat to be defined with correct types */

Since struct stat in that case uses fixed-bit-size types, the header  
fine <kabi/stat.h> needs to include a file providing those types.  If  
it used stdint.h types, such as uint32_t, then it would need to  
#include <stdint.h> or provide the stdint.h types itself.  In order  
to remain POSIX compliant, sys/stat.h must not include stdint.h or  
assume that stdint.h is included or that those types were defined by  
the user program.  Therefore, kabi/*.h cannot use the stdint.h types  
at all!  The solution is a separate file, kabi/types.h, which  
properly defines __kabi_[su]{8,16,32,64} which are safe to include  
and reuse anywhere.  Then sys/types.h would look like this:

# include <kabi/types.h>
typedef __kabi_u8 u_int8;
typedef __kabi_s8 int8;
typedef __kabi_u16 u_int16;
typedef __kabi_s16 int16;
[...]

Cheers,
Kyle Moffett

--
Premature optimization is the root of all evil in programming
   -- C.A.R. Hoare

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[linux-os (Dick Johnson)]

On Wed, 14 Sep 2005, Kyle Moffett wrote:

> On Sep 14, 2005, at 09:46:04, Bill Davidsen wrote:
>> H. Peter Anvin wrote:
>>> Followup to:  <20050902235833.GA28238@codepoet.org>
>>> By author:    Erik Andersen <andersen@codepoet.org>
>>> In newsgroup: linux.dev.kernel
>>>> <uClibc maintainer hat on>
>>>> That would be wonderful.
>>>> </off>
>>>>
>>>> It would be especially nice if everything targeting user space
>>>> were to use only all the nice standard ISO C99 types as defined
>>>> in include/stdint.h such as uint32_t and friends...
>>> Absolutely not.  This would be a POSIX namespace violation; they
>>> *must* use double-underscore types.
>>
>> Could you explain why you think it would be a violation to use
>> POSIX types instead of defining our own? That's what the types are
>> for, to avoid having everyone define some slightly conflicting types.
>>
>> The kernel predates C99, sort of, and it would be a massive but
>> valuable  task to figure out where a type is really, for instance,
>> 32 bits rather than "size of default int" in length, etc, and use
>> POSIX types where they are correct. Fewer things to maintain, and
>> would make it clear when something is 32 bits by default and when
>> it really must be 32 bits.
>
> Argh, it seems I'm going to be giving this example forever!  Here's
> why this won't work.  We want to have sys/stat.h do something like
> the following:
>
> # define __kabi_stat64 stat
> # include <kabi/stat.h>
> /* Now we expect struct stat to be defined with correct types */
>
> Since struct stat in that case uses fixed-bit-size types, the header
> fine <kabi/stat.h> needs to include a file providing those types.  If
> it used stdint.h types, such as uint32_t, then it would need to
> #include <stdint.h> or provide the stdint.h types itself.  In order
> to remain POSIX compliant, sys/stat.h must not include stdint.h or
> assume that stdint.h is included or that those types were defined by
> the user program.  Therefore, kabi/*.h cannot use the stdint.h types
> at all!  The solution is a separate file, kabi/types.h, which
> properly defines __kabi_[su]{8,16,32,64} which are safe to include
> and reuse anywhere.  Then sys/types.h would look like this:
>

No No. The solution is to do it right. If the standard says that
the header file can't include a header file defining the types used
within that header file (and I don't think the "standard" says that
at all), then the correct solution is to include the correct header file
in the user program. It is truly just that simple.

The whole purpose of POSIX types, a.k.a. ISO C99: 7.18 Integer types,
<stdint.h> was to PREVENT the crap like you describe. We do NOT
need, for instance....

 	uint8, u8, u8_t, __u8, _u8, -> inf., to
correctly define an unsigned 8-bit char type. All we need is the
POSIX type, uint8_t, nothing else.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.13 on an i686 machine (5589.53 BogoMips).
Warning : 98.36% of all statistics are fiction.
.
I apologize for the following. I tried to kill it with the above dot :

****************************************************************
The information transmitted in this message is confidential and may be privileged.  Any review, retransmission, dissemination, or other use of this information by persons or entities other than the intended recipient is prohibited.  If you are not the intended recipient, please notify Analogic Corporation immediately - by replying to this message or by sending an email to DeliveryErrors@analogic.com - and destroy all copies of this information, including any attachments, without reading or disclosing them.

Thank you.

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

linux-os (Dick Johnson) wrote:
> 
> No No. The solution is to do it right. If the standard says that
> the header file can't include a header file defining the types used
> within that header file (and I don't think the "standard" says that
> at all), then the correct solution is to include the correct header file
> in the user program. It is truly just that simple.
> 

Dear Wrongbot,

No, it's not that simple.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 14, 2005, at 15:09:09, linux-os (Dick Johnson) wrote:
> No No. The solution is to do it right. If the standard says that
> the header file can't include a header file defining the types used
> within that header file (and I don't think the "standard" says that
> at all), then the correct solution is to include the correct header  
> file
> in the user program. It is truly just that simple.
>

I don't even have to say anything substantial in my response to this  
flame, because I've already said everything substantial that needs to  
be said, but just for clarity, let me repeat myself.

First, let me be repetitious and say this again:
> On Wed, 14 Sep 2005, Kyle Moffett wrote:
>> Argh, it seems I'm going to be giving this example forever!

Then:
>> If it [sys/types.h] used stdint.h types, such as uint32_t, then it  
>> would need to #include <stdint.h> or provide the stdint.h types  
>> itself.

Finally (with extra emphasis added):
>> In order to remain POSIX compliant, sys/stat.h _*MUST*_*NOT*_ not  
>> include stdint.h or
>> assume that stdint.h is included.

This means that the stat structure *CANNOT* use stdint.h types. Those  
are absolutely forbidden by the standard, because they have been used  
and reused, defined and redefined by userspace programs since the  
dawn of time.  There are standard definitions provided by libc if a  
program wants them, but libc _*MUST*_*NOT*_ force those definitions  
on anybody.  If you don't believe me, quit flaming and go read the  
standards yourself, that's exactly what they say, and for good reason  
too.

PS: This crap apology doesn't cut it, please use a different email  
service that does not append such garbage to your emails when sending  
stuff to the LKML:

> .
> I apologize for the following. I tried to kill it with the above dot :
>
> ****************************************************************
> The information transmitted in this message is confidential and may  
> be privileged.  Any review, retransmission, dissemination, or other  
> use of this information by persons or entities other than the  
> intended recipient is prohibited.  If you are not the intended  
> recipient, please notify Analogic Corporation immediately - by  
> replying to this message or by sending an email to  
> DeliveryErrors@analogic.com - and destroy all copies of this  
> information, including any attachments, without reading or  
> disclosing them.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to  
make it so simple that there are obviously no deficiencies. And the  
other way is to make it so complicated that there are no obvious  
deficiencies.  The first method is far more difficult.
   -- C.A.R. Hoare

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Jeff Dike]

On Thu, Sep 01, 2005 at 11:00:16PM -0400, Kyle Moffett wrote:
>   1)  There are a couple header files upon which almost everything
> else depends, among them {asm,linux}/{posix_,}types.h, which have
> some significant duplications.  Many of the archs have weird sizes
> for those types to preserve some backwards-compatibility ABI, but
> nowhere does it explain if there are any type-size restrictions in
> general.  I would propose that those headers be reorganized so that
> there are sane defaults for all the types in kabi/types.h, and
> archs that require different would #define exceptions in their
> kabi/arch-foo/types.h.  This would allow new archs to start with a
> sane standard ABI before it becomes set in stone.
> 
>   2)  There is a bunch of stuff that would be _really_ useful in
> userspace programs as well, even though not kernel ABI, such as
> list.h, atomic.h (with a few archs modified due to privilege
> restrictions), etc.  If there is interest, I would attempt to split
> off those headers into a kcore/kerncore/linuxcore/whatever inline
> header collection included in the linux distribution and installed
> as part of the kernel headers.

UML really needs something like this, both 1 and 2.  See
	http://groups.google.com/group/fa.linux.kernel/browse_thread/thread/34d3c02372861a5c/71816a3c7863ea2b?lnk=st&q=%22jeff+dike%22&rnum=27&hl=en#71816a3c7863ea2b
for my take on system.h and ptrace.h when a change in the host
architecture broke the UML build.

UML takes most of its headers from the underlying arch.  It simplifies
things since most of the definitions are usable in UML.  I don't have
to clone and maintain my versions of all the other arch headers.

OTOH, there are things in those headers which UML can't use, and these
are eliminated in various ways (undefining them after the include of
the host arch header, redefining them before the include).  But this
is a pain.

It has long been my opinion that splitting headers into userspace
usable and userspace unusable pieces is the right thing for UML.  Less
clear for the host arch.

Your post seems to indicate that there is a non-UML demand for exactly
this.


>   3)  What names are preferable for the above?  My personal
> preferences are "kabi" and "kcore", because those save the most
> typing for the sucker trying to do all this (IE: me), although if
> someone has good reasons otherwise, I'll listen.

If you look at my patch referenced above, I used -abi, which worked
well for ptrace.h, for which this is what the split amounted to.  For
system.h, it's not so good, since this is your case #2 above.  These
are random userspace usable things which don't appear in the ABI, like
the memory barriers, xchg, and alternative().

> I realize this project is only slightly short of massive, however I
> do have a bunch of time and am willing to do the grunt work if
> enlightened as to the community desires.  

You have my full support on this.

> I have a few different
> semi-patches almost ready, and I can probably finish up a couple
> this weekend if I can figure out which way people want to go. 

Grab my ptrace.h and system.h patches if you don't have them already.

				Jeff

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Followup to:  <20050902214231.GA10230@ccure.user-mode-linux.org>
By author:    Jeff Dike <jdike@addtoit.com>
In newsgroup: linux.dev.kernel
> 
> UML really needs something like this, both 1 and 2.  See
> 	http://groups.google.com/group/fa.linux.kernel/browse_thread/thread/34d3c02372861a5c/71816a3c7863ea2b?lnk=st&q=%22jeff+dike%22&rnum=27&hl=en#71816a3c7863ea2b
> for my take on system.h and ptrace.h when a change in the host
> architecture broke the UML build.
> 
> UML takes most of its headers from the underlying arch.  It simplifies
> things since most of the definitions are usable in UML.  I don't have
> to clone and maintain my versions of all the other arch headers.
> 
> OTOH, there are things in those headers which UML can't use, and these
> are eliminated in various ways (undefining them after the include of
> the host arch header, redefining them before the include).  But this
> is a pain.
> 
> It has long been my opinion that splitting headers into userspace
> usable and userspace unusable pieces is the right thing for UML.  Less
> clear for the host arch.
> 
> Your post seems to indicate that there is a non-UML demand for exactly
> this.
> 

There definitely is.  The kernel needs to export its ABI in a way that
userspace (UML, various libcs, etc) can import in a sane manner.  In
addition, the Linux kernel contains a fair bit of
architecture-specific support which go well beyond what one can
typically find in userspace, and it would be nice to have those.

The current linux-libc-headers aren't it, because they have a fair bit
of glibc-centric assumptions in those headers.  That's part of why
klibc doesn't use them.

We should probably also consider the licensing of headers that are
meant to be included into userspace.  Userspace still includes a fair
bit of GPL headers, which is technically not kosher.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 2, 2005, at 17:55:54, H. Peter Anvin wrote:
>> UML really needs something like this, both 1 and 2.  See
>>     http://groups.google.com/group/fa.linux.kernel/browse_thread/ 
>> thread/34d3c02372861a5c/71816a3c7863ea2b?lnk=st&q=%22jeff+dike% 
>> 22&rnum=27&hl=en#71816a3c7863ea2b
>> for my take on system.h and ptrace.h when a change in the host
>> architecture broke the UML build.
>>
>> UML takes most of its headers from the underlying arch.  It  
>> simplifies
>> things since most of the definitions are usable in UML.  I don't have
>> to clone and maintain my versions of all the other arch headers.
>>
>> OTOH, there are things in those headers which UML can't use, and  
>> these
>> are eliminated in various ways (undefining them after the include of
>> the host arch header, redefining them before the include).  But this
>> is a pain.
>>
>> It has long been my opinion that splitting headers into userspace
>> usable and userspace unusable pieces is the right thing for UML.   
>> Less
>> clear for the host arch.
>>
>> Your post seems to indicate that there is a non-UML demand for  
>> exactly
>> this.
>
> There definitely is.  The kernel needs to export its ABI in a way that
> userspace (UML, various libcs, etc) can import in a sane manner.  In
> addition, the Linux kernel contains a fair bit of
> architecture-specific support which go well beyond what one can
> typically find in userspace, and it would be nice to have those.
>
> The current linux-libc-headers aren't it, because they have a fair bit
> of glibc-centric assumptions in those headers.  That's part of why
> klibc doesn't use them.

What I would try to do is package up as much architecture/abi knowledge
in one place as possible, the former in kcore/kern-core/whatever, the
latter in kabi/kern-abi/linux-abi/whatever.  I would also try (as much
as possible), to make everything in those directories use some kind of
prefix guaranteed not to clash with other stuff, so list_add() for
example would become _kcore_list_add().  The linux kernel headers in
such a modified kernel would then just do this to make the kernel code
happy:
#ifdef __KERNEL__
# define list_add(x,y) _kcore_list_add(x,y)
/*....*/
#endif

My far-into-the-future ideal for this is to have a generic vDSO-type
library that is compiled into the kernel that provides a collection of
architecture-optimized routines available in both kernelspace and
userspace by mapping it into each process' address space.  Such a
library could effectively automatically provide correct and optimized
assembly routines for the currently booted CPU/arch/subarch/etc, so
that userspace tools could be compiled once and run on an entire
family of CPUs without modification.  On the other hand, for those
applications that need every last ounce of speed (Including parts of
the kernel), you could pass appropriate options to the compiler to
tell it to inline the assembly routines (alternative) for a single
CPU make/model.

Possibly some of the generic-arch stuff should be pushed back
upstream to GCC, maybe have __builtin_{s,u,i,f}{8,16,32,64,128} types,
etc, provided directly by GCC, so we don't have to mess with that
so much.

> We should probably also consider the licensing of headers that are
> meant to be included into userspace.  Userspace still includes a fair
> bit of GPL headers, which is technically not kosher.

I think that this is mostly a nonissue.  The copyright holders of the
headers/inline assembly/etc should look at perhaps licensing those
as LGPL or providing an exception to allow glibc, klibc, etc to link
with them.  On the other hand, were glibc to use the optimized
routines to provide the Standard C Library, programs using said
Standard C Library would not be infringing, because just like with
the "userspace <=syscall=> kernelspace" boundary, that does not imply
that the code is a derived work.  IANAL, however, so if you know one
who is willing to contribute some time, this might be an interesting
issue.  (Also:  What procedure might be required to get some of the
stuff relicensed as LGPL?  How do we find all significant copyright
holders/contributors from whom we need permission?)

Thanks for the encouraging posts!  It's good to hear that others are
interested in the project, because maybe I won't need to do it _all_
myself :-D.  I'll take a look at the patches mentioned, to get more
of an idea on the various technical issues.

Cheers,
Kyle Moffett

--
Simple things should be simple and complex things should be possible
   -- Alan Kay

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Kyle Moffett wrote:
> 
> My far-into-the-future ideal for this is to have a generic vDSO-type
> library that is compiled into the kernel that provides a collection of
> architecture-optimized routines available in both kernelspace and
> userspace by mapping it into each process' address space.  Such a
> library could effectively automatically provide correct and optimized
> assembly routines for the currently booted CPU/arch/subarch/etc, so
> that userspace tools could be compiled once and run on an entire
> family of CPUs without modification.  On the other hand, for those
> applications that need every last ounce of speed (Including parts of
> the kernel), you could pass appropriate options to the compiler to
> tell it to inline the assembly routines (alternative) for a single
> CPU make/model.
> 

I don't see why this should be compiled into the kernel.

	-hpa

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[Kyle Moffett]

On Sep 2, 2005, at 19:24:22, H. Peter Anvin wrote:
> Kyle Moffett wrote:
>> My far-into-the-future ideal for this is to have a generic vDSO-type
>> library that is compiled into the kernel that provides a  
>> collection of
>> architecture-optimized routines available in both kernelspace and
>> userspace by mapping it into each process' address space.  Such a
>> library could effectively automatically provide correct and optimized
>> assembly routines for the currently booted CPU/arch/subarch/etc, so
>> that userspace tools could be compiled once and run on an entire
>> family of CPUs without modification.  On the other hand, for those
>> applications that need every last ounce of speed (Including parts of
>> the kernel), you could pass appropriate options to the compiler to
>> tell it to inline the assembly routines (alternative) for a single
>> CPU make/model.
>
> I don't see why this should be compiled into the kernel.

The kernel already needs those same optimized routines for its own
operation (EX: all the ASM alternative() statements).  Since userspace
wants some of those as well, it would make sense to share them between
kernel and userspace and reduce the number of libraries you would need
to optimize when adding a new arch.  I don't think that we should add
optimized assembly for things that _aren't_ needed in the kernel, but
it should share what code it does have.

A side benefit of the vDSO method is that you would be able to take a
standard distro install and have the kernel automatically select the
correct vDSO image at runtime, simultaneously optimizing itself and
chunks of userspace.

Cheers,
Kyle Moffett

--
Unix was not designed to stop people from doing stupid things,  
because that
would also stop them from doing clever things.
   -- Doug Gwyn

Re: [RFC] Splitting out kernel<=>userspace ABI headers

[H. Peter Anvin]

Followup to:  <76E84FF2-A76E-4114-8E80-E07E6A497C7D@mac.com>
By author:    Kyle Moffett <mrmacman_g4@mac.com>
In newsgroup: linux.dev.kernel
> 
> The kernel already needs those same optimized routines for its own
> operation (EX: all the ASM alternative() statements).  Since userspace
> wants some of those as well, it would make sense to share them between
> kernel and userspace and reduce the number of libraries you would need
> to optimize when adding a new arch.  I don't think that we should add
> optimized assembly for things that _aren't_ needed in the kernel, but
> it should share what code it does have.
> 
> A side benefit of the vDSO method is that you would be able to take a
> standard distro install and have the kernel automatically select the
> correct vDSO image at runtime, simultaneously optimizing itself and
> chunks of userspace.
> 

First of all, a lot of these are inlines, and they derive a chunk of
their benefit from being inline.  Second, even if bundled with the
kernel, which I'm not sure is wise, there is no reason they can't just
be turned into libraries.  *Some functions* you're right, can be
optimized this way, but I'm not sure if that justifies compiling them
into the kernel that way.

	-hpa