[PATCH] 'make headers_install' kbuild target.

[PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

Attached is the current patch from mainline to my working tree at
git://git.infradead.org/~dwmw2/headers-2.6.git -- visible in gitweb at
http://git.infradead.org/?p=users/dwmw2/headers-2.6.git;a=summary

It adds a 'make headers_install' target to the kernel makefiles, which
exports a subset of the headers and runs 'unifdef' on them to clean them
up for installation in /usr/include.

You'll need unifdef, which is available at 
http://www.cs.cmu.edu/~ajw/public/dist/unifdef-1.0.tar.gz and can
probably be put into our scripts/ directory since it's BSD-licensed.

I expect the kbuild folks to reimplement what I've done in the Makefile,
but it works well enough to get us started. The text file listing the
header files will probably want to change -- maybe we'll have a file in
each directory listing the exportable files in that directory, or maybe
we'll put a marker in the public files which we can grep for. I don't
care much.

Implementation details aside, the point is that we can now work on
refining the choice of headers to be exported, and more importantly we
can start fixing the _contents_ of those headers so that nothing which
should be private is exported in them outside #ifdef __KERNEL__.

I've chosen headers in the generic directories and in asm-powerpc; the
other asm directories could do with a proper selection being made; the
rest of the current list is just inherited from Fedora's
glibc-kernheaders package for now.

For a start, the headers I've marked for export are sometimes including
headers which _weren't_ so marked, and hence which don't exist in our
exported set of headers. I've started to move those inclusions into
#ifdef __KERNEL__ where appropriate, but there's more of that to do
before we can even use these for building anything and actually start to
test them in earnest.

Adrian, I'm hoping we can persuade you to help us audit the resulting
contents of usr/include/* and apply your usual treatment to the headers
until it looks sane. That assistance would be very much appreciated.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Adrian Bunk]

On Sat, Apr 22, 2006 at 03:17:20AM +0100, David Woodhouse wrote:
>...
> Implementation details aside, the point is that we can now work on
> refining the choice of headers to be exported, and more importantly we
> can start fixing the _contents_ of those headers so that nothing which
> should be private is exported in them outside #ifdef __KERNEL__.
> 
> I've chosen headers in the generic directories and in asm-powerpc; the
> other asm directories could do with a proper selection being made; the
> rest of the current list is just inherited from Fedora's
> glibc-kernheaders package for now.
> 
> For a start, the headers I've marked for export are sometimes including
> headers which _weren't_ so marked, and hence which don't exist in our
> exported set of headers. I've started to move those inclusions into
> #ifdef __KERNEL__ where appropriate, but there's more of that to do
> before we can even use these for building anything and actually start to
> test them in earnest.
> 
> Adrian, I'm hoping we can persuade you to help us audit the resulting
> contents of usr/include/* and apply your usual treatment to the headers
> until it looks sane. That assistance would be very much appreciated.

My thirst thought is:
Is this really the best approach, or could this be done better?

I'm currently more a fan of a separate kabi/ subdir with headers used by 
both headers under linux/ and userspace.

Why?

Unless I'm misunderstanding this, your changes are giving a result 
identical result to simply using the current kernel headers (stripping 
the #ifdef __KERNEL__ stuff doesn't change anything).

If we want to define an ABI and do it right, the resulting ABI headers
should be write-only, and should even continue to contain the ABI for
code already removed from the kernel.

It should be possible to do this incrementically:
- without breaking the kernel at any time
- with no userspace breakages when switching to the ABI headers for 
  userspace (there might be corner cases, and if e.g. the asm-i386/atomic.h
  abuse by MySQL breaks that's not a loss)

This might take a bit longer, but as a result we have:
- an ABI where diffstat is able to tell us if someone has touched the ABI
  (can git trigger electric shocks for everyone trying to commit or pull 
   changes to the kabi/ subdir?  ;-)  )
- as a side effect, some cleanup of the current kernel headers

Unless someone can tell me a reason why this wouldn't work (except for 
being a bit more work than your approach), this is the approach I have 
in mind for working on.

> dwmw2

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 11:33 +0200, Adrian Bunk wrote:
> My thirst thought is:
> Is this really the best approach, or could this be done better?

I think it's the best way to start, although I agree with you entirely
about what we should strive for in the end.

> I'm currently more a fan of a separate kabi/ subdir with headers used by 
> both headers under linux/ and userspace.

I agree -- I'd like to see that too. But Linus doesn't like that
approach very much. 

> Unless I'm misunderstanding this, your changes are giving a result 
> identical result to simply using the current kernel headers (stripping 
> the #ifdef __KERNEL__ stuff doesn't change anything).

It's not quite the same. Some headers which just shouldn't be there at
all are removed -- and others can no longer be abused by defining
__KERNEL__ to get at stuff which shouldn't be there.

Also, if we have an 'exported' set of files which is supposed to be
clean, we can easily _see_ when there's stuff which shouldn't be there.
It makes the cleanup easier, by making the mess more obvious. We can
also take diffs of the output between one kernel and the next, applying
electric shocks as necessary, as you suggest later.

It's a small step, but it's the _first_ step towards the point we want
to reach, and it's something we're likely to get away with.

Ideally, I'd like to proceed by splitting files into user-visible and
kernel-private parts in _separate_ headers, so that the 'unifdef' part
becomes unnecessary (and __KERNEL__ disappears entirely). I've done some
of that already in include/mtd). It would also be nice if we could then
put the user-visible files into a separate directory, so that the
'headers_export' step becomes nothing more than a 'cp -a'. We do need to
do this incrementally though, and I think this is going to be the best
way to do it, and to get it accepted.

> Unless someone can tell me a reason why this wouldn't work (except for 
> being a bit more work than your approach), this is the approach I have 
> in mind for working on.

Your approach is basically what we proposed at last year's Kernel
Summit. It was shot down though, so we're trying to start simple and do
it incrementally. 

The important thing is that we all get our editors out and clean up the
_contents_ our own headers, and actually start to _think_ about the
visibility of any new header-file content we introduce. Let's not
concentrate too much on the implementation details of how we actually
get those to userspace.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Adrian Bunk]

On Sat, Apr 22, 2006 at 01:03:03PM +0100, David Woodhouse wrote:
> On Sat, 2006-04-22 at 11:33 +0200, Adrian Bunk wrote:
>...
> Ideally, I'd like to proceed by splitting files into user-visible and
> kernel-private parts in _separate_ headers, so that the 'unifdef' part
> becomes unnecessary (and __KERNEL__ disappears entirely). I've done some
> of that already in include/mtd). It would also be nice if we could then
> put the user-visible files into a separate directory, so that the
> 'headers_export' step becomes nothing more than a 'cp -a'. We do need to
> do this incrementally though, and I think this is going to be the best
> way to do it, and to get it accepted.

This sounds like what I had in mind.

> > Unless someone can tell me a reason why this wouldn't work (except for 
> > being a bit more work than your approach), this is the approach I have 
> > in mind for working on.
> 
> Your approach is basically what we proposed at last year's Kernel
> Summit. It was shot down though, so we're trying to start simple and do
> it incrementally. 

What was the recommended way for getting userspace header at last year's 
kernel summit?

> The important thing is that we all get our editors out and clean up the
> _contents_ our own headers, and actually start to _think_ about the
> visibility of any new header-file content we introduce. Let's not
> concentrate too much on the implementation details of how we actually
> get those to userspace.

Currently, it's said the kernel headers aren't suitable for userspace.

After the cleanups you propose, the kernel headers will be suitable for 
userspace (the copy steps you propose are not required, distributions 
could equally start to copy the verbatim headers again).

If everyone is working in a different direction, this is only wasting 
work.

> dwmw2

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 14:38 +0200, Adrian Bunk wrote:
> What was the recommended way for getting userspace header at last
> year's kernel summit?

It was said that we need _incremental_ changes, and this is an attempt
to satisfy that request.

> > The important thing is that we all get our editors out and clean up the
> > _contents_ our own headers, and actually start to _think_ about the
> > visibility of any new header-file content we introduce. Let's not
> > concentrate too much on the implementation details of how we actually
> > get those to userspace.
> 
> Currently, it's said the kernel headers aren't suitable for userspace.

Indeed they aren't.

> After the cleanups you propose, the kernel headers will be suitable for 
> userspace (the copy steps you propose are not required, distributions 
> could equally start to copy the verbatim headers again).

After the _first_ stage of the cleanups I propose, the export step will
still be necessary. You'll need to pick those headers which are intended
to be user-visible, and leave behind those which are not. 

If we actually go on to abolish __KERNEL__ and move the public headers
to a separate directory, you're right -- as I said, one day hopefully
it'll just be 'cp -a'. But that is not the _first_ stage. We need to do
this incrementally.

> If everyone is working in a different direction, this is only wasting 
> work.

The stated plan is to start with a simple export mechanism which lets us
see the mess we have at the moment and work on the _real_ problem -- the
actual contents of the headers. Then to proceed towards the goal you
stated, which is what we wanted all along.

Who would be working in a different direction? 

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Adrian Bunk]

On Sat, Apr 22, 2006 at 01:48:43PM +0100, David Woodhouse wrote:
> On Sat, 2006-04-22 at 14:38 +0200, Adrian Bunk wrote:
> > What was the recommended way for getting userspace header at last
> > year's kernel summit?
> 
> It was said that we need _incremental_ changes, and this is an attempt
> to satisfy that request.
> 
> > > The important thing is that we all get our editors out and clean up the
> > > _contents_ our own headers, and actually start to _think_ about the
> > > visibility of any new header-file content we introduce. Let's not
> > > concentrate too much on the implementation details of how we actually
> > > get those to userspace.
> > 
> > Currently, it's said the kernel headers aren't suitable for userspace.
> 
> Indeed they aren't.
> 
> > After the cleanups you propose, the kernel headers will be suitable for 
> > userspace (the copy steps you propose are not required, distributions 
> > could equally start to copy the verbatim headers again).
> 
> After the _first_ stage of the cleanups I propose, the export step will
> still be necessary. You'll need to pick those headers which are intended
> to be user-visible, and leave behind those which are not. 
> 
> If we actually go on to abolish __KERNEL__ and move the public headers
> to a separate directory, you're right -- as I said, one day hopefully
> it'll just be 'cp -a'. But that is not the _first_ stage. We need to do
> this incrementally.
>...

Why can't the splitting happen incrementally?

Assume you have a header include/linux/foo.h:
- Add an #include <kabi/linux/foo.h> at the top.
- Move the part of the contents that is part of the userspace ABI to 
  include/kabi/linux/foo.h.

When this is done for all headers containing parts of the userspace ABI:
- test the kabi/ headers in userspace
- review all headers under kabi/ since what's there will become a fixed
  ABI that can never be changed
- test the kabi/ headers in userspace
- make the ABI headers official

For kernel code, this header splitting should be completely transparent
(and nothing outside include/ should directly include headers under 
include/kabi/).

For userspace, this will be one switch from the many different header 
packages floating around to the new ABI headers, but it should break 
nearly none usespace applications (it will break some abuses, but
that's OK).

> dwmw2

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 15:20 +0200, Adrian Bunk wrote:
> Why can't the splitting happen incrementally?

It can, and has already started that way. There's no reason why the
'make headers_export' mechanism can't work with that -- it already does,
because it exports the include/mtd directory and nothing from
include/linux/mtd -- as I said, ideally the mechanism ends up being just
'cp -a' on certain directories. And then it can be abolished. We've got
a long way to go before we get there, though.

> Assume you have a header include/linux/foo.h:
> - Add an #include <kabi/linux/foo.h> at the top.
> - Move the part of the contents that is part of the userspace ABI to 
>   include/kabi/linux/foo.h.

Absolutely. That's what I've done with MTD headers already, although the
directory names are different. The directory names don't _matter_
either, because important part was that the files themselves are cleaned
up.

Linus isn't keen on splitting it into a new directory, and I don't want
to start off by demanding that. As I said, the important part of the
above is the bit where one of us goes to the file with an editor and
identifies the public parts vs. the private parts, then splits them up
-- possibly with #ifdef __KERNEL__, but _preferably_ into separate
files. And it doesn't _matter_ which directories we put those files
into, for now. I don't want to talk about it _yet_ because it's just
taking attention away from the real problem.

The more we screw around with such minutiae, the less likely we are to
get traction with Linus -- despite the fact that almost everyone who's
expressed an opinion is _agreeing_ with you about where we want to end
up.

We need to keep it simple and unintrusive to start with. Concentrate on
the _contents_ and then we can deal with the less important details
later.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Adrian Bunk]

On Sat, Apr 22, 2006 at 02:36:04PM +0100, David Woodhouse wrote:
> On Sat, 2006-04-22 at 15:20 +0200, Adrian Bunk wrote:
>...
> > Assume you have a header include/linux/foo.h:
> > - Add an #include <kabi/linux/foo.h> at the top.
> > - Move the part of the contents that is part of the userspace ABI to 
> >   include/kabi/linux/foo.h.
> 
> Absolutely. That's what I've done with MTD headers already, although the
> directory names are different. The directory names don't _matter_
> either, because important part was that the files themselves are cleaned
> up.
> 
> Linus isn't keen on splitting it into a new directory, and I don't want
> to start off by demanding that. As I said, the important part of the
> above is the bit where one of us goes to the file with an editor and
> identifies the public parts vs. the private parts, then splits them up
> -- possibly with #ifdef __KERNEL__, but _preferably_ into separate
> files. And it doesn't _matter_ which directories we put those files
> into, for now. I don't want to talk about it _yet_ because it's just
> taking attention away from the real problem.
> 
> The more we screw around with such minutiae, the less likely we are to
> get traction with Linus -- despite the fact that almost everyone who's
> expressed an opinion is _agreeing_ with you about where we want to end
> up.
> 
> We need to keep it simple and unintrusive to start with. Concentrate on
> the _contents_ and then we can deal with the less important details
> later.

Sorry, but I'm not a fan of doing much more work than required instead 
of getting a consensus first and then implementing the solution.

Let's agree on the way to go first, give me two weeks, and I can submit 
a first batch of the header splitting that will both not break any 
kernel code and be enough for compiling most of the userspace.

Perhaps two weeks are too optimistic for some parts considering the 
state of our headers, but getting it done before OLS seems realistic.

But otherwise, I think I have better things to do in the part of my 
spare time I'm devoting to Linux kernel development.

> dwmw2

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] 'make headers_install' kbuild target.

[Sam Ravnborg]

On Sat, Apr 22, 2006 at 01:03:03PM +0100, David Woodhouse wrote:
> On Sat, 2006-04-22 at 11:33 +0200, Adrian Bunk wrote:
> > My thirst thought is:
> > Is this really the best approach, or could this be done better?
> 
> I think it's the best way to start, although I agree with you entirely
> about what we should strive for in the end.
> 
> > I'm currently more a fan of a separate kabi/ subdir with headers used by 
> > both headers under linux/ and userspace.
> 
> I agree -- I'd like to see that too. But Linus doesn't like that
> approach very much.
Thats bacause the kabi subdir is broken by design.
Any approach that does not take into account the existing userbase is
broken by design and should be avoided.
The only sensible solution is to move out the kernel internal headers
from include/* to somewhere else.
And then slowly but steady let include/linux and include/asm-* be the
KABI.

In this way we avoid breaking a lot of current users. And new users will
continue to just work.

	Sam

Re: [PATCH] 'make headers_install' kbuild target.

[Adrian Bunk]

On Sat, Apr 22, 2006 at 04:14:10PM +0200, Sam Ravnborg wrote:
> On Sat, Apr 22, 2006 at 01:03:03PM +0100, David Woodhouse wrote:
> > On Sat, 2006-04-22 at 11:33 +0200, Adrian Bunk wrote:
> > > My thirst thought is:
> > > Is this really the best approach, or could this be done better?
> > 
> > I think it's the best way to start, although I agree with you entirely
> > about what we should strive for in the end.
> > 
> > > I'm currently more a fan of a separate kabi/ subdir with headers used by 
> > > both headers under linux/ and userspace.
> > 
> > I agree -- I'd like to see that too. But Linus doesn't like that
> > approach very much.
> Thats bacause the kabi subdir is broken by design.
> Any approach that does not take into account the existing userbase is
> broken by design and should be avoided.
> The only sensible solution is to move out the kernel internal headers
> from include/* to somewhere else.
> And then slowly but steady let include/linux and include/asm-* be the
> KABI.
>...

What exactly is the problem with creating the userspace ABI in 
include/kabi/ and letting distributions do an
  cd /usr/include && ln -s kabi/* .
?

Or with creating the userspace ABI in include/kabi/ and letting 
distributions install the subdirs of include/kabi/ directly under 
/usr/include?

These are two doable approaches with a new kabi/ that avoid needless 
breaking of userspace.

> 	Sam

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 16:11 +0200, Adrian Bunk wrote:
> Sorry, but I'm not a fan of doing much more work than required instead 
> of getting a consensus first and then implementing the solution.

It _isn't_ significantly more work. The _real_ work is in reading
through the header files and deciding which parts should be private and
which can be public, then splitting them up accordingly into separate
files (or using 'ifdef __KERNEL__' if appropriate). That's the kind of
thing that you seem particularly good at, which is why I've asked if you
could help us with it.

Moving the public files from one directory to another, if they've been
suitably marked or listed somewhere, is _trivial_. Even if you've used
#ifdef __KERNEL__ it's simple enough to do it automatically with tools
like unifdef. The _real_ work which requires human attention is the same
either way.

But if you're not willing to help, that's fine. I just thought you'd be
particularly suited to the task, that's all.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Sam Ravnborg]

> >...
> 
> What exactly is the problem with creating the userspace ABI in 
> include/kabi/ and letting distributions do an
>   cd /usr/include && ln -s kabi/* .
> ?
> 
> Or with creating the userspace ABI in include/kabi/ and letting 
> distributions install the subdirs of include/kabi/ directly under 
> /usr/include?
> 
> These are two doable approaches with a new kabi/ that avoid needless 
> breaking of userspace.

First off:
There are many other users that poke direct in the kernel source also.

Secondly and more importantly:
Introducing kabi/ you will have a half solution where several users will
have to find their stuff in two places for a longer period.
kabi/ does not allow you to do it incrementally - it requires you to
move everything over from a start.
You may argue that you can just move over a little bit mroe than needed
but then we ruin the incremental approach.

	Sam

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 16:20 +0200, Adrian Bunk wrote:
> > Thats bacause the kabi subdir is broken by design.
> > Any approach that does not take into account the existing userbase is
> > broken by design and should be avoided.
> > The only sensible solution is to move out the kernel internal headers
> > from include/* to somewhere else.
> > And then slowly but steady let include/linux and include/asm-* be the
> > KABI.
> >...

The 'make headers_export' stage also works just as well as the above,
and it gives us something more concrete to work on, where we can _see_
the bits which are needlessly exported and start to clean them up.

> What exactly is the problem with creating the userspace ABI in 
> include/kabi/ and letting distributions do an
>   cd /usr/include && ln -s kabi/* .
> ?
> 
> Or with creating the userspace ABI in include/kabi/ and letting 
> distributions install the subdirs of include/kabi/ directly under 
> /usr/include?

The problem is that Linus is unlikely to accept a trivial cleanup patch
which, for example, removes the contents of linux/auxvec.h and creates
kabi/auxvec.h. He'll argue, as he did last year, that it's moving stuff
around for the sake of it.

On the other hand, he _would_ be likely to accept patches which split
existing files into two _within_ the include/linux directory. And that's
the important part of the task; it doesn't _matter_ where the files are
put, because we can deal with that in the 'export' stage, and we can
trivially move them around later anyway once we do have a consensus.

We're doing it this way because we want to get on with the real work
without getting bogged down in a discussion with Linus about the
pointless details. Unfortunately we're getting bogged down in a
discussion about the pointless details anyway.

If you can get the trivial patches to move stuff into kabi/ merged,
that's _FINE_. Go wild. If you can't, then it's still useful to do the
same cleanups but keep all the resulting files in the same directories.

The 'headers_export' make target is still useful in the meantime,
because it produces the set of headers which we expect people to put
into /usr/include, and which we can do sanity checks on.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Adrian Bunk]

On Sat, Apr 22, 2006 at 03:26:14PM +0100, David Woodhouse wrote:
> On Sat, 2006-04-22 at 16:11 +0200, Adrian Bunk wrote:
> > Sorry, but I'm not a fan of doing much more work than required instead 
> > of getting a consensus first and then implementing the solution.
> 
> It _isn't_ significantly more work. The _real_ work is in reading
> through the header files and deciding which parts should be private and
> which can be public, then splitting them up accordingly into separate
> files (or using 'ifdef __KERNEL__' if appropriate). That's the kind of
> thing that you seem particularly good at, which is why I've asked if you
> could help us with it.
> 
> Moving the public files from one directory to another, if they've been
> suitably marked or listed somewhere, is _trivial_. Even if you've used
> #ifdef __KERNEL__ it's simple enough to do it automatically with tools
> like unifdef. The _real_ work which requires human attention is the same
> either way.

The problem is you need #ifdef's everywhere.

If part of a header file is part of the userspace ABI, this often means 
that you need #ifdef __KERNEL__'s for the #include's of headers that 
will not be part of the userspace ABI (like linux/compiler.h).

And how do you express that in header foo.h, the userspace part requires 
the userspace part of bar.h, while the kernel-internal part of foo.h 
also requires the kernel-internal part of bar.h?

And reading through header files doesn't become easier after adding five 
#ifdef __KERNEL__'s to a header file.

> But if you're not willing to help, that's fine. I just thought you'd be
> particularly suited to the task, that's all.

I'm sorry, but I don't like your approach.

> dwmw2

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 16:28 +0200, Sam Ravnborg wrote:
> First off:
> There are many other users that poke direct in the kernel source also.
> 
> Secondly and more importantly:
> Introducing kabi/ you will have a half solution where several users will
> have to find their stuff in two places for a longer period.
> kabi/ does not allow you to do it incrementally - it requires you to
> move everything over from a start.

Not really, because we have a 'make headers_install' target which puts
everything together in one place, cleaned up, for userspace. It can
quite happily pick up files from both the existing directories and the
new kabi/ directory. 

If Adrian wants to move stuff into a kabi/ directory as part of the
cleanups, that's fine. I think it's pointless, and I think it reduces
his chances of actually getting his cleanups merged, but if that's the
only way he'll contribute his time, then so be it. It doesn't do any
harm, technically.

But really, this whole discussion is counterproductive, IMO. 

The post-processing step to export headers to userspace is fine for now.
It lets us get on with the real work of cleaning the headers up. We've
been blocked on the trivia for too long already, and we're going down
that path again.

Once we have the actual contents of the headers classified as public vs.
private, and we know what we've got, _THEN_ we will have a clearer idea
of how those public headers should be organised. And it's _trivial_ to
move stuff around once it's been split out. It just doesn't _matter_
where we put it, for now.

What I'd really like to do is go into KS with the headers already
cleaned up, using 'make headers_install'. Show Linus the results, have a
coherent plan for how it _should_ be laid out so that the 'make
headers_install' step becomes just a copy, as Adrian and everyone else
wants. And have him actually agree to it this time.

We can't do it the other ways round. We tried that. If that was the only
way we could get Adrian to help us, then that's unfortunate, because
_this_ is the only way I think Linus is going to take it.

The end result is fairly much the same either way, and the amount of
work is the same. 

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Adrian Bunk]

On Sat, Apr 22, 2006 at 04:28:53PM +0200, Sam Ravnborg wrote:
> > >...
> > 
> > What exactly is the problem with creating the userspace ABI in 
> > include/kabi/ and letting distributions do an
> >   cd /usr/include && ln -s kabi/* .
> > ?
> > 
> > Or with creating the userspace ABI in include/kabi/ and letting 
> > distributions install the subdirs of include/kabi/ directly under 
> > /usr/include?
> > 
> > These are two doable approaches with a new kabi/ that avoid needless 
> > breaking of userspace.
> 
> First off:
> There are many other users that poke direct in the kernel source also.

Kernel space users?
User space users?

Can you give an example of what you are thinking of?

> Secondly and more importantly:
> Introducing kabi/ you will have a half solution where several users will
> have to find their stuff in two places for a longer period.
> kabi/ does not allow you to do it incrementally - it requires you to
> move everything over from a start.
> You may argue that you can just move over a little bit mroe than needed
> but then we ruin the incremental approach.

For kernel space, you can do it incrementally, since the whole kabi/ 
stuff should be transparent for in-kernel uses.

For user space, you need one switch.
But this switch goes from the current mess with several independent 
user space header implementations to one official implementation.

> 	Sam

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 16:44 +0200, Adrian Bunk wrote:
> The problem is you need #ifdef's everywhere.

No. You don't need ifdefs _anywhere_. Go and take a look at the MTD
headers which I already cleaned up.

You can split the messy files into two (or more) files and _keep_ them
in the same directory structure, although admittedly I did move the
user-visible parts out into include/mtd/ -- with the 'headers_install'
make target I didn't need to do that though.

> If part of a header file is part of the userspace ABI, this often means 
> that you need #ifdef __KERNEL__'s for the #include's of headers that 
> will not be part of the userspace ABI (like linux/compiler.h).

Er, if you're including those headers in a _purely_ user-visible file
then they're an error and should be removed altogether.

And if you're including those headers in a file which already _has_ some
ifdef __kernel__ in it, then you can just move the inclusion so that it
lands within the existing ifdefs. Like this, for example:
http://git.infradead.org/?p=users/dwmw2/headers-2.6.git;a=commitdiff;h=90a333de75b3f5efdc332f34f53768286967c306

> And how do you express that in header foo.h, the userspace part requires 
> the userspace part of bar.h, while the kernel-internal part of foo.h 
> also requires the kernel-internal part of bar.h?

You use foo-user.h, and foo-kernel.h. Or you include <kabi/foo.h> from
linux/foo.h if you can get Linus to take the patches. Nothing prevents
you from doing that. Please do that if that's what you want.

> I'm sorry, but I don't like your approach.

OK. This approach lets you send patches to Linus to move user-visible
stuff into a kabi/ directory if you can actually get him to accept such
patches. It allows you to move stuff over to a kabi/ directory
incrementally, without having to move everything in one go -- the export
process can pick files from wherever they happen to be, and put them in
the appropriate place in the exported header tree.

This approach _also_ allows those of us who are being more realistic to
do slightly less ambitious work which is just as useful but more likely
to get accepted, because we do the same cleanups to the _code_ but
without mucking around with new directories. 

But we're open to alternative plans -- what approach would you prefer?

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 15:20 +0200, Adrian Bunk wrote:
> On Sat, Apr 22, 2006 at 01:48:43PM +0100, David Woodhouse wrote:
> > On Sat, 2006-04-22 at 14:38 +0200, Adrian Bunk wrote:
> > > What was the recommended way for getting userspace header at last
> > > year's kernel summit?
> > 
> > It was said that we need _incremental_ changes, and this is an attempt
> > to satisfy that request.

Let me expand on that...

There are a number of steps which we need to take.

1. Identify those files which contain stuff which should be
user-visible. I've done a first pass of this for arch-independent files
and asm-powerpc already. We need it done for other architectures.

2. Examine the contents of those files, and separate private parts from
parts which should be user-visible. This can be by splitting them into
separate files, or by using ifdefs -- it doesn't matter conceptually at
_this_ stage as long as the important work of _reading_ the code and
identifying it as either public or private is done. All else can be
scripted after that.

This much is undisputed -- _everyone_ agrees that it's necessary. We can
do that, and we can have a fairly simple post-processing stage which
creates a set of 'sanitised' kernel headers which we can inspect, and
which we can compare between releases. This is _also_ the bulk of the
real work. It's where we actually have to look at the code and _think_
about it.

But it's not _all_ we want to do, ideally. Ideally we want to take it
further...

3. Eliminate all the #ifdef __KERNEL__. Ifdefs are horrid -- we all know
that. We should split public bits into entirely separate files. Once
it's marked with #ifdef __KERNEL__ after stage #2, it's fairly simple to
split the files up -- it's scriptable; doesn't really need much thought.

4. Clean up the directory structure. It really ought to be just a copy
of certain directories, rather than picking selected files from
include/linux et al. Again, once we _have_ the public files marked in
some way, it's trivial to move them around once we agree where they
should go.

Linus has said he doesn't want to jump straight in at #4 and start
moving things around en masse. Adrian, you seem to be saying you won't
help _unless_ you can send patches which mix #2 and #4 instead of doing
it incrementally. 

Now, I'm perfectly happy with patches which do that, if you can actually
get them merged. The export step can happily pick up the headers you've
moved to include/kabi/ and put them back where userspace expects to see
them for now.

In fact, I think it's best to hold off on #4 for now -- not only because
Linus won't take the patches, but because once we've done steps 1-3
above, we'll have a _much_ clearer idea of what we have left and how it
should best be laid out. Maybe once we've moved the crap out of the way
we'll end up wanting chrdev/*.h, ioctl/*.h, sockopt/*.h etc....

But even though I think it's premature, I still don't _mind_ moving
stuff into kabi/ -- because as I said it's _trivial_ to move stuff
around after it's been cleaned up. That part isn't the interesting part
of the problem, and I'm uninterested in arguing about it. 

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Arnd Bergmann]

On Saturday 22 April 2006 17:30, David Woodhouse wrote:
> 1. Identify those files which contain stuff which should be
> user-visible. I've done a first pass of this for arch-independent files
> and asm-powerpc already. We need it done for other architectures.

FWIW, I've just gone through the s390 specific header files in 2.6.17-rc2
to see who is supposed to see them, resulting in

** strictly kernel only:
bug.h                extmem.h       pgtable.h         string.h
bugs.h               futex.h        processor.h       suspend.h
cacheflush.h         hardirq.h      qdio.h            system.h
ccwdev.h             idals.h        qeth.h            thread_info.h
ccwgroup.h           io.h           rwsem.h           timer.h
checksum.h           irq.h          s390_ext.h        tlb.h
cio.h                kmap_types.h   s390_rdev.h       tlbflush.h
compat.h             local.h        scatterlist.h     todclk.h
cpcmd.h              lowcore.h      sections.h        topology.h
cputime.h            mathemu.h      segment.h         uaccess.h
current.h            mmu.h          semaphore.h       unaligned.h
delay.h              mmu_context.h  setup.h           user.h
div64.h              mutex.h        sfp-machine.h     vtoc.h
dma-mapping.h        namei.h        sigp.h            xor.h
dma.h                pci.h          smp.h
ebcdic.h             percpu.h       spinlock.h
emergency-restart.h  pgalloc.h      spinlock_types.h

** have stuff of interest to user space:
cmb.h    ioctls.h  param.h        shmbuf.h      sockios.h   timex.h
dasd.h   ipc.h     poll.h         shmparam.h    stat.h      types.h
debug.h  ipcbuf.h  posix_types.h  sigcontext.h  statfs.h    ucontext.h
errno.h  mman.h    ptrace.h       siginfo.h     tape390.h   unistd.h
fcntl.h  msgbuf.h  resource.h     signal.h      termbits.h
ioctl.h  page.h    sembuf.h       socket.h      termios.h

** non-trivial to decide, need to coordinate with other archs:
a.out.h   auxvec.h  byteorder.h  elf.h    linkage.h
atomic.h  bitops.h  cache.h      kexec.h  module.h

Hope that helps,

	Arnd <><

Re: [PATCH] 'make headers_install' kbuild target.

[Arjan van de Ven]

> atomic.h 

this one for sure isn't for userspace; simply because at least on x86 it
isn't atomic there (well depending on the phase of the moon) and for
some it can't be done at all.

Re: [PATCH] 'make headers_install' kbuild target.

[Arnd Bergmann]

Am Sunday 23 April 2006 09:09 schrieb Arjan van de Ven:
> > atomic.h
>
> this one for sure isn't for userspace; simply because at least on x86 it
> isn't atomic there (well depending on the phase of the moon) and for
> some it can't be done at all.

Right, my mistake.

	Arnd <><

Re: [PATCH] 'make headers_install' kbuild target.

[Joshua Hudson]

I've got an old script lying around that does it the reverse way.
It is ran in /usr/include, and finds all #include <linux/* and
#include <asm/* and copies the files over. After doing so,
it repeats the pass until there are no more to copy. Interested?

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 03:17 +0100, David Woodhouse wrote:
> I expect the kbuild folks to reimplement what I've done in the Makefile,
> but it works well enough to get us started. The text file listing the
> header files will probably want to change -- maybe we'll have a file in
> each directory listing the exportable files in that directory, or maybe
> we'll put a marker in the public files which we can grep for. I don't
> care much. 

And lo... within 10 hours a patch landed in my mailbox which did it all
properly. Now we have Kbuild files in each directory under include/
which list $(header-y) files for copying and $(unifdef-y) files which
need unifdef run on them. Thanks Arnd :)

This is in the git tree at
git://git.infradead.org/~dwmw2/headers-2.6.git

I'm going to stick that in a public directory and give group access to
it, and give out accounts to anyone who wants to assist. Although that
probably isn't necessary -- the individual header file cleanups can
probably go upstream directly rather than into this tree.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Sat, 2006-04-22 at 03:17 +0100, David Woodhouse wrote:
> Attached is the current patch from mainline to my working tree at
> git://git.infradead.org/~dwmw2/headers-2.6.git -- visible in gitweb at
> http://git.infradead.org/?p=users/dwmw2/headers-2.6.git;a=summary
> 
> It adds a 'make headers_install' target to the kernel makefiles, which
> exports a subset of the headers and runs 'unifdef' on them to clean
> them up for installation in /usr/include. 

I've now added a 'make headers_check' target which goes through the
resulting headers and checks that they are a closed set -- they don't
attempt to include any kernel header which isn't selected for export.
I've also committed enough header cleanups to ensure that the checks
pass, at least for ARCH=powerpc. 

In time, I'd like to see additional checks added, such as checking for
namespace pollution and perhaps attempting to compile each header
standalone.

Now that we can see the mess we have when we export it, there should be
plenty of fun pickings for kernel janitor-types. There's no particular
reason why those cleanups should come through my tree, although I'm
happy enough to collect them and to give accounts on git.infradead.org
to anyone who's going to make a habit of sending them. 

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Rob Landley]

On Friday 21 April 2006 10:17 pm, David Woodhouse wrote:
> Attached is the current patch from mainline to my working tree at
> git://git.infradead.org/~dwmw2/headers-2.6.git -- visible in gitweb at
> http://git.infradead.org/?p=users/dwmw2/headers-2.6.git;a=summary
>
> It adds a 'make headers_install' target to the kernel makefiles, which
> exports a subset of the headers and runs 'unifdef' on them to clean them
> up for installation in /usr/include.

I'd like to test this out.  Is there an easy way for a non-git-user to get a 
patch against a known -linus release via the above web interface?

Rob
-- 
Never bet against the cheap plastic solution.

Re: [PATCH] 'make headers_install' kbuild target.

[Rob Landley]

On Saturday 22 April 2006 10:50 am, Adrian Bunk wrote:
> > > These are two doable approaches with a new kabi/ that avoid needless
> > > breaking of userspace.
> >
> > First off:
> > There are many other users that poke direct in the kernel source also.
>
> Kernel space users?
> User space users?
>
> Can you give an example of what you are thinking of?

Busybox's losetup code wants linux/loop.c because nothing else #defines that 
structure, and cut and pasting it into a local header file is incredibly 
painful because portions of that structure vary by architecture, namely 
__kernel_dev_t has different sizes on alpha, arm, x86... meaning we have to 
#include linux/posix_types.h unless we want to hard wire into our code 
#ifdefs that type for every architecture out there.  This is _bypassing_ the 
fact that we already #include linux/version.h and check if we're building on 
2.6, and if so just use the 64 bit structure instead to avoid the whole 
_old_kernel_dev_t rename entirely.

We spent _two years_ trying to figure out a clean way to do all that.  There 
isn't one.

And no, util-linux isn't doing it cleanly either.  They just hide the ugliness 
by nesting #include files.  mount/loop.h #includes my_dev_t.h which #includes 
<linux/posix_types.h> and <linux/version.h> just like we do, because THERE IS 
NO ALTERNATIVE.

I've been using Mazur's cleaned up headers (still am, it's stuck on 2.6.12 but 
that still works until something better comes along).  I've been tracking 
various attempts to come up with a successor.

The gentoo people have been collecting patches to clean up the headers:
http://www.gentoo.org/cgi-bin/viewcvs.cgi/src/patchsets/gentoo-headers/?root=gentoo
Plus usable headers and a sanitizing script:
http://packages.gentoo.org/search/?sstring=linux-headers
http://packages.gentoo.org/search/?sstring=genkernel

The cross-linux-from-scratch people have been working on their own header 
sanitizing script:
http://headers.cross-lfs.org
http://cross-lfs.org/view/svn/ppc/final-system/linux-headers.html
http://ninja.linux-phreak.biz/pipermail/clfs-dev/2006-April/000007.html
http://ninja.linux-phreak.biz/pipermail/clfs-dev/2006-April/000019.html
http://ninja.linux-phreak.biz/pipermail/clfs-dev/2006-April/000027.html

Fedora recently migrated from a linux-kernel-headers package that smells a bit 
like Mazur's to the glibc-kernheaders package.  Debian has its own patches 
for this...  There have been LOTS of attempts to clean up all the #ifdef 
KERNEL stuff 

I'm am _thrilled_ that somebody's trying to create a path into the kernel for 
all these diverse efforts, because this is important to a lot of people and 
we can't wait two years for a big flag day kabi effort to possibly become 
useful someday.

The beauty of the "make headers_install" thing is I don't have to wait for it 
to be finished before I can use it.  It's no worse than the current 
situation, and it's something I can collect and maintain existing patches 
against, something I can _submit_ patches against.

P.S.  I spent yesterday dealing with this stuff.  Why?  I want to build a 
cross-compiler for arm, using gcc 4.1.0.  But gcc doesn't just require a 
cross version of binutils, it also requires headers for the target platform 
(because now stack unwinding is built even if you disable c++ support).  
These target headers come from your cross-compiled C library.  You need a 
cross compiler to build that C library.  And you need the library to build 
the cross compiler.

Luckily, uClibc has a "make headers" that can use the native compiler.  But 
the uClibc build requires the correct kernel headers preprepared for the 
target platform you're going to build for when you do that.  And so, I 
dredged up Mazur's 2.6.12 headers again.  It's that or use the raw ones from 
linux-2.6.16...


> > Secondly and more importantly:
> > Introducing kabi/ you will have a half solution where several users will
> > have to find their stuff in two places for a longer period.
> > kabi/ does not allow you to do it incrementally - it requires you to
> > move everything over from a start.
> > You may argue that you can just move over a little bit mroe than needed
> > but then we ruin the incremental approach.
>
> For kernel space, you can do it incrementally, since the whole kabi/
> stuff should be transparent for in-kernel uses.

kabi does not exist yet, and it will be useless to me until it's finished.  
Wake me when you've teleported from point A to point B with no road in 
between.  In the meantime, I'm happy to test the incremental cleanup and 
report any problems I have with it, possibly with patches if I manage to fix 
it myself.

Make headers_install interests me, as a consumer of said headers.  Today.  
Right now.  The kabi thing does not interest me, does not serve my needs, and 
will not get any effort from me to advance it.

> For user space, you need one switch.
> But this switch goes from the current mess with several independent
> user space header implementations to one official implementation.

We have an official implementation, the existing kernel headers.  Its's just 
broken.  The "make headers_install" approach sounds to me like a marvelous 
way to get consistently cleaned up versions of those kernel headers.  (I 
didn't say perfectly, I said consistently.  I can submit patches against 
consistently, and I know where to submit them _to_.)

Rob
-- 
Never bet against the cheap plastic solution.

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Fri, 2006-04-28 at 14:13 -0400, Rob Landley wrote:
> I'd like to test this out.  Is there an easy way for a non-git-user to
> get a patch against a known -linus release via the above web
> interface?

In the general case, not really. Just install git.

There's a 'combined' tree containing both hdrcleanup-2.6 and
hdrinstall-2.6 tree at git://git.infradead.org/headers-2.6

I have just taken a diff of that and put it at
http://david.woodhou.se/headers-diff.patch but in general that won't be
up to date.

It should also be in the next -mm tree too.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[David Woodhouse]

On Fri, 2006-04-28 at 14:15 -0400, Rob Landley wrote:
> Fedora recently migrated from a linux-kernel-headers package that smells a bit 
> like Mazur's to the glibc-kernheaders package.

Fedora used to be on an ancient version of the headers, forked and
manually sanitised from 2.4 some time ago and manually (but
inconsistently) updated to date with new syscalls &c as and when bugs
got filed against the package.

As of two days ago, Fedora is using the result of 'make headers_install'
instead. Speaking as maintainer of Fedora's glibc-kernheaders, I think
it's a massive improvement, 

Other distributions look like they should be able to change too -- the
whole point in approaching them before implementing this was to confirm
that they'd be happy with it. I don't know _when_ that'll happen though.
Obviously it makes sense for them to wait while I use Fedora rawhide as
a test bed.

-- 
dwmw2

Re: [PATCH] 'make headers_install' kbuild target.

[Rob Landley]

On Friday 28 April 2006 2:27 pm, David Woodhouse wrote:
> On Fri, 2006-04-28 at 14:15 -0400, Rob Landley wrote:
> > Fedora recently migrated from a linux-kernel-headers package that smells
> > a bit like Mazur's to the glibc-kernheaders package.
>
> Fedora used to be on an ancient version of the headers, forked and
> manually sanitised from 2.4 some time ago and manually (but
> inconsistently) updated to date with new syscalls &c as and when bugs
> got filed against the package.
>
> As of two days ago, Fedora is using the result of 'make headers_install'
> instead. Speaking as maintainer of Fedora's glibc-kernheaders, I think
> it's a massive improvement,
>
> Other distributions look like they should be able to change too -- the
> whole point in approaching them before implementing this was to confirm
> that they'd be happy with it. I don't know _when_ that'll happen though.
> Obviously it makes sense for them to wait while I use Fedora rawhide as
> a test bed.

I'm not waiting. :)

I'm making a cross-compiler for ARM (by hand, figuring out how to do it), and 
I have a whole weekend to thump on it.  I want to build a kernel, uClibc, and 
busybox, and get them to boot under qemu-system-arm.  That will be the "ok, 
declare victory and document what I just did" moment.

I'll let you know what breaks.  (I have Mazur's old 2.6.12 here for 
comparison, so I may even have patches.  You never know... :)

Rob
-- 
Never bet against the cheap plastic solution.