2.6.18 Headers - Long

2.6.18 Headers - Long

[Jim Gifford]

I was really glad to see that something was going to be done with the 
headers, but I don't think it's enough. I'm going to share my concerns 
and hopefully we can all figure what is the right thing to do.

First off, my research in this has been going on since LLH announced 
that it was not going to produce any more headers. I started a project 
to sanitize the headers myself. http://headers.cross-lfs.org.

I will only document one issue, but there are several more like this in 
the kernel.

I'm going to use the MIPS architecture in my example, along with the 
file page.h.

With the 2.6.18 headers, the file that gets created looks like this. If 
you notice on lines 17, 20, 23, 26, and 34 they all use CONFIG_{...}, 
these variables are called from linux/autoconf.h. Yes, the simple fix 
would be to include this file.

   1.
      /*
   2.
       * This file is subject to the terms and conditions of the GNU
      General Public
   3.
       * License.  See the file "COPYING" in the main directory of this
      archive
   4.
       * for more details.
   5.
       *
   6.
       * Copyright (C) 1994 - 1999, 2000, 03 Ralf Baechle
   7.
       * Copyright (C) 1999, 2000 Silicon Graphics, Inc.
   8.
       */
   9.
      #ifndef _ASM_PAGE_H
  10.
      #define _ASM_PAGE_H
  11.
       
  12.
       
  13.
       
  14.
      /*
  15.
       * PAGE_SHIFT determines the page size
  16.
       */
  17.
      #ifdef CONFIG_PAGE_SIZE_4KB
  18.
      #define PAGE_SHIFT      12
  19.
      #endif
  20.
      #ifdef CONFIG_PAGE_SIZE_8KB
  21.
      #define PAGE_SHIFT      13
  22.
      #endif
  23.
      #ifdef CONFIG_PAGE_SIZE_16KB
  24.
      #define PAGE_SHIFT      14
  25.
      #endif
  26.
      #ifdef CONFIG_PAGE_SIZE_64KB
  27.
      #define PAGE_SHIFT      16
  28.
      #endif
  29.
      #define PAGE_SIZE       (1UL << PAGE_SHIFT)
  30.
      #define PAGE_MASK       (~((1 << PAGE_SHIFT) - 1))
  31.
       
  32.
       
  33.
       
  34.
      #ifdef CONFIG_LIMITED_DMA
  35.
      #define WANT_PAGE_VIRTUAL
  36.
      #endif
  37.
       
  38.
      #include <asm-generic/memory_model.h>
  39.
      #include <asm-generic/page.h>
  40.
       
  41.
      #endif /* _ASM_PAGE_H */

Here's the header I produce with my sanitize script, I use a glibc call 
to get the information, which would be more appropriate for the user 
space. This is very similar to what LLH provided. Even my example is not 
prefect due to line 14.

   1.
      #define _ASM_PAGE_H
   2.
       
   3.
      #include <unistd.h>
   4.
       
   5.
      #define PAGE_SIZE       (getpagesize())
   6.
      static __inline__ int getpageshift()
   7.
      {
   8.
          int pagesize = getpagesize();
   9.
          return (__builtin_clz(pagesize) ^ 31);
  10.
      }
  11.
      #define PAGE_SHIFT      (getpageshift())
  12.
      #define PAGE_MASK       (~(PAGE_SIZE-1))
  13.
       
  14.
      #ifdef CONFIG_LIMITED_DMA
  15.
      #define WANT_PAGE_VIRTUAL
  16.
      #endif
  17.
       
  18.
      #endif /* !(_ASM_PAGE_H) */


So even with the new headers_install, the headers still need to be 
sanitized to overcome the missing variables from linux/autoconf.h.

Just wanted to bring this up to everyone's attention and look forward to 
helping get things resolved.

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Tue, 2006-07-11 at 17:35 -0700, Jim Gifford wrote:
> I will only document one issue, but there are several more like this
> in the kernel.

Please elaborate, preferably in 'diff -u' form as below...

> I'm going to use the MIPS architecture in my example, along with the 
> file page.h. 

[PATCH] Reduce user-visible noise in asm-mips/page.h

Since PAGE_SIZE is variable according to configuration options, don't
expose it to userspace. Userspace should be using sysconf(_SC_PAGE_SIZE)
instead. Move some other noise inside __KERNEL__ too while we're at it.

Signed-off-by: David Woodhouse <dwmw2@infradead.org>

diff --git a/include/asm-mips/page.h b/include/asm-mips/page.h
index 6ed1151..ee2ef88 100644
--- a/include/asm-mips/page.h
+++ b/include/asm-mips/page.h
@@ -14,8 +14,6 @@ #ifdef __KERNEL__
 
 #include <spaces.h>
 
-#endif
-
 /*
  * PAGE_SHIFT determines the page size
  */
@@ -35,7 +33,6 @@ #define PAGE_SIZE	(1UL << PAGE_SHIFT)
 #define PAGE_MASK       (~((1 << PAGE_SHIFT) - 1))
 
 
-#ifdef __KERNEL__
 #ifndef __ASSEMBLY__
 
 extern void clear_page(void * page);
@@ -168,7 +165,6 @@ #define VM_DATA_DEFAULT_FLAGS	(VM_READ |
 #define UNCAC_ADDR(addr)	((addr) - PAGE_OFFSET + UNCAC_BASE)
 #define CAC_ADDR(addr)		((addr) - UNCAC_BASE + PAGE_OFFSET)
 
-#endif /* defined (__KERNEL__) */
 
 #ifdef CONFIG_LIMITED_DMA
 #define WANT_PAGE_VIRTUAL
@@ -177,4 +173,6 @@ #endif
 #include <asm-generic/memory_model.h>
 #include <asm-generic/page.h>
 
+#endif /* defined (__KERNEL__) */
+
 #endif /* _ASM_PAGE_H */


-- 
dwmw2

Re: 2.6.18 Headers - Long

[Jim Gifford]

David,
    There are several of them like that. I'll just list the files, and 
the issues

    asm-ia64/page.h - Config variables
    asm-sparc/page.h - Config variables
    asm-sparc64/page.h - Config variables
    asm-powerpc/page.h - Config variables

I'll go through the rest over the next few days.

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Thu, 2006-07-13 at 19:18 -0700, Jim Gifford wrote:
>     asm-ia64/page.h - Config variables
>     asm-sparc/page.h - Config variables
>    asm-sparc64/page.h - Config variables

Yeah, just as with MIPS, the bits which are affected by CONFIG variables
just shouldn't be outside the __KERNEL__ ifdef at all -- they shouldn't
be seen.

>     asm-powerpc/page.h - Config variables 

Er, asm-powerpc/page.h is empty after unifdef, which is entirely
correect. In fact asm/page.h is one of the files which should probably
be removed from user visibility entirely. There's nothing there which
userspace should see, in general.

-- 
dwmw2

Re: 2.6.18 Headers - Long

[Jim Gifford]

Unfortunately, a lot programs out there are using page.h, and a lot of 
people are using that in their programs. The 2 program I know for sure 
that use page.h are glibc and util-linux.

Re: 2.6.18 Headers - Long

[Arjan van de Ven]

On Fri, 2006-07-14 at 12:28 -0700, Jim Gifford wrote:
> Unfortunately, a lot programs out there are using page.h, and a lot of 
> people are using that in their programs. 

.... and they're all broken....
(hint: page size on many platforms is a kernel compile option, eg not a
constant)

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Fri, 2006-07-14 at 12:28 -0700, Jim Gifford wrote:
> Unfortunately, a lot programs out there are using page.h, and a lot of 
> people are using that in their programs. The 2 program I know for sure 
> that use page.h are glibc and util-linux. 

I don't believe glibc does. It certainly copes fine with an empty page.h
on PowerPC.

-- 
dwmw2

Re: 2.6.18 Headers - Long

[Erik Andersen]

On Fri Jul 14, 2006 at 12:28:34PM -0700, Jim Gifford wrote:
> Unfortunately, a lot programs out there are using page.h, and a lot of 
> people are using that in their programs. The 2 program I know for sure 
> that use page.h are glibc and util-linux.

util-linux should be using getpagesize() or sysconf(_SC_PAGESIZE)
from the C library.  And libc should be getting the page size
within ldso (or _start for static apps) by parsing the AT_PAGESZ
entry from the ELF auxiliary vector.  Should that be 0 (i.e.
because the kernel is horribly broken) then and only then should
libc fall back to guessing, i.e. a page size of 4k.  A quick check
shows that glibc, uClibc, and klibc all get this right.  But you
are right, that util-linux looks to have this wrong in a few
places.  I expect fixing the kernel headers will get that fixed
in short order.  :-)

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Fri, 2006-07-14 at 21:39 +0200, Arjan van de Ven wrote:
> On Fri, 2006-07-14 at 12:28 -0700, Jim Gifford wrote:
> > Unfortunately, a lot programs out there are using page.h, and a lot of 
> > people are using that in their programs. 
> 
> .... and they're all broken.... 

More to the point, now we're doing this from upstream we can be
_consistent_ about telling them to sod off and fix their broken crap.

We no longer have to deal with cries of 'but it works on $SOMEDISTRO'
when another distro tries to impose some sanity rather than just
pandering to it.

Kernel headers are _not_ a library of random crap for userspace to use.

-- 
dwmw2

Re: 2.6.18 Headers - Long

[Christoph Hellwig]

On Fri, Jul 14, 2006 at 01:25:48AM +0100, David Woodhouse wrote:
> On Tue, 2006-07-11 at 17:35 -0700, Jim Gifford wrote:
> > I will only document one issue, but there are several more like this
> > in the kernel.
> 
> Please elaborate, preferably in 'diff -u' form as below...
> 
> > I'm going to use the MIPS architecture in my example, along with the 
> > file page.h. 
> 
> [PATCH] Reduce user-visible noise in asm-mips/page.h

The right fix is to not expose page.h at all on any architecture.  There's
nothing userspace should care about in this file.

Re: 2.6.18 Headers - Long

[David Miller]

From: David Woodhouse <dwmw2@infradead.org>
Date: Fri, 14 Jul 2006 21:16:42 +0100

> More to the point, now we're doing this from upstream we can be
> _consistent_ about telling them to sod off and fix their broken crap.
> 
> We no longer have to deal with cries of 'but it works on $SOMEDISTRO'
> when another distro tries to impose some sanity rather than just
> pandering to it.
> 
> Kernel headers are _not_ a library of random crap for userspace to use.

I totally agree.  When I saw that some dists make asm/page.h define
PAGE_SIZE to "getpagesize()" for userspace, I nearly crapped myself.

That is insane, and encourages the problem to persist instead of
encouraging the right fix.

Re: 2.6.18 Headers - Long

[Jim Gifford]

 When I wrote my script to sanitize, I was really surprised on which 
headers gets utilized and which ones didn't.. I have it down to the bare 
minimums in my script.

As far as glibc goes, from my people who are helping me the alpha 
architecture is the culprit there.

Do we have a list of what headers are "user-space" and which ones should 
not be "user-space"?

Also David W, let me know what I can do to help you out, a lot of people 
on my end want to get this working properly.

Who's maintaining util-linux these days, we probably should get a patch 
to them.

Re: 2.6.18 Headers - Long

[Randy.Dunlap]

On Fri, 14 Jul 2006 13:57:39 -0700 Jim Gifford wrote:

>  When I wrote my script to sanitize, I was really surprised on which 
> headers gets utilized and which ones didn't.. I have it down to the bare 
> minimums in my script.
> 
> As far as glibc goes, from my people who are helping me the alpha 
> architecture is the culprit there.
> 
> Do we have a list of what headers are "user-space" and which ones should 
> not be "user-space"?
> 
> Also David W, let me know what I can do to help you out, a lot of people 
> on my end want to get this working properly.
> 
> Who's maintaining util-linux these days, we probably should get a patch 
> to them.

  Adrian Bunk <bunk@stusta.de>


---
~Randy

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Fri, 2006-07-14 at 13:19 -0700, David Miller wrote:
> I totally agree.  When I saw that some dists make asm/page.h define
> PAGE_SIZE to "getpagesize()" for userspace, I nearly crapped myself.
> 
> That is insane, and encourages the problem to persist instead of
> encouraging the right fix. 

Until a week or two ago, we had no mechanism with which to implement the
'right fix'. Any distribution taking the initiative and reducing
visibility of stuff like that would be inundated with complaints that
"it works OK on other distributions".

We can be stricter now that we have some expectation of _consistency_
across distributions.

-- 
dwmw2

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Fri, 2006-07-14 at 13:57 -0700, Jim Gifford wrote:
> Do we have a list of what headers are "user-space" and which ones should 
> not be "user-space"?

Well, we have the lists in include/*/Kbuild files, of course -- but
that's all. As I've stated before, I've been somewhat liberal with the
exports to far, to match what's currently (ab)used, because I wanted to
concentrate on the _mechanism_, not the policy.

The intention is that we can now start to tighten it up -- I've already
sent the patches which drop asm/atomic.h and asm/io.h, and there are
more which should go. Next on that list (and already commented as such
in include/asm-generic/Kbuild.asm) are asm/page.h and asm/elf.h.

> Also David W, let me know what I can do to help you out, a lot of people 
> on my end want to get this working properly.

Thanks. One thing you can do which would be extremely useful is to
investigate dropping page.h and elf.h, and make sure that stuff like gdb
(and anything using ptrace.h) will still build. Then just look for
anything else which should be removed from view. The git repository at
http://git.kernel.org/git/?p=linux/kernel/git/dwmw2/kernel-headers.git
(git://git.kernel.org/pub/scm/linux/kernel/git/dwmw2/kernel-headers.git)
should help with that task. Provide patches to move stuff within #ifdef
__KERNEL__ or to move it to unexported files.

There are some who think that it would be nice to get rid of __KERNEL__
entirely -- files would be either entirely exported, or not at all. I
don't think we necessarily need to go that far; the export step with
unifdef isn't so bad.

-- 
dwmw2

Re: 2.6.18 Headers - Long

[Albert Cahalan]

David Woodhouse writes:

> Kernel headers are _not_ a library of random crap for userspace to use.

The attraction is that the kernel abstractions are very nice.
Much of the POSIX API sucks ass. The kernel stuff is NOT crap.

Here we have a full-featured set of atomic ops, byte swapping
with readable names and a distinction for pointers, nice macros
for efficient data structure manipulation...

Sure, you'd like all the app developers to write bloated C++
and use sucky POSIX threads stuff, but then you're not the
one who has to write or maintain it.

Don't blame app developers if they go for what is good.
To stop them, provide the goodness in a sane way.
(alternately, make the Linux code suck ass more than POSIX)

Re: 2.6.18 Headers - Long

[Arjan van de Ven]

On Sat, 2006-07-15 at 17:09 -0400, Albert Cahalan wrote:
> David Woodhouse writes:
> 
> > Kernel headers are _not_ a library of random crap for userspace to use.
> 
> The attraction is that the kernel abstractions are very nice.
> Much of the POSIX API sucks ass. The kernel stuff is NOT crap.
> 
> Here we have a full-featured set of atomic ops,

which are not atomic actually in userspace (hint: most apps don't have
CONFIG_SMP set)

Re: 2.6.18 Headers - Long

[Adrian Bunk]

On Sat, Jul 15, 2006 at 05:09:28PM -0400, Albert Cahalan wrote:
> David Woodhouse writes:
> 
> >Kernel headers are _not_ a library of random crap for userspace to use.
> 
> The attraction is that the kernel abstractions are very nice.
> Much of the POSIX API sucks ass. The kernel stuff is NOT crap.
> 
> Here we have a full-featured set of atomic ops, byte swapping
> with readable names and a distinction for pointers, nice macros
> for efficient data structure manipulation...
>...

These are _kernel_ headers.

Sure, applications are abusing the kernel headers as userspace library.

But this is wrong, that's not what they are designed for.
Look at the code MySQL uses for including atomic.h and you understand 
what I'm saying.

If stuff from the headers is generally considered useful for userspace, 
someone should simply start a new project containing the small subset of 
the headers containing such stuff cleaned up for userspace.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Sat, 2006-07-15 at 17:09 -0400, Albert Cahalan wrote:
> David Woodhouse writes:
> > Kernel headers are _not_ a library of random crap for userspace to 
> > use.

> Don't blame app developers if they go for what is good.
> To stop them, provide the goodness in a sane way.
> (alternately, make the Linux code suck ass more than POSIX) 

Kernel headers are _not_ a library of random crap for userspace to use.

There is no justification for asm/atomic.h being installed
in /usr/include. Especially since, as Arjan points out, it doesn't
actually provide atomic operations in many cases anyway.

However, the kernel is released under a licence which allows you to
re-use code from it if you really want. If you want to provide a
'libkernelstuff', the GPL permits you to do that. The kernel's ABI
headers (and lkml) are not the appropriate place for such a project
though. If you want to do that, take it where the C++ and microkernel
people go.

Btw, your mail client omitted the References: and In-Reply-To: headers
which RFC2822 says it SHOULD have included. On a list with as much
traffic as linux-kernel, that's _very_ suboptimal, because you've
detached your message from the thread to which you replied. Please try
to fix or work around that.

-- 
dwmw2

Re: 2.6.18 Headers - Long

[Albert Cahalan]

On 7/15/06, David Woodhouse <dwmw2@infradead.org> wrote:
> On Sat, 2006-07-15 at 17:09 -0400, Albert Cahalan wrote:

> > Don't blame app developers if they go for what is good.
> > To stop them, provide the goodness in a sane way.
> > (alternately, make the Linux code suck ass more than POSIX)
>
> Kernel headers are _not_ a library of random crap for userspace to use.

Says you, and a number of other people around here.
App developers seem to feel differently. Accept reality.

> There is no justification for asm/atomic.h being installed
> in /usr/include. Especially since, as Arjan points out, it doesn't
> actually provide atomic operations in many cases anyway.

It's fixable via #ifdef __KERNEL__ of course.

> However, the kernel is released under a licence which allows you to
> re-use code from it if you really want.

Well, ideally it'd be LGPL, but yes.

> If you want to provide a
> 'libkernelstuff', the GPL permits you to do that. The kernel's ABI
> headers (and lkml) are not the appropriate place for such a project

Perhaps. This is duplication of effort though.

You're the person trying to change the app developers.
If you want to change them, provide an alternative that
doesn't totally suck ass.

> Btw, your mail client omitted the References: and In-Reply-To: headers
> which RFC2822 says it SHOULD have included. On a list with as much
> traffic as linux-kernel, that's _very_ suboptimal, because you've
> detached your message from the thread to which you replied. Please try
> to fix or work around that.

Most clients won't allow adding such headers manually.
I don't actually subscribe to the list. Most clients won't
expire old list traffic, and I certainly can't have lkml pour
unhindered into my inbox. It really doesn't work for me,
so I cut and paste from a list archive. Sorry. Ideas are
welcome of course, but I'm not expecting any reasonable
solution to exist.

Re: 2.6.18 Headers - Long

[Arjan van de Ven]

> Perhaps. This is duplication of effort though.
> 
> You're the person trying to change the app developers.
> If you want to change them, provide an alternative that
> doesn't totally suck ass.

apr provides all this and doesn't suck.

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Sun, 2006-07-16 at 02:18 -0400, Albert Cahalan wrote:
> On 7/15/06, David Woodhouse <dwmw2@infradead.org> wrote:
> > On Sat, 2006-07-15 at 17:09 -0400, Albert Cahalan wrote:
> 
> > > Don't blame app developers if they go for what is good.
> > > To stop them, provide the goodness in a sane way.
> > > (alternately, make the Linux code suck ass more than POSIX)
> >
> > Kernel headers are _not_ a library of random crap for userspace to use.
> 
> Says you, and a number of other people around here.
> App developers seem to feel differently. Accept reality.

I accept that app developers feel differently. I feel sympathy for them.
There, reality accepted -- can we get back to kernel business now?

Kernel headers are _not_ a library of random crap for userspace to use.

> > There is no justification for asm/atomic.h being installed
> > in /usr/include. Especially since, as Arjan points out, it doesn't
> > actually provide atomic operations in many cases anyway.
> 
> It's fixable via #ifdef __KERNEL__ of course.

No, it's fixed by commit e035cc35e54230eb704ee7feccf476ed2fb2ae29
already. Running 'make headers_install' in the kernel tree no longer
exports a copy of asm/atomic.h

The existence of #ifdef __KERNEL__ in a file not listed for export to
userspace should probably now be considered a bug, and eradicated.

> > If you want to provide a 'libkernelstuff', the GPL permits you to do
> > that. The kernel's ABI headers (and lkml) are not the appropriate 
> > place for such a project
> 
> Perhaps. This is duplication of effort though.

Not really. The kernel developers make no effort to ensure that this
stuff works in userspace -- as evidenced by the fact that it _doesn't_
actually work in userspace. If someone wants to take this kernel code
and make it work in userspace, that's effort which is entirely unique.

If someone wants to just whine that they _want_ it to be available (and
reliable) in userspace, then we don't care. It isn't viable in
userspace, it's _never_ been viable in userspace, and it's _your_ turn
to accept reality.

> You're the person trying to change the app developers.
> If you want to change them, provide an alternative that
> doesn't totally suck ass.

We're only talking about that subset of app developers who are still
abusing kernel headers despite the fact that they've repeatedly been
told not to and despite that fact that it's often actually broken to do
so (in the case of atomic.h, which is one of the more common abuses).

We've been trying to educate those morons for years, and they seem
particularly resilient to it. A consistent (and minimal) set of kernel
headers across distributions, selected and sanitised by Makefiles within
the kernel itself, may well do the trick though.

I don't see that we need to provide an alternative. These people will
find some other way to shoot themselves in the foot whatever we do.

> > Btw, your mail client omitted the References: and In-Reply-To: headers
> > which RFC2822 says it SHOULD have included. On a list with as much
> > traffic as linux-kernel, that's _very_ suboptimal, because you've
> > detached your message from the thread to which you replied. Please try
> > to fix or work around that.
> 
> Most clients won't allow adding such headers manually.
> I don't actually subscribe to the list. Most clients won't
> expire old list traffic, and I certainly can't have lkml pour
> unhindered into my inbox. It really doesn't work for me,
> so I cut and paste from a list archive. Sorry. Ideas are
> welcome of course, but I'm not expecting any reasonable
> solution to exist.

Mailing list archives ought to have mailto: links with &In-Reply-To= set
up properly so that you get a correct In-Reply-To: header when you click
on the link.

Personally, I just have lists pour 'unhindered' into their own folder,
so that I can behave considerately and keep threads together even when I
don't actually read the list in question very often. It's easy enough to
expire old mail. Actually I just archive it with a simple script
(http://david.woodhou.se/archivemail.sh) but it's trivial to just delete
it instead, if that's what you want.

-- 
dwmw2

Re: 2.6.18 Headers - Long

[Jakub Jelinek]

On Sat, Jul 15, 2006 at 05:09:28PM -0400, Albert Cahalan wrote:
> Here we have a full-featured set of atomic ops, byte swapping
> with readable names and a distinction for pointers, nice macros
> for efficient data structure manipulation...

And userland has GCC __sync_* atomic builtins, <byteswap.h>, etc.
That stuff works in userspace, unlike most of the stuff provided by kernel
headers.

	Jakub

Re: 2.6.18 Headers - Long

[Kyle Moffett]

On Jul 15, 2006, at 17:09:28, Albert Cahalan wrote:
> Here we have a full-featured set of atomic ops,

You realize that on a couple architectures it's fundamentally  
impossible to get atomic ops completely in userspace, right?  Some of  
the mechanisms that various archs use (IIRC including in one or two  
cases just completely disabling interrupts) don't work anywhere but  
the kernel.

> byte swapping with readable names and a distinction for pointers,  
> nice macros for efficient data structure manipulation...

Both of which may be easily cut and pasted into your GPL programs  
with little or no effort (Hint: I do this all the time).  Those are  
so stable you don't even have to maintain it!  IMHO, what you really  
want, though, is for GCC to export a library of ASM intrinsics (like  
memory barriers, atomic ops, etc), that are available on your current  
architecture.  If there is no __gcc_atomic_inc then it wouldn't  
#define it and you can just go back to pthread_mutex_lock/unlock for  
protecting an atomic variable.  Such a library layer certainly  
doesn't belong in the kernel, although if GCC got such a library  
right the kernel might start to use it (although only the most recent  
GCC would support it so it wouldn't be very useful).

> Sure, you'd like all the app developers to [...] use sucky POSIX  
> threads stuff

There is *NO* portable way to get atomic operations or locking in  
userspace except through libpthread.  Atomic operations in the kernel  
are sometimes _not_ atomic in userspace and the only decent way to do  
userspace locking is to call into the kernel on contention (or you  
waste the rest of your timeslice spinning).  POSIX thread operations  
are extremely efficient under Linux, but the kernel shouldn't export  
the varying underlying operations.

Cheers,
Kyle Moffett

Re: 2.6.18 Headers - Long

[Russell King]

On Sun, Jul 16, 2006 at 08:34:53AM -0400, Kyle Moffett wrote:
> There is *NO* portable way to get atomic operations or locking in  
> userspace except through libpthread.

And as one of the maintainers of an architecture where this is true
(neither atomic.h nor bitops.h will work in userspace atomically) I
whole heartedly agree that the kernel's atomic operations must not
be exported to userspace via header files.

Anyone who thinks they will or should do is just kidding themselves.

The final point to add to this is that folk might be lucky, and the
kernel's atomic ops might _appear_ to work without raising any faults,
but that does _not_ mean that they are necessarily atomic in their
operation.  For example:

	unsigned long *a;

	a[bit >> 5] |= 1 << (bit & 31);

and

	set_bit(bit, &a);

are 100% equivalent in userspace if you use the kernel's asm-arm/bitops.h.
Neither will be atomic.  More importantly, _neither_ could be made to be
atomic.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: 2.6.18 Headers - Long

[Nix]

On 15 Jul 2006, Randy Dunlap prattled cheerily:
> On Fri, 14 Jul 2006 13:57:39 -0700 Jim Gifford wrote:
> 
>> Who's maintaining util-linux these days, we probably should get a patch 
>> to them.
> 
>   Adrian Bunk <bunk@stusta.de>

... and I sent him a patch months ago, and it was applied: the cramfs
tools in util-linux no longer use PAGE_SIZE, and hasn't since
2.13-pre5. (mkswap still references it, but that usage is *optional* and
doesn't happen if PAGE_SIZE isn't defined.)

Re: 2.6.18 Headers - Long

[Albert Cahalan]

On 7/16/06, Kyle Moffett <mrmacman_g4@mac.com> wrote:
> On Jul 15, 2006, at 17:09:28, Albert Cahalan wrote:

> You realize that on a couple architectures it's fundamentally
> impossible to get atomic ops completely in userspace, right?

Sure. Those architectures don't need to drag down the rest.
Plenty of headers are only exported for some architectures.

(Well actually, such architectures could just give apps a
writable flag to disable the scheduler -- this is acceptable
for the embedded things these architectures are used for.
I've seen it done for user-space spinlocks. It works great.)

It's not as if the app developers would care to support
those architectures anyway. They don't even support all
the non-defective ones: I use the second or third most
popular architecture (ppc), and the app developers have
made it very clear that they don't give a damn. Something
else will get you: char being unsigned by default, wrong
endianness, stack growing the HP way, etc.

Re: 2.6.18 Headers - Long

[Russell King]

On Sun, Jul 16, 2006 at 02:38:45PM -0400, Albert Cahalan wrote:
> On 7/16/06, Kyle Moffett <mrmacman_g4@mac.com> wrote:
> >On Jul 15, 2006, at 17:09:28, Albert Cahalan wrote:
> 
> >You realize that on a couple architectures it's fundamentally
> >impossible to get atomic ops completely in userspace, right?
> 
> Sure. Those architectures don't need to drag down the rest.
> Plenty of headers are only exported for some architectures.

Wrong perspective.  The problem is that they may _appear_ to work as
described, but not actually work in the intended way.  That's a bug,
and it's a _hard_ bug to locate.

> (Well actually, such architectures could just give apps a
> writable flag to disable the scheduler -- this is acceptable
> for the embedded things these architectures are used for.

Cloud cuckoo land.  In the embedded world, there are folk still want
to have the security aspects as well.  Moreover, there are far more
folk who do _NOT_ want to have things like "disable the scheduler" -
think the realtime folk.

> It's not as if the app developers would care to support
> those architectures anyway.

That's actually more of a reason to take away the toys they shouldn't
be playing with in the first place - the only reason these (wrong)
interfaces are being used is because they're easily accessible.  Had
they _never_ been visible to userspace, no one would've attempted to
use them in the first place.

So, take away the "easily accessible" bit on the common architectures
and they'll find different ways to meet their needs.  Hopefully, that
will be some portable solution instead of one where racy code goes by
unnoticed.  If it isn't, at least there will be build errors to tell
you something needs to be fixed (probably from the assembler!)

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: 2.6.18 Headers - Long

[Albert Cahalan]

On 7/16/06, Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> On Sun, Jul 16, 2006 at 02:38:45PM -0400, Albert Cahalan wrote:
> > On 7/16/06, Kyle Moffett <mrmacman_g4@mac.com> wrote:
> > >On Jul 15, 2006, at 17:09:28, Albert Cahalan wrote:
> >
> > >You realize that on a couple architectures it's fundamentally
> > >impossible to get atomic ops completely in userspace, right?
> >
> > Sure. Those architectures don't need to drag down the rest.
> > Plenty of headers are only exported for some architectures.
>
> Wrong perspective.  The problem is that they may _appear_ to work as
> described, but not actually work in the intended way.  That's a bug,
> and it's a _hard_ bug to locate.

Again:

Plenty of headers are only exported for some architectures.

In other words, for all architectures where things work.

> > (Well actually, such architectures could just give apps a
> > writable flag to disable the scheduler -- this is acceptable
> > for the embedded things these architectures are used for.
>
> Cloud cuckoo land.  In the embedded world, there are folk still want
> to have the security aspects as well.  Moreover, there are far more
> folk who do _NOT_ want to have things like "disable the scheduler" -
> think the realtime folk.

Now you are really wrong. :-)

The Linux system I saw using this hack was carefully tuned
for hard real-time performance. Latency was a few dozen
microseconds. Disabling the scheduler essentially put you
at the highest SCHED_FIFO; aside from that there was
working priority inheritance for both kernel and user code.
The real-time folk love this kind of thing.

Anyway, it's fine to just not let ARM export the header.

> > It's not as if the app developers would care to support
> > those architectures anyway.
>
> That's actually more of a reason to take away the toys they shouldn't
> be playing with in the first place - the only reason these (wrong)
> interfaces are being used is because they're easily accessible.

No. The normal interfaces are dreadful.

App developers will just cut-and-paste the i386 kernel
code if you take away the header files. They do that
often enough already. So all you succeed in doing is
eliminating any hope of portability to ppc and similar.
This is not an improvement.

Re: 2.6.18 Headers - Long

[Russell King]

On Sun, Jul 16, 2006 at 03:22:01PM -0400, Albert Cahalan wrote:
> On 7/16/06, Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> >On Sun, Jul 16, 2006 at 02:38:45PM -0400, Albert Cahalan wrote:
> >> On 7/16/06, Kyle Moffett <mrmacman_g4@mac.com> wrote:
> >> >On Jul 15, 2006, at 17:09:28, Albert Cahalan wrote:
> >>
> >> >You realize that on a couple architectures it's fundamentally
> >> >impossible to get atomic ops completely in userspace, right?
> >>
> >> Sure. Those architectures don't need to drag down the rest.
> >> Plenty of headers are only exported for some architectures.
> >
> >Wrong perspective.  The problem is that they may _appear_ to work as
> >described, but not actually work in the intended way.  That's a bug,
> >and it's a _hard_ bug to locate.
> 
> Again:
> 
> Plenty of headers are only exported for some architectures.

So?  I don't see the relevance of this statement.

> In other words, for all architectures where things work.

And this doesn't make sense (it doesn't contain a verb for a start.)

> >Cloud cuckoo land.  In the embedded world, there are folk still want
> >to have the security aspects as well.  Moreover, there are far more
> >folk who do _NOT_ want to have things like "disable the scheduler" -
> >think the realtime folk.
> 
> Now you are really wrong. :-)

ARM is heavily embedded, and I've never ever had any requests concerning
"can we disable the scheduler from userspace" from anyone...

> >> It's not as if the app developers would care to support
> >> those architectures anyway.
> >
> >That's actually more of a reason to take away the toys they shouldn't
> >be playing with in the first place - the only reason these (wrong)
> >interfaces are being used is because they're easily accessible.
> 
> No. The normal interfaces are dreadful.
> 
> App developers will just cut-and-paste the i386 kernel
> code if you take away the header files. They do that
> often enough already.

And by that statement you just agreed with the part of my mail that you
conveniently cut.

> So all you succeed in doing is eliminating any hope of portability
> to ppc and similar.  This is not an improvement.

I tend to disagree.  In reality folk work towards removing such
dependencies.  We saw it with the old minix tools - they're now fixed.
We've seen it with others, which also eventually got fixed.

The general trend has been to be to fix up these problems as they're
found.

Yes, there are some subborn userspace developers out there, but they'll
continue to be stubborn all the time that their way works.

And finally, bear in mind that we have almost 175 ARM platform designs
registered so far this year, most of them embedded.  So it's a _very_
active space.  Couple that with the lack of folk reporting the problem
you're alluding to... despite ARM atomic ops being protected by
__KERNEL__ and use of ARM kernel bitops in userspace would cause link
errors.

My experience has been obviously different from yours, and therefore I
hold different views.  I don't see the problem.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: 2.6.18 Headers - Long

[David Miller]

From: Kyle Moffett <mrmacman_g4@mac.com>
Date: Sun, 16 Jul 2006 08:34:53 -0400

> Both of which may be easily cut and pasted into your GPL programs  
> with little or no effort (Hint: I do this all the time).  Those are  
> so stable you don't even have to maintain it!  IMHO, what you really  
> want, though, is for GCC to export a library of ASM intrinsics (like  
> memory barriers, atomic ops, etc), that are available on your current  
> architecture.  If there is no __gcc_atomic_inc then it wouldn't  
> #define it and you can just go back to pthread_mutex_lock/unlock for  
> protecting an atomic variable.  Such a library layer certainly  
> doesn't belong in the kernel, although if GCC got such a library  
> right the kernel might start to use it (although only the most recent  
> GCC would support it so it wouldn't be very useful).

I agree with your assertions that the stuff in asm/atomic.h should be
steered away from, since they generally are not expected to work in
userspace.  In fact, if you try to use the 32-bit sparc or parisc
ones, it simply won't link because the implementations are external
and use a hash table spinlock scheme which is in the kernel image.

However, the pthread based locking is bad if you want to recover
from arbitrary signals correctly.  I ran into this problem while
trying to get Linux/Sparc Mono working well.

When people want atomic cmpxchg or atomic increment/decrement, they
want a true atomic.  This means either finish the whole thing, or
leave no trace of the atomic having started in the first place.
Spinlock based atomic implementations, which is what a lot of
platforms lacking true atomic use, absolutely cannot reasonably
provide this semantic.

Re: 2.6.18 Headers - Long

[Arjan van de Ven]

On Sun, 2006-07-16 at 15:22 -0400, Albert Cahalan wrote:
> On 7/16/06, Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> > On Sun, Jul 16, 2006 at 02:38:45PM -0400, Albert Cahalan wrote:
> > > On 7/16/06, Kyle Moffett <mrmacman_g4@mac.com> wrote:
> > > >On Jul 15, 2006, at 17:09:28, Albert Cahalan wrote:
> > >
> > > >You realize that on a couple architectures it's fundamentally
> > > >impossible to get atomic ops completely in userspace, right?
> > >
> > > Sure. Those architectures don't need to drag down the rest.
> > > Plenty of headers are only exported for some architectures.
> >
> > Wrong perspective.  The problem is that they may _appear_ to work as
> > described, but not actually work in the intended way.  That's a bug,
> > and it's a _hard_ bug to locate.
> 
> Again:
> 
> Plenty of headers are only exported for some architectures.


and guess what... atomic.h does not work on i386, at least it doesn't
provide atomic access in userspace!

Re: 2.6.18 Headers - Long

[Ralf Baechle]

On Sat, Jul 15, 2006 at 11:19:06PM +0200, Arjan van de Ven wrote:

> > The attraction is that the kernel abstractions are very nice.
> > Much of the POSIX API sucks ass. The kernel stuff is NOT crap.
> > 
> > Here we have a full-featured set of atomic ops,
> 
> which are not atomic actually in userspace (hint: most apps don't have
> CONFIG_SMP set)

Atomic ops for some architectures / configuration won't even work at al
in userspace because they're implemented by disabling interrupts.

  Ralf

Re: 2.6.18 Headers - Long

[David Woodhouse]

On Fri, 2006-07-14 at 01:25 +0100, David Woodhouse wrote:
> On Tue, 2006-07-11 at 17:35 -0700, Jim Gifford wrote:
> > I will only document one issue, but there are several more like this
> > in the kernel.
> 
> Please elaborate, preferably in 'diff -u' form as below...

Got any more, Jim?

> > I'm going to use the MIPS architecture in my example, along with the 
> > file page.h. 

Ralf, you haven't applied this yet AFAICT...

> [PATCH] Reduce user-visible noise in asm-mips/page.h
> 
> Since PAGE_SIZE is variable according to configuration options, don't
> expose it to userspace. Userspace should be using sysconf(_SC_PAGE_SIZE)
> instead. Move some other noise inside __KERNEL__ too while we're at it.
> 
> Signed-off-by: David Woodhouse <dwmw2@infradead.org>
> 
> diff --git a/include/asm-mips/page.h b/include/asm-mips/page.h
> index 6ed1151..ee2ef88 100644
> --- a/include/asm-mips/page.h
> +++ b/include/asm-mips/page.h
> @@ -14,8 +14,6 @@ #ifdef __KERNEL__
>  
>  #include <spaces.h>
>  
> -#endif
> -
>  /*
>   * PAGE_SHIFT determines the page size
>   */
> @@ -35,7 +33,6 @@ #define PAGE_SIZE	(1UL << PAGE_SHIFT)
>  #define PAGE_MASK       (~((1 << PAGE_SHIFT) - 1))
>  
> 
> -#ifdef __KERNEL__
>  #ifndef __ASSEMBLY__
>  
>  extern void clear_page(void * page);
> @@ -168,7 +165,6 @@ #define VM_DATA_DEFAULT_FLAGS	(VM_READ |
>  #define UNCAC_ADDR(addr)	((addr) - PAGE_OFFSET + UNCAC_BASE)
>  #define CAC_ADDR(addr)		((addr) - UNCAC_BASE + PAGE_OFFSET)
>  
> -#endif /* defined (__KERNEL__) */
>  
>  #ifdef CONFIG_LIMITED_DMA
>  #define WANT_PAGE_VIRTUAL
> @@ -177,4 +173,6 @@ #endif
>  #include <asm-generic/memory_model.h>
>  #include <asm-generic/page.h>
>  
> +#endif /* defined (__KERNEL__) */
> +
>  #endif /* _ASM_PAGE_H */
> 
> 
-- 
dwmw2

Re: 2.6.18 Headers - Long

[Ralf Baechle]

On Sat, Aug 26, 2006 at 05:09:20PM +0100, David Woodhouse wrote:

> Ralf, you haven't applied this yet AFAICT...

Well, it now is.

  Ralf