linux vs. linux-libc-headers?

linux vs. linux-libc-headers?

[Steffen Sledz]

We're using Angstrom distro at hipox machine.

For this machine we need to use kernel version 2.6.24 (selected
by DEFAULT_PREFERENCE_hipox = "1" in linux_2.6.24.bb) because of
some special machine/driver support.

Angstrom uses linux-libc-headers version 2.6.31.

This results in some dirty problems, e.g. the headers report the
existence of inotify_init1 (introduced with 2.6.27) but it is not
available in 2.6.24. :(

It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
linux-libc-headers recipes. So what's the right way to handle this?
Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
in angstrom-2008.1.conf?

Steffen

-- 
Steffen Sledz
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str.3, D-10319 Berlin, Germany
Tel: +49 (30) 515932237 mailto:sledz@DResearch.DE
Fax: +49 (30) 515932299 http://www.DResearch.DE
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 54412;
Ust.-IDNr. DE169013825; WEEE Reg.-Nr. DE 85995642

Re: linux vs. linux-libc-headers?

[Graeme Gregory]

On Wed, May 05, 2010 at 11:36:32AM +0200, Steffen Sledz wrote:
> We're using Angstrom distro at hipox machine.
> 
> For this machine we need to use kernel version 2.6.24 (selected
> by DEFAULT_PREFERENCE_hipox = "1" in linux_2.6.24.bb) because of
> some special machine/driver support.
> 
> Angstrom uses linux-libc-headers version 2.6.31.
> 
> This results in some dirty problems, e.g. the headers report the
> existence of inotify_init1 (introduced with 2.6.27) but it is not
> available in 2.6.24. :(
> 
> It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
> linux-libc-headers recipes. So what's the right way to handle this?
> Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
> in angstrom-2008.1.conf?

I thought glibc was supposed to gracefully fall back on missing syscalls?

Graeme

Re: linux vs. linux-libc-headers?

[Steffen Sledz]

Am 05.05.2010 11:42, wrote Graeme Gregory:
> I thought glibc was supposed to gracefully fall back on missing syscalls?

May be, but not satisfactorily.

E.g. we've seen that g_file_monitor (from glib) falls back to
polling each second instead of using inotify_init.

In addition we've mentioned problems with rsyslog (epoll_create1)
or udev-151.

-- 
Steffen Sledz
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str.3, D-10319 Berlin, Germany
Tel: +49 (30) 515932237 mailto:sledz@DResearch.DE
Fax: +49 (30) 515932299 http://www.DResearch.DE
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 54412;
Ust.-IDNr. DE169013825; WEEE Reg.-Nr. DE 85995642

Re: linux vs. linux-libc-headers?

[Phil Blundell]

On Wed, 2010-05-05 at 12:00 +0200, Steffen Sledz wrote:
> Am 05.05.2010 11:42, wrote Graeme Gregory:
> > I thought glibc was supposed to gracefully fall back on missing syscalls?
> 
> May be, but not satisfactorily.
> 
> E.g. we've seen that g_file_monitor (from glib) falls back to
> polling each second instead of using inotify_init.

That sounds like it's just a bug in glibc.  I guess you should fix it
there.

p.

Re: linux vs. linux-libc-headers?

[Steffen Sledz]

Am 05.05.2010 12:05, wrote Phil Blundell:
>>> I thought glibc was supposed to gracefully fall back on missing syscalls?
>>
>> May be, but not satisfactorily.
>>
>> E.g. we've seen that g_file_monitor (from glib) falls back to
>> polling each second instead of using inotify_init.
> 
> That sounds like it's just a bug in glibc.  I guess you should fix it
> there.

As mentioned before that's just one problem and we have some more.
I do not have time and knowledge to hunt and fix them all.

So for the moment i would prefer to work with linux-libc-headers
of version 2.6.24.

But what's the recommended way to select them in OE/Angstrom?

Steffen

-- 
Steffen Sledz
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str.3, D-10319 Berlin, Germany
Tel: +49 (30) 515932237 mailto:sledz@DResearch.DE
Fax: +49 (30) 515932299 http://www.DResearch.DE
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 54412;
Ust.-IDNr. DE169013825; WEEE Reg.-Nr. DE 85995642

[PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Steffen Sledz]

Signed-off-by: Steffen Sledz <sledz@dresearch.de>
---
 conf/distro/angstrom-2008.1.conf |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/conf/distro/angstrom-2008.1.conf b/conf/distro/angstrom-2008.1.conf
index 1df0c0b..760166b 100644
--- a/conf/distro/angstrom-2008.1.conf
+++ b/conf/distro/angstrom-2008.1.conf
@@ -81,7 +81,8 @@ PREFERRED_VERSION_linux-handhelds-2.6 ?= "2.6.21-hh20"
 #KERNEL_INITRAMFS_PATH = "${KERNEL_INITRAMFS_DIR}/$(readlink ${KERNEL_INITRAMFS_DIR}initramfs-bootmenu-image-${MACHINE}.cpio.gz)"
 
 #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
-PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
+PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
+PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
 
 #Prefer glibc 2.6 and uclibc 0.9.30, these have had the most testing.
 PREFERRED_VERSION_glibc 		?= "2.9"
-- 
1.6.4.2

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06-05-10 12:15, Steffen Sledz wrote:
> Signed-off-by: Steffen Sledz <sledz@dresearch.de>
> ---
>  conf/distro/angstrom-2008.1.conf |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/conf/distro/angstrom-2008.1.conf b/conf/distro/angstrom-2008.1.conf
> index 1df0c0b..760166b 100644
> --- a/conf/distro/angstrom-2008.1.conf
> +++ b/conf/distro/angstrom-2008.1.conf
> @@ -81,7 +81,8 @@ PREFERRED_VERSION_linux-handhelds-2.6 ?= "2.6.21-hh20"
>  #KERNEL_INITRAMFS_PATH = "${KERNEL_INITRAMFS_DIR}/$(readlink ${KERNEL_INITRAMFS_DIR}initramfs-bootmenu-image-${MACHINE}.cpio.gz)"
>  
>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
>  
>  #Prefer glibc 2.6 and uclibc 0.9.30, these have had the most testing.
>  PREFERRED_VERSION_glibc 		?= "2.9"

NACK, that creates undefined behaviour for multimachine builds.

Also, please remove the legacy staging behaviour from the
linux-libc-headers recipe you just pushed.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFL4p0mMkyGM64RGpERAug1AJ9uEHk3KtRBXbILlfAM9IKV8kPC2ACfQHF4
p8l5iA3z+D1L5x+17Yq/u+Y=
=rjzW
-----END PGP SIGNATURE-----

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Steffen Sledz]

Am 06.05.2010 12:42, wrote Koen Kooi:
>> Signed-off-by: Steffen Sledz <sledz@dresearch.de>
>> ---
>>  conf/distro/angstrom-2008.1.conf |    3 ++-
>>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
>> diff --git a/conf/distro/angstrom-2008.1.conf b/conf/distro/angstrom-2008.1.conf
>> index 1df0c0b..760166b 100644
>> --- a/conf/distro/angstrom-2008.1.conf
>> +++ b/conf/distro/angstrom-2008.1.conf
>> @@ -81,7 +81,8 @@ PREFERRED_VERSION_linux-handhelds-2.6 ?= "2.6.21-hh20"
>>  #KERNEL_INITRAMFS_PATH = "${KERNEL_INITRAMFS_DIR}/$(readlink ${KERNEL_INITRAMFS_DIR}initramfs-bootmenu-image-${MACHINE}.cpio.gz)"
> 
>>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
> 
>>  #Prefer glibc 2.6 and uclibc 0.9.30, these have had the most testing.
>>  PREFERRED_VERSION_glibc 		?= "2.9"
> 
> NACK, that creates undefined behaviour for multimachine builds.

Hmmm? So what's the right way to force s special headers version
for a machine?

> Also, please remove the legacy staging behaviour from the
> linux-libc-headers recipe you just pushed.

OK. I'll have a look at this.

-- 
Steffen Sledz
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str.3, D-10319 Berlin, Germany
Tel: +49 (30) 515932237 mailto:sledz@DResearch.DE
Fax: +49 (30) 515932299 http://www.DResearch.DE
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 54412;
Ust.-IDNr. DE169013825; WEEE Reg.-Nr. DE 85995642

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Steffen Sledz]

Am 06.05.2010 12:42, wrote Koen Kooi:
> Also, please remove the legacy staging behaviour from the
> linux-libc-headers recipe you just pushed.

Fixed.

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Steffen Sledz]

>>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
>
> NACK, that creates undefined behaviour for multimachine builds.

-PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
+PREFERRED_VERSION_linux-libc-headers       = "2.6.31"
+PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"

Is this better?

-- 
Steffen Sledz
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str.3, D-10319 Berlin, Germany
Tel: +49 (30) 515932237 mailto:sledz@DResearch.DE
Fax: +49 (30) 515932299 http://www.DResearch.DE
Geschäftsführer: Dr. Michael Weber, Werner Mögle;
Amtsgericht Berlin Charlottenburg; HRB 54412;
Ust.-IDNr. DE169013825; WEEE Reg.-Nr. DE 85995642

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07-05-10 08:59, Steffen Sledz wrote:
>>>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
>>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>>> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
>>> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
>>
>> NACK, that creates undefined behaviour for multimachine builds.
> 
> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
> +PREFERRED_VERSION_linux-libc-headers       = "2.6.31"
> +PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
> 
> Is this better?

No, it still changes the headers for one machine, which leads to
undefined behaviour for other machines using the same arch.
Any solution that doesn't mark *all* packages as machine specific for
hipox is going to cause that behaviour.

regards,

Koen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFL47/0MkyGM64RGpERAmtKAJ4i6iHnKPD3Jhc2KGiFtQzj1g2kVQCeP8aa
fthjVFJAj6ahKSlqhgHA3yk=
=v8r3
-----END PGP SIGNATURE-----

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Steffen Sledz]

Am 07.05.2010 09:23, wrote Koen Kooi:
>>>>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
>>>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>>>> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
>>>> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
>>>
>>> NACK, that creates undefined behaviour for multimachine builds.
> 
>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers       = "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
> 
>> Is this better?
> 
> No, it still changes the headers for one machine, which leads to
> undefined behaviour for other machines using the same arch.
> Any solution that doesn't mark *all* packages as machine specific for
> hipox is going to cause that behaviour.

Hmmmm? So what is the right method to change the header version
for a machine which needs a specific kernel version? Something like
PREFERRED_VERSION_linux-libc-headers in machine config?

Evidently glibc does not fall back gracefully on missing syscalls.

Regards,
Steffen

Re: linux vs. linux-libc-headers?

[Phil Blundell]

On Wed, 2010-05-05 at 16:50 +0200, Steffen Sledz wrote:
> As mentioned before that's just one problem and we have some more.
> I do not have time and knowledge to hunt and fix them all.

That is unfortunate.

> So for the moment i would prefer to work with linux-libc-headers
> of version 2.6.24.
> 
> But what's the recommended way to select them in OE/Angstrom?

As far as OE goes, it's fairly easy: you just set
PREFERRED_VERSION_linux-libc-headers appropriately.  But, if Angstrom
has taken a distro decision to use the 2.6.31 headers then you will have
a problem: if you just make that change for one MACHINE, your binaries
will now be different to everybody else's and hence what you are
building is not really Angstrom anymore.

If you are not willing to fix glibc then I think there are only three
sensible ways to proceed:

a) update your machine to use the 2.6.31 kernel so that the problem goes
away; or

b) persuade the Angstrom maintainers to adopt 2.6.24 headers as their
DISTRO standard, though this will presumably cause some fallout in terms
of other incompatibilities so I suspect it will be a struggle; or

c) create your own DISTRO, which can be a derivative/fork of Angstrom if
you wish, and in which you can obviously do whatever you like.  This
probably seems like a bit of a large hammer to crack that particular nut
but it is more or less the only way in which you can use a different
version from the other Angstrom folks.

p.

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Steffen Sledz]

>>>>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
>>>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>>>> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
>>>> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
>>>
>>> NACK, that creates undefined behaviour for multimachine builds.
>
>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers       = "2.6.31"
>> +PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
>
>> Is this better?
>
> No, it still changes the headers for one machine, which leads to
> undefined behaviour for other machines using the same arch.
> Any solution that doesn't mark *all* packages as machine specific for
> hipox is going to cause that behaviour.

Would be a

-PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
+PREFERRED_VERSION_linux-libc-headers  ?= "2.6.31"

acceptable to allow overwrites in local.conf?

BTW: I did not found any fallback handling for inotify_init1 (added with 2.6.27) in the glibc.

Regards,
Steffen

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10-05-10 09:36, Steffen Sledz wrote:
>>>>>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
>>>>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>>>>> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
>>>>> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
>>>>
>>>> NACK, that creates undefined behaviour for multimachine builds.
>>
>>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
>>> +PREFERRED_VERSION_linux-libc-headers       = "2.6.31"
>>> +PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
>>
>>> Is this better?
>>
>> No, it still changes the headers for one machine, which leads to
>> undefined behaviour for other machines using the same arch.
>> Any solution that doesn't mark *all* packages as machine specific for
>> hipox is going to cause that behaviour.
> 
> Would be a
> 
> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
> +PREFERRED_VERSION_linux-libc-headers  ?= "2.6.31"
> 
> acceptable to allow overwrites in local.conf?

This comes up about every month (search the archives) and the answer
remains the same: no
This will introduce the same undefined behaviour that was pointed out
earlier.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFL58T7MkyGM64RGpERApf+AJ9vJq9lk5TnCW6ykMU/QzFyTZvqtQCfY5SI
HV39oHD0RIw4Y4sCFY5QpqY=
=LzFt
-----END PGP SIGNATURE-----

Re: linux vs. linux-libc-headers?

[Steffen Sledz]

Am 05.05.2010 12:05, Phil Blundell wrote:
>>> I thought glibc was supposed to gracefully fall back on missing syscalls?
>>
>> May be, but not satisfactorily.
>>
>> E.g. we've seen that g_file_monitor (from glib) falls back to
>> polling each second instead of using inotify_init.
> 
> That sounds like it's just a bug in glibc.  I guess you should fix it
> there.

That's all very unsatisfyingly.

I had a small look into the glibc sources. I could find the
declarations for inotfiy_init1 and epoll_create1, but i could not
find any fallback code if they do not exist in the running kernel.

I also could not find something about such fallback code at all.
FAQ 3.21 describes the problem what we have but for very old
kernel versions (2.0 vs. 2.1/2.2) and it says nothing about
glibc internal fallback handling but:

"Your program should check at runtime whether the function works,
and implement a fallback."

So where did this assumption come from?

Do you have examples for such fallbacks inside glibc?

Regards,
Steffen

Kernel Headers Quality Issue (was: linux vs. linux-libc-headers?)

[Thilo Fromm]

Hello Guys,

this is a somewhat cumulated response - I'd like to address some of the 
things mentioned at different points in the discussion in one place. 
Please forgive me should I misquote things.



Graeme Gregory in <20100505094242.GF2444@xora-desktop.xora.org.uk>:

[Steffen Sledz]
 > > It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
 > > linux-libc-headers recipes. So what's the right way to handle this?
 > > Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
 > > in angstrom-2008.1.conf?
 >
[Graeme Gregory]
 > I thought glibc was supposed to gracefully fall back on missing
 > syscalls?

Glibc is compiled against 2.6.31 headers, which is one of our main 
issues here. It only ever *runs* with a 2.6.24 kernel on the target 
system, though. So it cannot know about missing syscalls until runtime.

And no, it isn't very gracefully falling anywhere then - it simply 
cannot, as there is no runtime type checking in C. All you got is 
compile-time checking. You can happily link together completely 
incompatible interfaces if the header files are lying to you. And you 
won't even notice.

Basically, that's what's happening to us: We need to use 2.6.24 but the 
build system tells everybody (including the C library) that they will 
run with (at least) 2.6.31. As the kernel/userspace ABI is guaranteed to 
be stable (let's forget about EABI/OABI for a sec...) this will work the 
other way around, i.e. it's OK to build libc, say, against 2.6.28 kernel 
headers, and have it run on a 2.6.32 system (which, by the way, the 
latest ubuntu does). You're unable to use kernel ABI features which came 
after 2.6.28 in any userspace application linking to that libc, but the 
runtime will work.

I think pretending to have a 2.6.31 ABI while only providing a 2.6.24 
runtime is dangerous at least. And I think it is a massive bug in the 
build system if it lets you build such a dysfunctional system (because 
oe *knows* that 2.6.24 will run the system and still provides 2.6.31 
headers, thereby breaking things).


------------


Phil Blundell in <1273053926.18119.331.camel@mill.internal.reciva.com>:

 > That sounds like it's just a bug in glibc.  I guess you should fix it
 > there.

What way are we supposed to fix missing kernel ABI features (mind the 
2.6.31 / 2.6.24 feature gap) in the C library? By implementing the 
misssing functionality in the library? Then maybe port these 
library-added-kernel-features to uClibc, dietlibc, or any other C 
library if we feel the need to switch libraries? I don't think so.


------------


Phil Blundell in <1273222986.4378.18.camel@lenovo.internal.reciva.com>:

 > > So for the moment i would prefer to work with linux-libc-headers
 > > of version 2.6.24.
 > >
 > > But what's the recommended way to select them in OE/Angstrom?
 >
 > As far as OE goes, it's fairly easy: you just set
 > PREFERRED_VERSION_linux-libc-headers appropriately.

I think that's the way to go, then.

 >
 > If you are not willing to fix glibc then I think there are only three
 > sensible ways to proceed:
 >

I don't think you can "fix" missing kernel features in the library.

 > a) update your machine to use the 2.6.31 kernel so that the problem
 > goes away; or

That's my dream option ^_^ It's just that we inherited tons of legacy 
driver code. That's the sole reason we still use 2.6.24. And that won't 
change for a while. Meh.

 > b) persuade the Angstrom maintainers to adopt 2.6.24 headers as their
 > DISTRO standard, though this will presumably cause some fallout in
 > terms of other incompatibilities so I suspect it will be a struggle;
 > or

"Hey, we'd like you guys to build your C library against totally 
outdated kernel header files for your whole distribution, thereby 
breaking all the user space software which depends on newer features 
(like udev does), because we're sitting on a legacy kernel version for 
our architecture (which no one's using except us)."

Let me think.

NO. ^_^

 > c) create your own DISTRO, which can be a derivative/fork of Angstrom
 > if you wish, and in which you can obviously do whatever you like.
 > This probably seems like a bit of a large hammer to crack that
 > particular nut but it is more or less the only way in which you can
 > use a different version from the other Angstrom folks.

I think this is the way to go here. We need to make sure that everything 
is built against our build of glibc which in turn is built against 
kernel 2.6.24 header files, or things might silently break. We basically 
do have a different distribution already (as we use a *way* older kernel 
than Angstrom, and lie to the C library about it). We just didn't notice.


------------------


There's just one issue left: I can make openembedded build my target 
libraries and applications against a more recent kernel version than the 
one I'll provide on the target. I think building against a way newer 
kernel than the one the system will run is bad. And I think openembedded 
could detect, and warn about, this issue, but it doesn't, which is also 
bad. What do you guys think?


Regards,
Thilo

-- 
Dipl.-Ing (FH) Thilo Fromm, MSc., Embedded Systems Developer
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str. 3, D-10319 Berlin, Germany
Amtsgericht: Berlin Charlottenburg, HRB:54412
Tel: +49 (30) 515 932 228   mailto:t.fromm@dresearch.de
Fax: +49 (30) 515 932 77 http://www.dresearch.de

Re: [PATCH] angstrom-2008.1: use linux-libc-headers 2.6.24 for hipox machine

[Tom Rini]

On Mon, 2010-05-10 at 10:34 +0200, Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 10-05-10 09:36, Steffen Sledz wrote:
> >>>>>  #This is unrelated to the kernel version, but userspace apps (e.g. HAL) require a recent version to build against
> >>>>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
> >>>>> +PREFERRED_VERSION_linux-libc-headers       ?= "2.6.31"
> >>>>> +PREFERRED_VERSION_linux-libc-headers_hipox ?= "2.6.24"
> >>>>
> >>>> NACK, that creates undefined behaviour for multimachine builds.
> >>
> >>> -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
> >>> +PREFERRED_VERSION_linux-libc-headers       = "2.6.31"
> >>> +PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
> >>
> >>> Is this better?
> >>
> >> No, it still changes the headers for one machine, which leads to
> >> undefined behaviour for other machines using the same arch.
> >> Any solution that doesn't mark *all* packages as machine specific for
> >> hipox is going to cause that behaviour.
> > 
> > Would be a
> > 
> > -PREFERRED_VERSION_linux-libc-headers 	= "2.6.31"
> > +PREFERRED_VERSION_linux-libc-headers  ?= "2.6.31"
> > 
> > acceptable to allow overwrites in local.conf?
> 
> This comes up about every month (search the archives) and the answer
> remains the same: no
> This will introduce the same undefined behaviour that was pointed out
> earlier.

Perhaps the fact that this comes up once a month means the Angstrom
folks may want to think about how to have a very clear to the user not
compatible with the main feeds and so on custom distribution, but that
doesn't start out with cp conf/distro/angstrom-2008.1.conf
conf/distro/mystrom.conf ?

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Kernel Headers Quality Issue (was: linux vs. linux-libc-headers?)

[Tom Rini]

On Mon, 2010-05-10 at 16:53 +0200, Thilo Fromm wrote:
> Hello Guys,
> 
> this is a somewhat cumulated response - I'd like to address some of the 
> things mentioned at different points in the discussion in one place. 
> Please forgive me should I misquote things.
> 
> 
> 
> Graeme Gregory in <20100505094242.GF2444@xora-desktop.xora.org.uk>:
> 
> [Steffen Sledz]
>  > > It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
>  > > linux-libc-headers recipes. So what's the right way to handle this?
>  > > Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
>  > > in angstrom-2008.1.conf?
>  >
> [Graeme Gregory]
>  > I thought glibc was supposed to gracefully fall back on missing
>  > syscalls?
> 
> Glibc is compiled against 2.6.31 headers, which is one of our main 
> issues here. It only ever *runs* with a 2.6.24 kernel on the target 
> system, though. So it cannot know about missing syscalls until runtime.

So, I think some of the confusion here stems from confusion about (and I
don't know the right answer off-hand) how glibc handles the
--with-kernel=VERSION stuff.

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Kernel Headers Quality Issue (was: linux vs. linux-libc-headers?)

[Khem Raj]

On (10/05/10 09:55), Tom Rini wrote:
> On Mon, 2010-05-10 at 16:53 +0200, Thilo Fromm wrote:
> > Hello Guys,
> > 
> > this is a somewhat cumulated response - I'd like to address some of the 
> > things mentioned at different points in the discussion in one place. 
> > Please forgive me should I misquote things.
> > 
> > 
> > 
> > Graeme Gregory in <20100505094242.GF2444@xora-desktop.xora.org.uk>:
> > 
> > [Steffen Sledz]
> >  > > It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
> >  > > linux-libc-headers recipes. So what's the right way to handle this?
> >  > > Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
> >  > > in angstrom-2008.1.conf?
> >  >
> > [Graeme Gregory]
> >  > I thought glibc was supposed to gracefully fall back on missing
> >  > syscalls?
> > 
> > Glibc is compiled against 2.6.31 headers, which is one of our main 
> > issues here. It only ever *runs* with a 2.6.24 kernel on the target 
> > system, though. So it cannot know about missing syscalls until runtime.
> 
> So, I think some of the confusion here stems from confusion about (and I
> don't know the right answer off-hand) how glibc handles the
> --with-kernel=VERSION stuff.

Well based on --with-kernel supplied to it during build time it 
configures syscalls that are available in that version of kernel.
unlike uclibc which configures based on the kernel headers supplied
to it at buildtime glibc does not assume the version to be the 
one supplied during build but with --with-kernel you can make it
know that it will not be run on kernel versions less than one specified
to --with-kernel

so it should be ok to compile glibc using newer version of kernel and run
it on an older version of kernel.

-Khem

Re: Kernel Headers Quality Issue (was: linux vs. linux-libc-headers?)

[Tom Rini]

On Mon, 2010-05-10 at 16:53 +0200, Thilo Fromm wrote:
> Hello Guys,
> 
> this is a somewhat cumulated response - I'd like to address some of the 
> things mentioned at different points in the discussion in one place. 
> Please forgive me should I misquote things.
> 
> 
> 
> Graeme Gregory in <20100505094242.GF2444@xora-desktop.xora.org.uk>:
> 
> [Steffen Sledz]
>  > > It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
>  > > linux-libc-headers recipes. So what's the right way to handle this?
>  > > Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
>  > > in angstrom-2008.1.conf?
>  >
> [Graeme Gregory]
>  > I thought glibc was supposed to gracefully fall back on missing
>  > syscalls?
> 
> Glibc is compiled against 2.6.31 headers, which is one of our main 
> issues here. It only ever *runs* with a 2.6.24 kernel on the target 
> system, though. So it cannot know about missing syscalls until runtime.

Looking at it again and based on what Khem said (and re-reading your
failures), the problem is not with glibc, but with other programs (glib,
udev are both mentioned specifically) and these programs specifically
not failing gracefully.  For example, it's quoted over in:
http://lists.linuxtogo.org/pipermail/openembedded-devel/2010-May/019626.html
that glib falls way back.  I think Phil was wrong here and without
checking the code, I imagine glib is trying fancy newer syscall,
failing, and then falling far back, rather than to just inotify_init as
you expected.  So the bug here is with glib.

As for the udev issue, I do not know how they will react to making the
udev version be soft-assign as that too may raise issues.  And iirc,
udev is or at least can be more tied to min kernel versions.

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Kernel Headers Quality Issue

[Thilo Fromm]

Hello, Tom,

>> this is a somewhat cumulated response - I'd like to address some of the 
>> things mentioned at different points in the discussion in one place. 
>> Please forgive me should I misquote things.
 >>
>> Graeme Gregory in <20100505094242.GF2444@xora-desktop.xora.org.uk>:
>>
>> [Steffen Sledz]
>>  > > It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
>>  > > linux-libc-headers recipes. So what's the right way to handle this?
>>  > > Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
>>  > > in angstrom-2008.1.conf?
>>  >
>> [Graeme Gregory]
>>  > I thought glibc was supposed to gracefully fall back on missing
>>  > syscalls?
>>
>> Glibc is compiled against 2.6.31 headers, which is one of our main 
>> issues here. It only ever *runs* with a 2.6.24 kernel on the target 
>> system, though. So it cannot know about missing syscalls until runtime.
> 
> So, I think some of the confusion here stems from confusion about (and I
> don't know the right answer off-hand) how glibc handles the
> --with-kernel=VERSION stuff.

What is "--with-kernel"? Glibc offers a compile time configuration 
parameter "--enable-kernel", which enables/disables *backwards support* 
library compatibility support for *legacy kernel interfaces*. E.g. if 
you --enable-kernel=2.2.14, you should be able to compile ancient user 
space applications which make use of obsolete kernel 2.2.14 
functionality against the resulting glibc. This makes systems running 
*very recent kernels* provide legacy interfaces of much older kernels 
via the C library. Which is the exact opposite of our situation.

The release notes on glibc 2.2.4, which introduced this option, are 
quite educating (it's Ulrich Drepper writing, so be prepared): 
<http://sourceware.org/ml/libc-announce/2001/msg00000.html> (about 
halfway through the page).

But maybe there is a "--with-kernel" around which does the opposite to 
"--enable-kernel"?

Regards,
Thilo

-- 
Dipl.-Ing (FH) Thilo Fromm, MSc., Embedded Systems Developer
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str. 3, D-10319 Berlin, Germany
Amtsgericht: Berlin Charlottenburg, HRB:54412
Tel: +49 (30) 515 932 228   mailto:t.fromm@dresearch.de
Fax: +49 (30) 515 932 77 http://www.dresearch.de

Re: Kernel Headers Quality Issue

[Steffen Sledz]

Am 10.05.2010 21:14, Tom Rini wrote:
> Looking at it again and based on what Khem said (and re-reading your
> failures), the problem is not with glibc, but with other programs (glib,
> udev are both mentioned specifically) and these programs specifically
> not failing gracefully.  For example, it's quoted over in:
> http://lists.linuxtogo.org/pipermail/openembedded-devel/2010-May/019626.html
> that glib falls way back.  I think Phil was wrong here and without
> checking the code, I imagine glib is trying fancy newer syscall,
> failing, and then falling far back, rather than to just inotify_init as
> you expected.  So the bug here is with glib.
> 
> As for the udev issue, I do not know how they will react to making the
> udev version be soft-assign as that too may raise issues.  And iirc,
> udev is or at least can be more tied to min kernel versions.

No, it seems not to be a problem of glib, rsyslog, or udev.

As i see in the build logs glibc is build using --enable-kernel=2.6.16 (i do not know any --with-kernel option).

After running autotools config.h of e.g. glib-2.0 contains

 /* Define to 1 if you have the `inotify_init1' function. */
 #define HAVE_INOTIFY_INIT1 1

So the glib-2.0 is allowed to use inotify_init1. But this is only available for kernel version >= 2.6.27.

Same behaviour for epoll_create1 in e.g. rsyslog

 /* Define to 1 if you have the `epoll_create' function. */
 #define HAVE_EPOLL_CREATE 1

 /* Define to 1 if you have the `epoll_create1' function. */
 #define HAVE_EPOLL_CREATE1 1

Conclusion: glibc reports functions "available" which are not available for our kernel 2.6.24 (and not "emulated" inside glibc). :(

Regards, Steffen

Re: Kernel Headers Quality Issue

[Tom Rini]

On Tue, 2010-05-11 at 09:42 +0200, Steffen Sledz wrote:
> Am 10.05.2010 21:14, Tom Rini wrote:
> > Looking at it again and based on what Khem said (and re-reading your
> > failures), the problem is not with glibc, but with other programs (glib,
> > udev are both mentioned specifically) and these programs specifically
> > not failing gracefully.  For example, it's quoted over in:
> > http://lists.linuxtogo.org/pipermail/openembedded-devel/2010-May/019626.html
> > that glib falls way back.  I think Phil was wrong here and without
> > checking the code, I imagine glib is trying fancy newer syscall,
> > failing, and then falling far back, rather than to just inotify_init as
> > you expected.  So the bug here is with glib.
> > 
> > As for the udev issue, I do not know how they will react to making the
> > udev version be soft-assign as that too may raise issues.  And iirc,
> > udev is or at least can be more tied to min kernel versions.
> 
> No, it seems not to be a problem of glib, rsyslog, or udev.
> 
> As i see in the build logs glibc is build using --enable-kernel=2.6.16 (i do not know any --with-kernel option).

Yes, my fault, sorry, it's indeed --enable-kernel=.

> After running autotools config.h of e.g. glib-2.0 contains
> 
>  /* Define to 1 if you have the `inotify_init1' function. */
>  #define HAVE_INOTIFY_INIT1 1
> 
> So the glib-2.0 is allowed to use inotify_init1. But this is only available for kernel version >= 2.6.27.
> 
> Same behaviour for epoll_create1 in e.g. rsyslog
> 
>  /* Define to 1 if you have the `epoll_create' function. */
>  #define HAVE_EPOLL_CREATE 1
> 
>  /* Define to 1 if you have the `epoll_create1' function. */
>  #define HAVE_EPOLL_CREATE1 1
> 
> Conclusion: glibc reports functions "available" which are not available for our kernel 2.6.24 (and not "emulated" inside glibc). :(

All of this information is _not_ coming from glibc claiming that a
syscall exists, but it's coming from the linux-libc-headers having them
(which is why if you downgrade, they build and run as you expect them
to).  Then there's the problem, as others have stated, that it's up to
userland to gracefully handle if a syscall isn't really available.  You
noted before that glib-2.0 falls back to using something very
inefficient, rather than a less efficient syscall that does exist, yes?

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Kernel Headers Quality Issue

[Steffen Sledz]

11.05.2010 16:27, Tom Rini wrote:
> All of this information is _not_ coming from glibc claiming that a
> syscall exists, but it's coming from the linux-libc-headers having them
> (which is why if you downgrade, they build and run as you expect them
> to).  Then there's the problem, as others have stated, that it's up to
> userland to gracefully handle if a syscall isn't really available.

<grrrr> Now i'm totally confused. Here's a short summary as i understand
this thread (may be i misunderstand something).

My initial understanding was:

linux-libc-headers match to kernel version
-> autotools report existing syscalls
-> apps can use them
-> fine

Then Koen wrote: no chance to change linux-libc-headers for angstrom

Graeme and Phil wrote something like no problem, "glibc was supposed
to gracefully fall back on missing syscalls". What i understand as

linux-libc-headers may be newer than kernel version
-> autotools report availability of some syscalls (e.g. inotify_init1 or epoll_create1)
-> apps can use them
-> if called on older kernels glibc will handle this "gracefully"
-> fine

But tests showed that glibc did not handle syscalls this way.

Then you wrote that *every app* itself is responsible to handle syscalls
which are reported by autotools (linux-libc-headers) as *available* but
are *not available* ? I can't believe that. This sounds absurd for me.

So please enlighten my confusions.

Regards,
Steffen

PS:

> You noted before that glib-2.0 falls back to using something very
> inefficient, rather than a less efficient syscall that does exist, yes?

What glib does is checking the availability (by using autotools) of inotify (in various levels), fam, or plain polling.

Re: linux vs. linux-libc-headers?

[Phil Blundell]

On Mon, 2010-05-10 at 11:15 +0200, Steffen Sledz wrote:
> Do you have examples for such fallbacks inside glibc?

See do_pread64() in sysdeps/unix/sysv/linux/pread64.c for example.

p.

Re: Kernel Headers Quality Issue

[Tom Rini]

On Wed, 2010-05-12 at 08:02 +0200, Steffen Sledz wrote:
> 11.05.2010 16:27, Tom Rini wrote:
> > All of this information is _not_ coming from glibc claiming that a
> > syscall exists, but it's coming from the linux-libc-headers having them
> > (which is why if you downgrade, they build and run as you expect them
> > to).  Then there's the problem, as others have stated, that it's up to
> > userland to gracefully handle if a syscall isn't really available.
> 
> <grrrr> Now i'm totally confused. Here's a short summary as i understand
> this thread (may be i misunderstand something).

Well, here is my understanding.

> My initial understanding was:
> 
> linux-libc-headers match to kernel version

In an ideal world, yes.  Because the various <asm/...> and <linux/...>
files that are exported for use do indeed get new features added over
time. And...

> -> autotools report existing syscalls

All the autotools stuff does is:
(a) If we provide in openembedded/site/... a variable to set a check to
a value, use it
(b) Try and compile a test program to check for availability (since it
can't then run the program for a cross build).

> -> apps can use them
> -> fine

Right.

> Then Koen wrote: no chance to change linux-libc-headers for angstrom

Right.  Angstrom has a publicly usable download feed and there's no way
to tell if a file there was built vs .31 (and may have syscalls
introduced around then) or .25 (and won't).  That's a very bad situation
for Angstrom.

> Graeme and Phil wrote something like no problem, "glibc was supposed
> to gracefully fall back on missing syscalls". What i understand as
> 
> linux-libc-headers may be newer than kernel version
> -> autotools report availability of some syscalls (e.g. inotify_init1 or epoll_create1)
> -> apps can use them
> -> if called on older kernels glibc will handle this "gracefully"
> -> fine
> 
> But tests showed that glibc did not handle syscalls this way.

Yes.  And just now, Phil has provided an example of handling a syscall.

> Then you wrote that *every app* itself is responsible to handle syscalls
> which are reported by autotools (linux-libc-headers) as *available* but
> are *not available* ? I can't believe that. This sounds absurd for me.

This is how I read what Phil and possibly someone else too were saying,
and based on your observation that glib for example will fall back at
run time.

> So please enlighten my confusions.
> 
> Regards,
> Steffen
> 
> PS:
> 
> > You noted before that glib-2.0 falls back to using something very
> > inefficient, rather than a less efficient syscall that does exist, yes?
> 
> What glib does is checking the availability (by using autotools) of inotify (in various levels), fam, or plain polling.

So you did not see it change behavior at run time?  That's how I had
read your email before.

> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Kernel Headers Quality Issue

[Mark Brown]

On Wed, May 12, 2010 at 08:02:53AM +0200, Steffen Sledz wrote:
> 11.05.2010 16:27, Tom Rini wrote:

> -> autotools report availability of some syscalls (e.g. inotify_init1 or epoll_create1)
> -> apps can use them
> -> if called on older kernels glibc will handle this "gracefully"
> -> fine

> But tests showed that glibc did not handle syscalls this way.

It does but...

> Then you wrote that *every app* itself is responsible to handle syscalls
> which are reported by autotools (linux-libc-headers) as *available* but

...what it does not do is emulate syscalls which are being used by
applications directly.  There are a bunch of kernel features which are
used by glibc internally - for those it will just use fallbacks at
runtime.  For things like epoll which are used directly by applications
it's up to the application to decide if it is willing or able to cope on
older systems.  For many kernel features there's no sane possibility of
emulation.

> are *not available* ? I can't believe that. This sounds absurd for me.

glibc is frequently built for systems other than the one it runs on
(obviously, OE is a big example here) so it needs to support building
against a kernel other than the running one, and there's plenty of cases
even with desktop/server systems where it's useful to be able to link
against features which can't be run (eg, user installs old kernel for a
binary driver, or a vendor wants to distribute a binary which can
optionally use newer features but still run on older systems).

Re: Kernel Headers Quality Issue

[Phil Blundell]

On Wed, 2010-05-12 at 08:02 +0200, Steffen Sledz wrote:
> linux-libc-headers may be newer than kernel version
> -> autotools report availability of some syscalls (e.g. inotify_init1 or epoll_create1)
> -> apps can use them
> -> if called on older kernels glibc will handle this "gracefully"
> -> fine
> 
> But tests showed that glibc did not handle syscalls this way.

It sounds like we have, perhaps, been talking at slight cross purposes.

It is true that you can build glibc against a different version of
linux-libc-headers than the version of the kernel that will be used at
runtime.  So long as the runtime kernel version is newer than the
minimum kernel version that glibc was configured for (either via
--enable-kernel=XXX, or via an architecture-specific static definition)
then everything should still work fine, and glibc will use internal
fallbacks if it discovers that certain system calls are not available at
runtime.

Similarly, glibc will make some efforts to provide a backwards
compatible emulation for user-visible interfaces which might not be
available in all kernel versions.  However, what it doesn't attempt to
do is provide a completely virtualised ABI which would insulate the
application entirely from the details of the running kernel.  There are
some kernel features which simply aren't feasible to emulate in user
space.

In the specific case of epoll_create1() and inotify_init1(), it would be
reasonable for glibc to do something similar to the way that, say,
mmap64() is handled.  That is, if the kernel doesn't have
epoll_create1() but does have epoll_create(), and the behaviour that the
user has requested is something that epoll_create() can do (i.e. none of
the flag bits are set) then use epoll_create().  But if the user has
asked for EPOLL_CLOEXEC then epoll_create() is not a suitable
replacement and you are going to get ENOSYS: it's then up to the
application to take some suitable recovery action.

p.

Re: Kernel Headers Quality Issue

[Thilo Fromm]

Hello *.*,

>>> Graeme Gregory in <20100505094242.GF2444@xora-desktop.xora.org.uk>:
>>>
>>> [Steffen Sledz]
>>>  > > It seem's not to be possible to use DEFAULT_PREFERENCE_hipox in the
>>>  > > linux-libc-headers recipes. So what's the right way to handle this?
>>>  > > Something like PREFERRED_VERSION_linux-libc-headers_hipox = "2.6.24"
>>>  > > in angstrom-2008.1.conf?
>>>  >
>>> [Graeme Gregory]
>>>  > I thought glibc was supposed to gracefully fall back on missing
>>>  > syscalls?
>>>
>>> Glibc is compiled against 2.6.31 headers, which is one of our main 
>>> issues here. It only ever *runs* with a 2.6.24 kernel on the target 
>>> system, though. So it cannot know about missing syscalls until runtime.
>> So, I think some of the confusion here stems from confusion about (and I
>> don't know the right answer off-hand) how glibc handles the
>> --with-kernel=VERSION stuff.
> 
> Well based on --with-kernel supplied to it during build time it 
> configures syscalls that are available in that version of kernel.

No it doesn't. Quoting the documentation of "--enable-kernel=VERSION" 
from <http://sourceware.org/ml/libc-announce/2001/msg00000.html>:

-------------
One Linux systems the configure script has a new option
`--enable-kernel' (find the documentation in the INSTALL file).  This
option allows one to strip out compatibility code for older kernel
versions.  This is of interest since configuring for a 2.4.x kernel
reduces the libc size by about 1%.  This is no high percentage but all
the code gone is in the by far most often used functions.  The
compatibility code is needed because of poor design decisions of the
kernel developers who constantly have to adjust the interface to new
requirements.  If you never expect to run kernel versions older than
the one used at compile time of the library it is a good idea to pass
`--enable-kernel=current' to configure.  But be careful since with an
older kernel the program won't even start and the whole system might
be rendered useless (unless backup kernels are available).
-------------

What it really does is providing deprecated (very, very old) kernel 
interfaces to deprecated applications running on a very recent kernel 
(which does not provide these interfaces anymore).

> unlike uclibc which configures based on the kernel headers supplied
> to it at buildtime glibc does not assume the version to be the 
> one supplied during build but with --with-kernel you can make it
> know that it will not be run on kernel versions less than one specified
> to --with-kernel

This way Glibc can decide which legacy interfaces to provide. It will be 
slightly less bloated e.g. when providing only 2.6 series compatibility 
(so applications which use 2.4 kernel interfaces deprecated by 2.6 won't 
compile or run).

> so it should be ok to compile glibc using newer version of kernel and run
> it on an older version of kernel.

No it isn't. --enable-kernel=VERSION provides kernel ABI backwards 
compatibility to applications. *At least* the kernel ABI described in 
the headers provided to glibc during compile time must be present at 
runtime.

Glibc simply has no other means than the kernel headers to figure out 
which kernel ABI functions will really be there at runtime. Glibc cannot 
figure out at runtime that it runs on an older kernel which lacks *some* 
of the ABI glibc relies on. It can only rely on the information in the 
kernel headers provided at compile time.

Even worse: At runtime glibc cannot detect that some of the kernel 
interfaces are different or not present at all. It will blindly use the 
interfaces it knows from its header files, resulting in strange 
application behaviour in these corner cases.

Regards,
Thilo

-- 
Dipl.-Ing (FH) Thilo Fromm, MSc., Embedded Systems Developer
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str. 3, D-10319 Berlin, Germany
Amtsgericht: Berlin Charlottenburg, HRB:54412
Tel: +49 (30) 515 932 228   mailto:t.fromm@dresearch.de
Fax: +49 (30) 515 932 77 http://www.dresearch.de

Re: Kernel Headers Quality Issue

[Thilo Fromm]

Hello *.*,

> It sounds like we have, perhaps, been talking at slight cross purposes.
> 
> It is true that you can build glibc against a different version of
> linux-libc-headers than the version of the kernel that will be used at
> runtime.  So long as the runtime kernel version is newer than the
> minimum kernel version that glibc was configured for (either via
> --enable-kernel=XXX, or via an architecture-specific static definition)
> then everything should still work fine, and glibc will use internal
> fallbacks if it discovers that certain system calls are not available at
> runtime.
> 
> Similarly, glibc will make some efforts to provide a backwards
> compatible emulation for user-visible interfaces which might not be
> available in all kernel versions.  However, what it doesn't attempt to
> do is provide a completely virtualised ABI which would insulate the
> application entirely from the details of the running kernel.  There are
> some kernel features which simply aren't feasible to emulate in user
> space.

In the openembedded world glibc will present the kernel headers provided 
to it at compile time to any application being compiled. So every 
application on our system thinks it's running on 2.6.31 (version of 
linux-libc-headers) while it runs on 2.6.24.

> 
> In the specific case of epoll_create1() and inotify_init1(), it would be
> reasonable for glibc to do something similar to the way that, say,
> mmap64() is handled.  That is, if the kernel doesn't have
> epoll_create1() but does have epoll_create(), and the behaviour that the
> user has requested is something that epoll_create() can do (i.e. none of
> the flag bits are set) then use epoll_create().  But if the user has
> asked for EPOLL_CLOEXEC then epoll_create() is not a suitable
> replacement and you are going to get ENOSYS: it's then up to the
> application to take some suitable recovery action.

It is a compile time matter: when an application is configured for 
compilation, then glibc system headers (2.6.31) will claim that certain 
interfaces (like e.g. inotify_init1) are there while they really are not 
at runtime (2.6.24).

Let's see how an application could handle this situation:

If my application includes <sys/inotify.h> and it is using 
inotify_init1() I expect the compilation to fail if the interface will 
not be available at runtime. I can, however, provide compatibility to 
older kernels at application level by providing different 
implementations based on configure.h and HAVE_INOTIFY_INIT1 (or 
similar). Configure figures out runtime availability of all non-portable 
interfaces, so based on the results of a configure run I can decide 
about which interfaces to use. Trying this at runtime is next to impossible.

I have no means of detecting a level of compatibility at runtime. How 
could that work, anyway? I get ENOSYS when calling inotify_init1(), so, 
well I *think* this interface is missing (although configure reported it 
being available). Then I would have to dynamically fall back to 
inotify_init() + fcntl(). What if that returns ENOSYS? Fall back to 
dnotify? And repeat this dance every time I need to call 
inotify_init1()? Basically, I'm blind at runtime concerning kernel 
feature availability when my system header files lied to me at compile 
time. Compatibility code would have to fumble for the interfaces 
*really* available at runtime by using trial-and-error. I would not 
expect anyone to provide this kind of compatibility.

Regards,
Thilo

-- 
Dipl.-Ing (FH) Thilo Fromm, MSc., Embedded Systems Developer
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str. 3, D-10319 Berlin, Germany
Amtsgericht: Berlin Charlottenburg, HRB:54412
Tel: +49 (30) 515 932 228   mailto:t.fromm@dresearch.de
Fax: +49 (30) 515 932 77 http://www.dresearch.de

Re: Kernel Headers Quality Issue

[Phil Blundell]

On Fri, 2010-05-14 at 11:59 +0200, Thilo Fromm wrote:
> I have no means of detecting a level of compatibility at runtime. How 
> could that work, anyway? I get ENOSYS when calling inotify_init1(), so, 
> well I *think* this interface is missing (although configure reported it 
> being available). Then I would have to dynamically fall back to 
> inotify_init() + fcntl().

Yes, precisely.  ENOSYS is indeed diagnostic of an unimplemented system
call, and what you have described is exactly what glibc does internally
for this kind of situation.  See for example the backwards compatibility
code for mmap2 vs mmap:

	DO_CALL (mmap2, 0)
	cmn	r0, $4096
# ifdef __ASSUME_MMAP2_SYSCALL
	ldr	r4, [sp], #4
	ldr	r5, [sp], #4
	RETINSTR(cc, lr)	
	b	PLTJMP(syscall_error)
# else
	ldrcc	r4, [sp], #4
	ldrcc	r5, [sp], #4
	RETINSTR(cc, lr)
	cmn	r0, $ENOSYS
	bne	.Lerror
	/* The current kernel does not support mmap2.  Fall back to plain
	   mmap if the offset is small enough.  */
	ldr	r5, [sp, $16]
	mov	r0, ip			@ first arg was clobbered
	teq	r5, $0
	ldreq	r4, [sp], #4
	ldreq	r5, [sp], #4
	beq	PLTJMP(__mmap)
#endif

> What if that returns ENOSYS? Fall back to dnotify?

If you wish to support kernels that don't have inotify at all, yes.
Obviously it's up to you how far back you want to go in terms of
compatibility, and I suspect most people would just give up at this
point.

> And repeat this dance every time I need to call inotify_init1()?

Yes.  Or, if this is performance critical code, you can cache the result
(since the kernel is unlikely to suddenly gain support for a syscall
that it didn't have last time) and go directly to the implementation
that works.

> Compatibility code would have to fumble for the interfaces 
> *really* available at runtime by using trial-and-error. I would not 
> expect anyone to provide this kind of compatibility.

Clearly you don't like the idea very much but yes, this is basically how
backwards compatibility code is done, and it is more or less an
inescapable consequence of the rather fluid nature of the Linux syscall
ABI.

Of course, you always have the option of simply writing to the lowest
common denominator and using inotify_init() directly rather than
inotify_init1().  Or, in the case of inotify, you have the option of
using a library which will hide this complexity for you and provide an
abstract userspace API which remains the same no matter what the
underlying implementation is doing.

This is not the only situation in which robust applications will adapt
themselves to the runtime environment in which they find themselves.  In
socket-based programs, a fairly common paradigm is something along the
lines of:

	fd = socket (PF_INET6, ...);
	if (fd < 0 && errno == EPFNOSUPPORT) {
		// IPv6 support is missing.  Use IPv4 instead.
		fd = socket (PF_INET, ...);
	}

in order that the application doesn't cease to work altogether for
people who have forgotten to load the ipv6 module.

p.

Re: Kernel Headers Quality Issue

[Thilo Fromm]

Hello *.*,

>> I have no means of detecting a level of compatibility at runtime. How 
>> could that work, anyway? I get ENOSYS when calling inotify_init1(), so, 
>> well I *think* this interface is missing (although configure reported it 
>> being available). Then I would have to dynamically fall back to 
>> inotify_init() + fcntl().
> 
> Yes, precisely.  ENOSYS is indeed diagnostic of an unimplemented system
> call, and what you have described is exactly what glibc does internally
> for this kind of situation.

That's not what I meant in the example. I was talking about an 
application which is using ./configure at compile time to figure out 
which features will be present at runtime.

>> What if that returns ENOSYS? Fall back to dnotify?
> 
> If you wish to support kernels that don't have inotify at all, yes.
> Obviously it's up to you how far back you want to go in terms of
> compatibility, and I suspect most people would just give up at this
> point.
> 
>> And repeat this dance every time I need to call inotify_init1()?
> 
> Yes.  Or, if this is performance critical code, you can cache the result
> (since the kernel is unlikely to suddenly gain support for a syscall
> that it didn't have last time) and go directly to the implementation
> that works.

This would mean that checking for kernel features (e.g. inotify_init1()) 
at compile time via an application's configure.ac is pointless, right? 
Because configure will report availability of this interface even though 
it won't be there when the application runs. In other words, everything 
configure is telling me about kernel features (native kernel thread 
availability, inotify, sysfs, ...) might be false anyway. Code like this

#ifdef HAVE_INOTIFY_INIT1
	fd = inotify_init1(O_NONBLOCK);
#else
	fd = inotify_init();
	flags = fcntl(fd, F_GETFL, 0);
	fcntl(ret, F_SETFL, flags | O_NONBLOCK);
#endif

will break at runtime even though everything was OK at compile time, and 
you are proposing that I should expect it to break.

>> Compatibility code would have to fumble for the interfaces 
>> *really* available at runtime by using trial-and-error. I would not 
>> expect anyone to provide this kind of compatibility.
> 
> Clearly you don't like the idea very much but yes, this is basically how
> backwards compatibility code is done, and it is more or less an
> inescapable consequence of the rather fluid nature of the Linux syscall
> ABI.

I don't like your proposal... but not from an application programmer's 
POV, but from the perspective of a system integrator (i.e. a 
distribution maintainer). I think just assuming that your proposal is 
true for every application I need to integrate would be a mistake and 
devastating to a distribution's overall quality.

> Of course, you always have the option of simply writing to the lowest
> common denominator and using inotify_init() directly rather than
> inotify_init1().  Or, in the case of inotify, you have the option of
> using a library which will hide this complexity for you and provide an
> abstract userspace API which remains the same no matter what the
> underlying implementation is doing.

Well, what to do _if_ a feature is not present is a whole different 
story. But for figuring out _whether_ it is present or not you propose 
(blind) runtime probing while I would like to rely on configure output, 
at least for the applications I did not write. As at least some of the 
applications integrated by openembedded do not share your views, so 
they'll do weird things at runtime. This is a severe quality issue. And 
we have only a small chance of noticing any application runtime 
misbehaviour. The syscall in question might only rarely be used, it 
might be some corner-case of an application that is executed e.g. only 
once in a month.

To be sure we'd have to have 100% code coverage of every application / 
library which might contain any syscalls.

Regards,
Thilo

-- 
Dipl.-Ing (FH) Thilo Fromm, MSc., Embedded Systems Developer
DResearch Digital Media Systems GmbH
Otto-Schmirgal-Str. 3, D-10319 Berlin, Germany
Amtsgericht: Berlin Charlottenburg, HRB:54412
Tel: +49 (30) 515 932 228   mailto:t.fromm@dresearch.de
Fax: +49 (30) 515 932 77 http://www.dresearch.de

Re: Kernel Headers Quality Issue

[Phil Blundell]

On Fri, 2010-05-14 at 13:40 +0200, Thilo Fromm wrote:
> This would mean that checking for kernel features (e.g. inotify_init1()) 
> at compile time via an application's configure.ac is pointless, right?

Not entirely pointless: the feature set that you determine at configure
time is the maximum set that you can expect to be available.  That is,
if you determine at configure time that __NR_inotify_init1 is not
defined, you clearly should not even attempt to use this syscall.

> Code like this
> 
> #ifdef HAVE_INOTIFY_INIT1
> 	fd = inotify_init1(O_NONBLOCK);
> #else
> 	fd = inotify_init();
> 	flags = fcntl(fd, F_GETFL, 0);
> 	fcntl(ret, F_SETFL, flags | O_NONBLOCK);
> #endif
> 
> will break at runtime even though everything was OK at compile time, and 
> you are proposing that I should expect it to break.

Correct.  In this particular case you would probably be better off just
using the "else" block at all times.

> Well, what to do _if_ a feature is not present is a whole different 
> story. But for figuring out _whether_ it is present or not you propose 
> (blind) runtime probing while I would like to rely on configure output, 

I'm not quite sure why you describe this as "blind" probing.  There is
nothing blind about it; the semantics of unimplemented syscalls are
quite well-defined.

In the general case, there simply isn't any way to determine statically
at compile time what features will be available at run time.  Some
features might require modules that aren't loaded, or might be disabled
by kernel commandline options, or might not be supported on the
particular CPU that the kernel finds itself running on, or any number of
other things.  As I have mentioned before, this is exactly what glibc
itself does internally in order to cope with multiple different kernel
versions.

If you are going to rely on nonstandard and volatile Linux-specific
programming interfaces then the onus is definitely on you as the
application developer to ensure that your application behaves sensibly
under all conditions that it is likely to encounter.  You simply have to
decide which features are mandatory for your program and which are
optional.  If a mandatory feature is not available then the program will
simply not run: you might choose to produce a "kernel too old" kind of
diagnostic, like glibc does if you configure it with --enable-kernel=X
and then try to run on an older version, or you might just let it crash
with ENOSYS or some such.  If an optional feature is unavailable then
you have to provide some suitable fallback action.

> As at least some of the 
> applications integrated by openembedded do not share your views, so 
> they'll do weird things at runtime. This is a severe quality issue. And 
> we have only a small chance of noticing any application runtime 
> misbehaviour. The syscall in question might only rarely be used, it 
> might be some corner-case of an application that is executed e.g. only 
> once in a month.

If you can't tolerate the risk of applications breaking in this way
then, as a distro maintainer, you can simply set and document an
appropriate policy that avoids the situation arising (e.g. compile all
applications against the lowest common denominator headers, or lock the
versions together so that the runtime kernel version always precisely
matches the compile time header version).  That would be a perfectly
acceptable way to proceed.  

p.

Re: Kernel Headers Quality Issue

[Phil Blundell]

On Fri, 2010-05-14 at 11:25 +0200, Thilo Fromm wrote:
> No it isn't. --enable-kernel=VERSION provides kernel ABI backwards 
> compatibility to applications. *At least* the kernel ABI described in 
> the headers provided to glibc during compile time must be present at 
> runtime.

This is incorrect.  The glibc policy for a long time has been that it
should be compiled against the most recent available headers, and that
it will adapt at run time to older kernels.  See for example the FAQ:

-- 8< --

1.8.	What version of the Linux kernel headers should be used?

{AJ,UD} The headers from the most recent Linux kernel should be used.  The
headers used while compiling the GNU C library and the kernel binary used
when using the library do not need to match.  The GNU C library runs without
problems on kernels that are older than the kernel headers used.  The other
way round (compiling the GNU C library with old kernel headers and running
on a recent kernel) does not necessarily work.  For example you can't use
new kernel features if you used old kernel headers to compile the GNU C
library.

{ZW} Even if you are using a 2.0 kernel on your machine, we recommend you
compile GNU libc with 2.2 kernel headers.  That way you won't have to
recompile libc if you ever upgrade to kernel 2.2.  To tell libc which
headers to use, give configure the --with-headers switch
(e.g. --with-headers=/usr/src/linux-2.2.0/include).

Note that you must configure the 2.2 kernel if you do this, otherwise libc
will be unable to find <linux/version.h>.  Just change the current directory
to the root of the 2.2 tree and do `make include/linux/version.h'.

-- >8 --

The function of the --enable-kernel option is to eliminate backwards
compatibility code for older versions: this gives you a slightly smaller
and faster library.  For example, the mmap2 syscall was introduced on
ARM at some point around kernel 2.3.35: so, if you configure glibc with
--enable-kernel=2.3.36, it knows that mmap2 must always be available and
there is no need to build the backwards compatibility logic that I
mentioned in my last email.

p.

Re: Kernel Headers Quality Issue

[Mark Brown]

On Fri, May 14, 2010 at 01:40:24PM +0200, Thilo Fromm wrote:

>> Yes.  Or, if this is performance critical code, you can cache the result
>> (since the kernel is unlikely to suddenly gain support for a syscall
>> that it didn't have last time) and go directly to the implementation
>> that works.

> This would mean that checking for kernel features (e.g. inotify_init1())  
> at compile time via an application's configure.ac is pointless, right?  
> Because configure will report availability of this interface even though  
> it won't be there when the application runs. In other words, everything  
> configure is telling me about kernel features (native kernel thread  
> availability, inotify, sysfs, ...) might be false anyway. Code like this

If you know that you are running on the same system as you are using for
the build you *can* write a configure check which actually tries to call
the syscall (and will therefore check for -ENOSYS) but obviously this
can't be relied upon if someone runs the binary on another system or
changes the configuration of the current system.

>> Clearly you don't like the idea very much but yes, this is basically how
>> backwards compatibility code is done, and it is more or less an
>> inescapable consequence of the rather fluid nature of the Linux syscall
>> ABI.

> I don't like your proposal... but not from an application programmer's  
> POV, but from the perspective of a system integrator (i.e. a  
> distribution maintainer). I think just assuming that your proposal is  
> true for every application I need to integrate would be a mistake and  
> devastating to a distribution's overall quality.

This is something that Linux distributions (and any other system
integrator working over multiple OS releases) have always had to deal
with.  If you're shipping a binary which may run in a different
environment to the one it was built in this has always been a
possibility.