[Buildroot] glibc and --enable-kernel

[Buildroot] glibc and --enable-kernel

[Sam Bobroff]

Hi Buildroot people,

I've noticed that when building glibc, buildroot's configure command doesn't
pass any "--enable-kernel" option, which according to the documentation...

http://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html

... means that glibc is always built to support the oldest possible kernel.

At first this seems like a good idea but the documentation is misleading, and
it actually causes two problems:

1) Unnecessary compat code: If you've built a kernel and glibc together
(presumably this is common when using buildroot) then you have no need of any
compat code because the only kernel you'll be using is the one you just built.
According to the doc (above) this causes glibc to include unnecessary compat
code which slows it down.

2) Availability of new features: It appears from the code that the
documentation leaves out some important information. What actually happens
while building glibc is that the "minimum kernel version" (which is either the
version given by "--enable-kernel=" or the oldest possible version for that
architecture) sets both __LINUX_KERNEL_VERSION and __ABI_TAG_VERSION so that
glibc will never use any kernel features *newer* than this version. Therefore
you cannot get access to recent kernel features *at all* via glibc (without
manually altering glibc.mk).

(An example of a recent feature would be the sendmmsg syscall, which
requires a kernel version of 3.0.0 but the minimum version for PowerPC 64 is
2.6.24.)

I'd like to prepare a patch to make this configurable, and it obviously
wouldn't be difficult to add but I'm not sure of the best way to do it.

The most obvious way seems to be to add a menu item to the Toolchain menu, just
after "glibc version" (and only enabled when glibc is the C library), called
"glibc kernel API version" with a default value of "same as kernel headers".
The other options would be "as old as possible" or a specific version.

Comments?

Is the default value reasonable even though it changes behaviour for current
configs? (Presumably it will just make them faster :-) )

Am I missing something? Would this be useful to others?

Cheers,
Sam.

[Buildroot] glibc and --enable-kernel

[Thomas Petazzoni]

Hello!

On Wed, 27 Jan 2016 15:21:04 +1100, Sam Bobroff wrote:

> I've noticed that when building glibc, buildroot's configure command doesn't
> pass any "--enable-kernel" option, which according to the documentation...
> 
> http://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
> 
> ... means that glibc is always built to support the oldest possible kernel.
> 
> At first this seems like a good idea but the documentation is misleading, and
> it actually causes two problems:
> 
> 1) Unnecessary compat code: If you've built a kernel and glibc together
> (presumably this is common when using buildroot) then you have no need of any
> compat code because the only kernel you'll be using is the one you just built.
> According to the doc (above) this causes glibc to include unnecessary compat
> code which slows it down.
> 
> 2) Availability of new features: It appears from the code that the
> documentation leaves out some important information. What actually happens
> while building glibc is that the "minimum kernel version" (which is either the
> version given by "--enable-kernel=" or the oldest possible version for that
> architecture) sets both __LINUX_KERNEL_VERSION and __ABI_TAG_VERSION so that
> glibc will never use any kernel features *newer* than this version. Therefore
> you cannot get access to recent kernel features *at all* via glibc (without
> manually altering glibc.mk).

Thanks a lot for spotting this problem and for the explanation,
definitely makes sense.

> (An example of a recent feature would be the sendmmsg syscall, which
> requires a kernel version of 3.0.0 but the minimum version for PowerPC 64 is
> 2.6.24.)

Ah, so you're working on PowerPC 64, great! We have a toolchain build
problem that I reported on Power PC 64, when building for Power 8:

  http://lists.busybox.net/pipermail/buildroot/2016-January/150338.html

If you have some time to look at it, it would be great.

> I'd like to prepare a patch to make this configurable, and it obviously
> wouldn't be difficult to add but I'm not sure of the best way to do it.
> 
> The most obvious way seems to be to add a menu item to the Toolchain menu, just
> after "glibc version" (and only enabled when glibc is the C library), called
> "glibc kernel API version" with a default value of "same as kernel headers".
> The other options would be "as old as possible" or a specific version.
> 
> Comments?

I don't think you need to add additional options. We should just use
the kernel version of the chosen kernel headers, since we're guaranteed
that the running kernel will be at least more recent than the kernel
headers being used.

I.e, can you try something such as:

	--enable-kernel=$(call qstrip,$(BR2_TOOLCHAIN_HEADERS_AT_LEAST))

BR2_TOOLCHAIN_HEADERS_AT_LEAST will be 3.0 if you use some 3.0.x kernel
headers for your toolchain.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] glibc and --enable-kernel

[Sam Bobroff]

On Wed, Jan 27, 2016 at 09:21:19AM +0100, Thomas Petazzoni wrote:
> Hello!
> 
> On Wed, 27 Jan 2016 15:21:04 +1100, Sam Bobroff wrote:
> 
> > I've noticed that when building glibc, buildroot's configure command doesn't
> > pass any "--enable-kernel" option, which according to the documentation...
> > 
> > http://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
> > 
> > ... means that glibc is always built to support the oldest possible kernel.
> > 
> > At first this seems like a good idea but the documentation is misleading, and
> > it actually causes two problems:
> > 
> > 1) Unnecessary compat code: If you've built a kernel and glibc together
> > (presumably this is common when using buildroot) then you have no need of any
> > compat code because the only kernel you'll be using is the one you just built.
> > According to the doc (above) this causes glibc to include unnecessary compat
> > code which slows it down.
> > 
> > 2) Availability of new features: It appears from the code that the
> > documentation leaves out some important information. What actually happens
> > while building glibc is that the "minimum kernel version" (which is either the
> > version given by "--enable-kernel=" or the oldest possible version for that
> > architecture) sets both __LINUX_KERNEL_VERSION and __ABI_TAG_VERSION so that
> > glibc will never use any kernel features *newer* than this version. Therefore
> > you cannot get access to recent kernel features *at all* via glibc (without
> > manually altering glibc.mk).
> 
> Thanks a lot for spotting this problem and for the explanation,
> definitely makes sense.
> 
> > (An example of a recent feature would be the sendmmsg syscall, which
> > requires a kernel version of 3.0.0 but the minimum version for PowerPC 64 is
> > 2.6.24.)
> 
> Ah, so you're working on PowerPC 64, great! We have a toolchain build
> problem that I reported on Power PC 64, when building for Power 8:
> 
>   http://lists.busybox.net/pipermail/buildroot/2016-January/150338.html
> 
> If you have some time to look at it, it would be great.

No problem, I've replied in that thread :-)

> > I'd like to prepare a patch to make this configurable, and it obviously
> > wouldn't be difficult to add but I'm not sure of the best way to do it.
> > 
> > The most obvious way seems to be to add a menu item to the Toolchain menu, just
> > after "glibc version" (and only enabled when glibc is the C library), called
> > "glibc kernel API version" with a default value of "same as kernel headers".
> > The other options would be "as old as possible" or a specific version.
> > 
> > Comments?
> 
> I don't think you need to add additional options. We should just use
> the kernel version of the chosen kernel headers, since we're guaranteed
> that the running kernel will be at least more recent than the kernel
> headers being used.
> 
> I.e, can you try something such as:
> 
> 	--enable-kernel=$(call qstrip,$(BR2_TOOLCHAIN_HEADERS_AT_LEAST))
> 
> BR2_TOOLCHAIN_HEADERS_AT_LEAST will be 3.0 if you use some 3.0.x kernel
> headers for your toolchain.
> 
> Thanks!

Ok that is certainly easier and works for me, so I'll post a patch :-)

I'm curious tho: when I hacked up some test code I used
BR2_DEFAULT_KERNEL_HEADERS (which seems to have the same value) but I must
confess I don't understand the implications of choosing between
BR2_DEFAULT_KERNEL_HEADERS or BR2_TOOLCHAIN_HEADERS_AT_LEAST. Could you explain
the difference so I don't have to read so much Make code? ;-)

Cheers,
Sam.

[Buildroot] glibc and --enable-kernel

[Thomas Petazzoni]

Sam,

On Thu, 28 Jan 2016 13:55:37 +1100, Sam Bobroff wrote:

> > Ah, so you're working on PowerPC 64, great! We have a toolchain build
> > problem that I reported on Power PC 64, when building for Power 8:
> > 
> >   http://lists.busybox.net/pipermail/buildroot/2016-January/150338.html
> > 
> > If you have some time to look at it, it would be great.
> 
> No problem, I've replied in that thread :-)

Seen that, thanks!

> Ok that is certainly easier and works for me, so I'll post a patch :-)
> 
> I'm curious tho: when I hacked up some test code I used
> BR2_DEFAULT_KERNEL_HEADERS (which seems to have the same value) but I must
> confess I don't understand the implications of choosing between
> BR2_DEFAULT_KERNEL_HEADERS or BR2_TOOLCHAIN_HEADERS_AT_LEAST. Could you explain
> the difference so I don't have to read so much Make code? ;-)

There would be no real difference between the two for your use case.

BR2_DEFAULT_KERNEL_HEADERS is a string that contains the version number
of the Linux kernel sources chosen in the linux-headers package. It
would contain things like 3.2.76, 4.3.4, etc. or a user-specified
kernel version. It would have worked all fine for your case.

BR2_TOOLCHAIN_HEADERS_AT_LEAST is a string that exists for both the
internal and external toolchains, and which indicates the "series" of
the kernel headers (i.e just 3.2, 3.4, 4.0, etc.).

The only reason that may encourage you to use
BR2_TOOLCHAIN_HEADERS_AT_LEAST instead of BR2_DEFAULT_KERNEL_HEADERS is
an upcoming patch from Yann E. Morin (already submitted but not merged
yet), which allows to tell Buildroot to use the kernel version
specified in the "Kernel" menu as the version for the kernel headers.
In this case, I believe BR2_DEFAULT_KERNEL_HEADERS will not be correct,
while BR2_TOOLCHAIN_HEADERS_AT_LEAST will be.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] glibc and --enable-kernel

[Arnout Vandecappelle]

On 28-01-16 09:36, Thomas Petazzoni wrote:
> Sam,
> 
> On Thu, 28 Jan 2016 13:55:37 +1100, Sam Bobroff wrote:
[snip]
>> Ok that is certainly easier and works for me, so I'll post a patch :-)
>>
>> I'm curious tho: when I hacked up some test code I used
>> BR2_DEFAULT_KERNEL_HEADERS (which seems to have the same value) but I must
>> confess I don't understand the implications of choosing between
>> BR2_DEFAULT_KERNEL_HEADERS or BR2_TOOLCHAIN_HEADERS_AT_LEAST. Could you explain
>> the difference so I don't have to read so much Make code? ;-)
> 
> There would be no real difference between the two for your use case.
> 
> BR2_DEFAULT_KERNEL_HEADERS is a string that contains the version number
> of the Linux kernel sources chosen in the linux-headers package. It
> would contain things like 3.2.76, 4.3.4, etc. or a user-specified
> kernel version. It would have worked all fine for your case.
> 
> BR2_TOOLCHAIN_HEADERS_AT_LEAST is a string that exists for both the
> internal and external toolchains, and which indicates the "series" of
> the kernel headers (i.e just 3.2, 3.4, 4.0, etc.).
> 
> The only reason that may encourage you to use
> BR2_TOOLCHAIN_HEADERS_AT_LEAST instead of BR2_DEFAULT_KERNEL_HEADERS is
> an upcoming patch from Yann E. Morin (already submitted but not merged
> yet), which allows to tell Buildroot to use the kernel version
> specified in the "Kernel" menu as the version for the kernel headers.
> In this case, I believe BR2_DEFAULT_KERNEL_HEADERS will not be correct,
> while BR2_TOOLCHAIN_HEADERS_AT_LEAST will be.

 A minor issue with _AT_LEAST, however, is that it will probably go through
deprecation eventually. Right now we have "2.6" as the lowest _AT_LEAST - how
will glibc deal with that? I can imagine at some point we'll deprecate 3,1, 3.2,
and 3.3, so if you're using 3.3 headers the _AT_LEAST will become 3.0. Right
now, we already have that for 2.6.3x, which will fall back to 2.6.0 (I guess -
should be checked if glibc doesn't barf on 2.6 without .0).

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] glibc and --enable-kernel

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >> The only reason that may encourage you to use
 >> BR2_TOOLCHAIN_HEADERS_AT_LEAST instead of BR2_DEFAULT_KERNEL_HEADERS is
 >> an upcoming patch from Yann E. Morin (already submitted but not merged
 >> yet), which allows to tell Buildroot to use the kernel version
 >> specified in the "Kernel" menu as the version for the kernel headers.
 >> In this case, I believe BR2_DEFAULT_KERNEL_HEADERS will not be correct,
 >> while BR2_TOOLCHAIN_HEADERS_AT_LEAST will be.

 >  A minor issue with _AT_LEAST, however, is that it will probably go through
 > deprecation eventually. Right now we have "2.6" as the lowest _AT_LEAST - how
 > will glibc deal with that? I can imagine at some point we'll deprecate 3,1, 3.2,
 > and 3.3, so if you're using 3.3 headers the _AT_LEAST will become 3.0. Right
 > now, we already have that for 2.6.3x, which will fall back to 2.6.0 (I guess -
 > should be checked if glibc doesn't barf on 2.6 without .0).

If it does, then I suggest we filter out --enable-kernel for the oldest
variant, so we're back to how it works now as a lowest common denominator.

-- 
Bye, Peter Korsgaard

[Buildroot] glibc and --enable-kernel

[Mike Frysinger]

On 27 Jan 2016 15:21, Sam Bobroff wrote:
> 1) Unnecessary compat code: If you've built a kernel and glibc together
> (presumably this is common when using buildroot) then you have no need of any
> compat code because the only kernel you'll be using is the one you just built.
> According to the doc (above) this causes glibc to include unnecessary compat
> code which slows it down.

the overhead is not that great.  glibc is written such that it tries the
latest feature first, and if it works, it's done.  the overhead is really
just the extra return value checking, and that only applies to some cases.
i suspect profiling of pretty much every program out there wouldn't even
register this overhead.

> 2) Availability of new features: It appears from the code that the
> documentation leaves out some important information. What actually happens
> while building glibc is that the "minimum kernel version" (which is either the
> version given by "--enable-kernel=" or the oldest possible version for that
> architecture) sets both __LINUX_KERNEL_VERSION and __ABI_TAG_VERSION so that
> glibc will never use any kernel features *newer* than this version. Therefore
> you cannot get access to recent kernel features *at all* via glibc (without
> manually altering glibc.mk).

this is dangerously incorrect FUD.  i don't know why you think this, but
it is *not* correct.  when a feature requires a newer kernel version than
the minimum set, it will first attempt to use the newer feature, and if
the kernel returns an error (e.g. ENOSYS), it will cache the result and
transparently fall back to the old code.

for example, from the glibc-2.22 release:
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/wordsize-64/posix_fallocate.c;h=8ae8a2927d174a70b7de724dc82f5c040174050b

if the min kernel version is high enough, __ASSUME_FALLOCATE is set to 1,
so the code ends up being the fallocate syscall directly.  but if it is
not, the syscall is tried, and if it fails w/ENOSYS, we cache the result
in __have_fallocate before falling back to older methods.  all future
calls will then see that cached value and immediately fallback.

> (An example of a recent feature would be the sendmmsg syscall, which
> requires a kernel version of 3.0.0 but the minimum version for PowerPC 64 is
> 2.6.24.)

i don't know where you're getting this information either.  starting with
glibc-2.20, the min required kernel version has been 2.6.32 for everyone.

the sendmmsg wrapper is tricky due to having to handle cancellation end
points.  so in this case, glibc will avoid using the syscall directly
unless it can assume it exists.
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/sendmmsg.c;h=67b7ca9bfc9d63320fde1cceb95f81336a340f33

so in the ppc case, it will use socketcall(sendmmsg) when the min kernel
is below 3.0.  otherwise it uses sendmmsg directly.  in this particular
case, we haven't found the runtime overhead significant relative to the
source code overhead of having to deal with pthread cancellation.  this
is another case i doubt you can get to show up in profiling.

> I'd like to prepare a patch to make this configurable, and it obviously
> wouldn't be difficult to add but I'm not sure of the best way to do it.

all that said, having buildroot provide an option to set the min kernel
version with the default matching the selected kernel should be fine.
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160202/ca69dc10/attachment.asc>

[Buildroot] glibc and --enable-kernel

[Sam Bobroff]

On Tue, Feb 02, 2016 at 11:56:49AM -0500, Mike Frysinger wrote:
> On 27 Jan 2016 15:21, Sam Bobroff wrote:
> > 1) Unnecessary compat code: If you've built a kernel and glibc together
> > (presumably this is common when using buildroot) then you have no need of any
> > compat code because the only kernel you'll be using is the one you just built.
> > According to the doc (above) this causes glibc to include unnecessary compat
> > code which slows it down.
> 
> the overhead is not that great.  glibc is written such that it tries the
> latest feature first, and if it works, it's done.  the overhead is really
> just the extra return value checking, and that only applies to some cases.
> i suspect profiling of pretty much every program out there wouldn't even
> register this overhead.

No argument here.

> > 2) Availability of new features: It appears from the code that the
> > documentation leaves out some important information. What actually happens
> > while building glibc is that the "minimum kernel version" (which is either the
> > version given by "--enable-kernel=" or the oldest possible version for that
> > architecture) sets both __LINUX_KERNEL_VERSION and __ABI_TAG_VERSION so that
> > glibc will never use any kernel features *newer* than this version. Therefore
> > you cannot get access to recent kernel features *at all* via glibc (without
> > manually altering glibc.mk).
> 
> this is dangerously incorrect FUD.  i don't know why you think this, but
> it is *not* correct.  when a feature requires a newer kernel version than
> the minimum set, it will first attempt to use the newer feature, and if
> the kernel returns an error (e.g. ENOSYS), it will cache the result and
> transparently fall back to the old code.

I certainly saw glibc calling an older interface when configured with older
--enable-kernel versions but I think you've explained what was going on below.
I just happened to look at one of the (few) cases where it acts that way. It's
good to know (and not surprising) that in general this isn't the case.

> for example, from the glibc-2.22 release:
> https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/wordsize-64/posix_fallocate.c;h=8ae8a2927d174a70b7de724dc82f5c040174050b
> 
> if the min kernel version is high enough, __ASSUME_FALLOCATE is set to 1,
> so the code ends up being the fallocate syscall directly.  but if it is
> not, the syscall is tried, and if it fails w/ENOSYS, we cache the result
> in __have_fallocate before falling back to older methods.  all future
> calls will then see that cached value and immediately fallback.
> 
> > (An example of a recent feature would be the sendmmsg syscall, which
> > requires a kernel version of 3.0.0 but the minimum version for PowerPC 64 is
> > 2.6.24.)
> 
> i don't know where you're getting this information either.  starting with
> glibc-2.20, the min required kernel version has been 2.6.32 for everyone.

From sysdeps/unix/sysv/linux/configure.ac, I just mis-remembered it :-P

> the sendmmsg wrapper is tricky due to having to handle cancellation end
> points.  so in this case, glibc will avoid using the syscall directly
> unless it can assume it exists.
> https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/sendmmsg.c;h=67b7ca9bfc9d63320fde1cceb95f81336a340f33
> 
> so in the ppc case, it will use socketcall(sendmmsg) when the min kernel
> is below 3.0.  otherwise it uses sendmmsg directly.  in this particular
> case, we haven't found the runtime overhead significant relative to the
> source code overhead of having to deal with pthread cancellation.  this
> is another case i doubt you can get to show up in profiling.

Thanks, that explains it.

> > I'd like to prepare a patch to make this configurable, and it obviously
> > wouldn't be difficult to add but I'm not sure of the best way to do it.
> 
> all that said, having buildroot provide an option to set the min kernel
> version with the default matching the selected kernel should be fine.
> -mike

Great :-)

Also thanks a lot for taking the time to explain/debunk the above! :-)

One last thing, where *is* a good place to get information like this?

Cheers,
Sam.