[Buildroot] [RFC PATCH] host-localedef: Compile against glibc

[Buildroot] [RFC PATCH] host-localedef: Compile against glibc

[Samuel Mendoza-Jonas]

In glibc 2.27 the following change occured:
"Statically compiled applications attempting to load locales compiled
for the GNU C Library version 2.27 will fail and fall back to the
builtin C/POSIX locale."

This impacts us since upstream buildroot uses a localdef built against
an older eglibc release [0].
Update the host-localedef package to build against glibc 2.27.
Unfortunately we can't compile just the localedef utility, but even so
this does not increase the build time too significantly. eg:

With glibc localedef:
real    33m11.278s
user    171m51.684s
sys     19m46.528s

With eglibc localedef:
real    31m12.715s
user    155m13.616s
sys     16m23.504s

[0] https://bugs.busybox.net/show_bug.cgi?id=11096

Signed-off-by: Samuel Mendoza-Jonas <sam@mendozajonas.com>
---
This is a patch we're using in our own Buildroot branch to keep us going
until a solution is found for the above bug report. While it does add a
few minutes to our example build it avoids hacking glibc to only build
localedef which is a solution some other users have gone for.
Definitely an RFC as I'm sure some of the configuration in localedef.mk
could be trimmed down but I took the safer bet of basing it off the
existing glibc.mk.


 .../0001-Don-t-include-xlocale.h.patch        | 50 -----------
 package/localedef/localedef.hash              |  2 +-
 package/localedef/localedef.mk                | 87 +++++++++++++++++--
 3 files changed, 79 insertions(+), 60 deletions(-)
 delete mode 100644 package/localedef/0001-Don-t-include-xlocale.h.patch

diff --git a/package/localedef/0001-Don-t-include-xlocale.h.patch b/package/localedef/0001-Don-t-include-xlocale.h.patch
deleted file mode 100644
index 9a6c2f8e90..0000000000
--- a/package/localedef/0001-Don-t-include-xlocale.h.patch
+++ /dev/null
@@ -1,50 +0,0 @@
-From: Bernhard Walle <bernhard@bwalle.de>
-Date: Mon, 2 Oct 2017 16:55:23 +0200
-Subject: [PATCH] Don't include <xlocale.h>
-
-This header has been removed in glibc 2.26:
-
-https://abi-laboratory.pro/tracker/changelog/glibc/2.26/log.html
-
------------------------- >8 ------------------------
-* The nonstandard header <xlocale.h> has been removed.  Most programs should
-  use <locale.h> instead.  If you have a specific need for the definition of
-  locale_t with no other declarations, please contact
-  libc-alpha at sourceware.org and explain.
------------------------- 8< ------------------------
-
-Signed-off-by: Bernhard Walle <bernhard@bwalle.de>
-
-Upstream: https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-eglibc-2.14.1-r17443-ptx1/0003-Don-t-include-xlocale.h.patch
-Signed-off-by: Peter Seiderer <ps.report@gmx.net>
----
- eglibc/locale/langinfo.h | 2 +-
- eglibc/locale/locale.h   | 2 +-
- 2 files changed, 2 insertions(+), 2 deletions(-)
-
-diff --git a/eglibc/locale/langinfo.h b/eglibc/locale/langinfo.h
-index 0a5336507196..76707a4584c1 100644
---- a/eglibc/locale/langinfo.h
-+++ b/eglibc/locale/langinfo.h
-@@ -589,7 +589,7 @@ extern char *nl_langinfo (nl_item __item) __THROW;
-    more information.  */
- 
- /* Get locale datatype definition.  */
--# include <xlocale.h>
-+# include <locale.h>
- 
- /* Just like nl_langinfo but get the information from the locale object L.  */
- extern char *nl_langinfo_l (nl_item __item, __locale_t __l);
-diff --git a/eglibc/locale/locale.h b/eglibc/locale/locale.h
-index 2aa19e76acb2..18be711a2d40 100644
---- a/eglibc/locale/locale.h
-+++ b/eglibc/locale/locale.h
-@@ -143,7 +143,7 @@ __END_NAMESPACE_STD
-    This is a proof-of-concept implementation.  */
- 
- /* Get locale datatype definition.  */
--# include <xlocale.h>
-+# include_next <locale.h>
- 
- /* Return a reference to a data structure representing a set of locale
-    datasets.  Unlike for the CATEGORY parameter for `setlocale' the
diff --git a/package/localedef/localedef.hash b/package/localedef/localedef.hash
index ee14fb16b8..6f2d6313b6 100644
--- a/package/localedef/localedef.hash
+++ b/package/localedef/localedef.hash
@@ -1,2 +1,2 @@
 # Locally calculated
-sha256 9a60f7cdab6fb39adf23a12102f2d950d5f07f0cd7e51e85ec327e07440a79c6  localedef-eglibc-2.14.1-r17443-ptx1.tar.bz2
+sha256 33189b3f10c88730a1f686fac794bc01f31765f12ffd75bc5e8a0f2a690d217a  localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e.tar.gz
diff --git a/package/localedef/localedef.mk b/package/localedef/localedef.mk
index 11d9ba3848..c93132cc3f 100644
--- a/package/localedef/localedef.mk
+++ b/package/localedef/localedef.mk
@@ -4,19 +4,88 @@
 #
 ################################################################################
 
-LOCALEDEF_VERSION = 2.14.1-r17443-ptx1
-LOCALEDEF_SOURCE = localedef-eglibc-$(LOCALEDEF_VERSION).tar.bz2
-LOCALEDEF_SITE = http://www.pengutronix.de/software/ptxdist/temporary-src
+# Use the same VERSION and SITE as target glibc
+LOCALEDEF_VERSION = glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e
+LOCALEDEF_SITE = $(call github,bminor,glibc,$(GLIBC_VERSION))
 
-HOST_LOCALEDEF_CONF_OPTS += \
-	--prefix=/usr \
-	--with-glibc=./eglibc
+HOST_LOCALEDEF_LICENSE = GPL-2.0+ (programs), LGPL-2.1+, BSD-3-Clause, MIT (library)
+HOST_LOCALEDEF_LICENSE_FILES = COPYING COPYING.LIB LICENSES
 
-HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline"
+# glibc is part of the toolchain so disable the toolchain dependency
+HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO
 
-# The makefile does not implement an install target
+HOST_LOCALEDEF_SUBDIR = build
+
+# Thumb build is broken, build in ARM mode
+ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -marm
+endif
+
+# MIPS64 defaults to n32 so pass the correct -mabi if
+# we are using a different ABI. OABI32 is also used
+# in MIPS so we pass -mabi=32 in this case as well
+# even though it's not strictly necessary.
+ifeq ($(BR2_MIPS_NABI64),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64
+else ifeq ($(BR2_MIPS_OABI32),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32
+endif
+
+ifeq ($(BR2_ENABLE_DEBUG),y)
+HOST_LOCALEDEF_EXTRA_CFLAGS += -g
+endif
+
+# The stubs.h header is not installed by install-headers, but is
+# needed for the gcc build. An empty stubs.h will work, as explained
+# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
+# is used by Crosstool-NG.
+ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y)
+define HOST_LOCALEDEF_ADD_MISSING_STUB_H
+	mkdir -p $(STAGING_DIR)/usr/include/gnu
+	touch $(STAGING_DIR)/usr/include/gnu/stubs.h
+endef
+endif
+
+# Even though we use the autotools-package infrastructure, we have to
+# override the default configure commands for several reasons:
+#
+#  1. We have to build out-of-tree, but we can't use the same
+#     'symbolic link to configure' used with the gcc packages.
+#
+#  2. We have to execute the configure script with bash and not sh.
+#
+# Note that as mentionned in
+# http://patches.openembedded.org/patch/38849/, glibc must be
+# built with -O2, so we pass our own CFLAGS and CXXFLAGS below.
+define HOST_LOCALEDEF_CONFIGURE_CMDS
+	mkdir -p $(@D)/build
+	# Do the configuration
+	(cd $(@D)/build; \
+		$(HOST_CONFIGURE_OPTS) \
+		CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \
+		CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \
+		$(SHELL) $(@D)/configure \
+		ac_cv_path_BASH_SHELL=/bin/bash \
+		libc_cv_forced_unwind=yes \
+		libc_cv_ssp=no \
+		--target=$(GNU_HOST_NAME) \
+		--host=$(GNU_HOST_NAME) \
+		--build=$(GNU_HOST_NAME) \
+		--prefix=/usr \
+		$(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \
+		$(if $(BR2_x86_64),--enable-lock-elision) \
+		--with-pkgversion="Buildroot" \
+		--without-cvs \
+		--disable-profile \
+		--without-gd \
+		--enable-obsolete-rpc)
+	$(HOST_LOCALEDEF_ADD_MISSING_STUB_H)
+endef
+
+$(eval $(autotools-package))
+# The makefile does not implement an install target for localedef
 define HOST_LOCALEDEF_INSTALL_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/localedef $(HOST_DIR)/bin/localedef
+	$(INSTALL) -D -m 0755 $(@D)/build/locale/localedef $(HOST_DIR)/bin/localedef
 endef
 
 $(eval $(host-autotools-package))
-- 
2.18.0

[Buildroot] [RFC PATCH] host-localedef: Compile against glibc

[Joel Stanley]

On 30 July 2018 at 15:09, Samuel Mendoza-Jonas <sam@mendozajonas.com> wrote:
> In glibc 2.27 the following change occured:
> "Statically compiled applications attempting to load locales compiled
> for the GNU C Library version 2.27 will fail and fall back to the
> builtin C/POSIX locale."
>
> This impacts us since upstream buildroot uses a localdef built against
> an older eglibc release [0].
> Update the host-localedef package to build against glibc 2.27.
> Unfortunately we can't compile just the localedef utility, but even so
> this does not increase the build time too significantly. eg:
>
> With glibc localedef:
> real    33m11.278s
> user    171m51.684s
> sys     19m46.528s
>
> With eglibc localedef:
> real    31m12.715s
> user    155m13.616s
> sys     16m23.504s
>
> [0] https://bugs.busybox.net/show_bug.cgi?id=11096
>
> Signed-off-by: Samuel Mendoza-Jonas <sam@mendozajonas.com>
> ---
> This is a patch we're using in our own Buildroot branch to keep us going
> until a solution is found for the above bug report. While it does add a
> few minutes to our example build it avoids hacking glibc to only build
> localedef which is a solution some other users have gone for.
> Definitely an RFC as I'm sure some of the configuration in localedef.mk
> could be trimmed down but I took the safer bet of basing it off the
> existing glibc.mk.

Thanks for doing this and submitting it upstream Sam. I think this is
idea preferable to the other solution we have, as it is easier to
maintain. The implementation might be able to be tweaked a little, but
I will defer to our buildroot maintainers on that.

Cheers,

Joel

>
>
>  .../0001-Don-t-include-xlocale.h.patch        | 50 -----------
>  package/localedef/localedef.hash              |  2 +-
>  package/localedef/localedef.mk                | 87 +++++++++++++++++--
>  3 files changed, 79 insertions(+), 60 deletions(-)
>  delete mode 100644 package/localedef/0001-Don-t-include-xlocale.h.patch
>
> diff --git a/package/localedef/0001-Don-t-include-xlocale.h.patch b/package/localedef/0001-Don-t-include-xlocale.h.patch
> deleted file mode 100644
> index 9a6c2f8e90..0000000000
> --- a/package/localedef/0001-Don-t-include-xlocale.h.patch
> +++ /dev/null
> @@ -1,50 +0,0 @@
> -From: Bernhard Walle <bernhard@bwalle.de>
> -Date: Mon, 2 Oct 2017 16:55:23 +0200
> -Subject: [PATCH] Don't include <xlocale.h>
> -
> -This header has been removed in glibc 2.26:
> -
> -https://abi-laboratory.pro/tracker/changelog/glibc/2.26/log.html
> -
> ------------------------- >8 ------------------------
> -* The nonstandard header <xlocale.h> has been removed.  Most programs should
> -  use <locale.h> instead.  If you have a specific need for the definition of
> -  locale_t with no other declarations, please contact
> -  libc-alpha at sourceware.org and explain.
> ------------------------- 8< ------------------------
> -
> -Signed-off-by: Bernhard Walle <bernhard@bwalle.de>
> -
> -Upstream: https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-eglibc-2.14.1-r17443-ptx1/0003-Don-t-include-xlocale.h.patch
> -Signed-off-by: Peter Seiderer <ps.report@gmx.net>
> ----
> - eglibc/locale/langinfo.h | 2 +-
> - eglibc/locale/locale.h   | 2 +-
> - 2 files changed, 2 insertions(+), 2 deletions(-)
> -
> -diff --git a/eglibc/locale/langinfo.h b/eglibc/locale/langinfo.h
> -index 0a5336507196..76707a4584c1 100644
> ---- a/eglibc/locale/langinfo.h
> -+++ b/eglibc/locale/langinfo.h
> -@@ -589,7 +589,7 @@ extern char *nl_langinfo (nl_item __item) __THROW;
> -    more information.  */
> -
> - /* Get locale datatype definition.  */
> --# include <xlocale.h>
> -+# include <locale.h>
> -
> - /* Just like nl_langinfo but get the information from the locale object L.  */
> - extern char *nl_langinfo_l (nl_item __item, __locale_t __l);
> -diff --git a/eglibc/locale/locale.h b/eglibc/locale/locale.h
> -index 2aa19e76acb2..18be711a2d40 100644
> ---- a/eglibc/locale/locale.h
> -+++ b/eglibc/locale/locale.h
> -@@ -143,7 +143,7 @@ __END_NAMESPACE_STD
> -    This is a proof-of-concept implementation.  */
> -
> - /* Get locale datatype definition.  */
> --# include <xlocale.h>
> -+# include_next <locale.h>
> -
> - /* Return a reference to a data structure representing a set of locale
> -    datasets.  Unlike for the CATEGORY parameter for `setlocale' the
> diff --git a/package/localedef/localedef.hash b/package/localedef/localedef.hash
> index ee14fb16b8..6f2d6313b6 100644
> --- a/package/localedef/localedef.hash
> +++ b/package/localedef/localedef.hash
> @@ -1,2 +1,2 @@
>  # Locally calculated
> -sha256 9a60f7cdab6fb39adf23a12102f2d950d5f07f0cd7e51e85ec327e07440a79c6  localedef-eglibc-2.14.1-r17443-ptx1.tar.bz2
> +sha256 33189b3f10c88730a1f686fac794bc01f31765f12ffd75bc5e8a0f2a690d217a  localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e.tar.gz
> diff --git a/package/localedef/localedef.mk b/package/localedef/localedef.mk
> index 11d9ba3848..c93132cc3f 100644
> --- a/package/localedef/localedef.mk
> +++ b/package/localedef/localedef.mk
> @@ -4,19 +4,88 @@
>  #
>  ################################################################################
>
> -LOCALEDEF_VERSION = 2.14.1-r17443-ptx1
> -LOCALEDEF_SOURCE = localedef-eglibc-$(LOCALEDEF_VERSION).tar.bz2
> -LOCALEDEF_SITE = http://www.pengutronix.de/software/ptxdist/temporary-src
> +# Use the same VERSION and SITE as target glibc
> +LOCALEDEF_VERSION = glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e
> +LOCALEDEF_SITE = $(call github,bminor,glibc,$(GLIBC_VERSION))
>
> -HOST_LOCALEDEF_CONF_OPTS += \
> -       --prefix=/usr \
> -       --with-glibc=./eglibc
> +HOST_LOCALEDEF_LICENSE = GPL-2.0+ (programs), LGPL-2.1+, BSD-3-Clause, MIT (library)
> +HOST_LOCALEDEF_LICENSE_FILES = COPYING COPYING.LIB LICENSES
>
> -HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline"
> +# glibc is part of the toolchain so disable the toolchain dependency
> +HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO
>
> -# The makefile does not implement an install target
> +HOST_LOCALEDEF_SUBDIR = build
> +
> +# Thumb build is broken, build in ARM mode
> +ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y)
> +HOST_LOCALEDEF_EXTRA_CFLAGS += -marm
> +endif
> +
> +# MIPS64 defaults to n32 so pass the correct -mabi if
> +# we are using a different ABI. OABI32 is also used
> +# in MIPS so we pass -mabi=32 in this case as well
> +# even though it's not strictly necessary.
> +ifeq ($(BR2_MIPS_NABI64),y)
> +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64
> +else ifeq ($(BR2_MIPS_OABI32),y)
> +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32
> +endif
> +
> +ifeq ($(BR2_ENABLE_DEBUG),y)
> +HOST_LOCALEDEF_EXTRA_CFLAGS += -g
> +endif
> +
> +# The stubs.h header is not installed by install-headers, but is
> +# needed for the gcc build. An empty stubs.h will work, as explained
> +# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
> +# is used by Crosstool-NG.
> +ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y)
> +define HOST_LOCALEDEF_ADD_MISSING_STUB_H
> +       mkdir -p $(STAGING_DIR)/usr/include/gnu
> +       touch $(STAGING_DIR)/usr/include/gnu/stubs.h
> +endef
> +endif
> +
> +# Even though we use the autotools-package infrastructure, we have to
> +# override the default configure commands for several reasons:
> +#
> +#  1. We have to build out-of-tree, but we can't use the same
> +#     'symbolic link to configure' used with the gcc packages.
> +#
> +#  2. We have to execute the configure script with bash and not sh.
> +#
> +# Note that as mentionned in
> +# http://patches.openembedded.org/patch/38849/, glibc must be
> +# built with -O2, so we pass our own CFLAGS and CXXFLAGS below.
> +define HOST_LOCALEDEF_CONFIGURE_CMDS
> +       mkdir -p $(@D)/build
> +       # Do the configuration
> +       (cd $(@D)/build; \
> +               $(HOST_CONFIGURE_OPTS) \
> +               CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \
> +               CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \
> +               $(SHELL) $(@D)/configure \
> +               ac_cv_path_BASH_SHELL=/bin/bash \
> +               libc_cv_forced_unwind=yes \
> +               libc_cv_ssp=no \
> +               --target=$(GNU_HOST_NAME) \
> +               --host=$(GNU_HOST_NAME) \
> +               --build=$(GNU_HOST_NAME) \
> +               --prefix=/usr \
> +               $(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \
> +               $(if $(BR2_x86_64),--enable-lock-elision) \
> +               --with-pkgversion="Buildroot" \
> +               --without-cvs \
> +               --disable-profile \
> +               --without-gd \
> +               --enable-obsolete-rpc)
> +       $(HOST_LOCALEDEF_ADD_MISSING_STUB_H)
> +endef
> +
> +$(eval $(autotools-package))
> +# The makefile does not implement an install target for localedef
>  define HOST_LOCALEDEF_INSTALL_CMDS
> -       $(INSTALL) -D -m 0755 $(@D)/localedef $(HOST_DIR)/bin/localedef
> +       $(INSTALL) -D -m 0755 $(@D)/build/locale/localedef $(HOST_DIR)/bin/localedef
>  endef
>
>  $(eval $(host-autotools-package))
> --
> 2.18.0
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [RFC PATCH] host-localedef: Compile against glibc

[Thomas Petazzoni]

Hello Samuel,

Thanks for taking a look into this!

On Mon, 30 Jul 2018 15:39:36 +1000, Samuel Mendoza-Jonas wrote:
> In glibc 2.27 the following change occured:
> "Statically compiled applications attempting to load locales compiled
> for the GNU C Library version 2.27 will fail and fall back to the
> builtin C/POSIX locale."

I don't really understand this explanation. Isn't the problem that
glibc 2.27 doesn't read properly locale files generated by localedef
from glibc 2.24 ?

I kind of fail to parse the explanation given here :-/

> This impacts us since upstream buildroot uses a localdef built against
> an older eglibc release [0].
> Update the host-localedef package to build against glibc 2.27.
> Unfortunately we can't compile just the localedef utility, but even so
> this does not increase the build time too significantly. eg:
> 
> With glibc localedef:
> real    33m11.278s
> user    171m51.684s
> sys     19m46.528s
> 
> With eglibc localedef:
> real    31m12.715s
> user    155m13.616s
> sys     16m23.504s

Meh, two more minutes just to build some not so complicated user-space
program :-/

> -HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline"
> +# glibc is part of the toolchain so disable the toolchain dependency
> +HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO

Not needed for host-localedef.

> -# The makefile does not implement an install target
> +HOST_LOCALEDEF_SUBDIR = build
> +
> +# Thumb build is broken, build in ARM mode
> +ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y)

This is a target related option...

> +HOST_LOCALEDEF_EXTRA_CFLAGS += -marm

... so it is not relevant to observe it when building a host program.

> +endif
> +
> +# MIPS64 defaults to n32 so pass the correct -mabi if
> +# we are using a different ABI. OABI32 is also used
> +# in MIPS so we pass -mabi=32 in this case as well
> +# even though it's not strictly necessary.
> +ifeq ($(BR2_MIPS_NABI64),y)
> +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64
> +else ifeq ($(BR2_MIPS_OABI32),y)
> +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32
> +endif

Same comment.

> +ifeq ($(BR2_ENABLE_DEBUG),y)
> +HOST_LOCALEDEF_EXTRA_CFLAGS += -g
> +endif

Same comment.

> +
> +# The stubs.h header is not installed by install-headers, but is
> +# needed for the gcc build. An empty stubs.h will work, as explained
> +# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
> +# is used by Crosstool-NG.
> +ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y)
> +define HOST_LOCALEDEF_ADD_MISSING_STUB_H
> +	mkdir -p $(STAGING_DIR)/usr/include/gnu
> +	touch $(STAGING_DIR)/usr/include/gnu/stubs.h

STAGING_DIR is observed when building target packages, not host
packages, so this seems useless.

> +endef
> +endif
> +
> +# Even though we use the autotools-package infrastructure, we have to
> +# override the default configure commands for several reasons:
> +#
> +#  1. We have to build out-of-tree, but we can't use the same
> +#     'symbolic link to configure' used with the gcc packages.
> +#
> +#  2. We have to execute the configure script with bash and not sh.
> +#
> +# Note that as mentionned in
> +# http://patches.openembedded.org/patch/38849/, glibc must be
> +# built with -O2, so we pass our own CFLAGS and CXXFLAGS below.
> +define HOST_LOCALEDEF_CONFIGURE_CMDS
> +	mkdir -p $(@D)/build
> +	# Do the configuration
> +	(cd $(@D)/build; \
> +		$(HOST_CONFIGURE_OPTS) \
> +		CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \
> +		CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \
> +		$(SHELL) $(@D)/configure \
> +		ac_cv_path_BASH_SHELL=/bin/bash \
> +		libc_cv_forced_unwind=yes \
> +		libc_cv_ssp=no \
> +		--target=$(GNU_HOST_NAME) \
> +		--host=$(GNU_HOST_NAME) \
> +		--build=$(GNU_HOST_NAME) \
> +		--prefix=/usr \

Should be --prefix=$(HOST_DIR)

> +		$(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \
> +		$(if $(BR2_x86_64),--enable-lock-elision) \

Both BR2_SOFT_FLOAT and BR2_x86_64 are target options, so observing
them when building a host package is almost always wrong.

> +		--with-pkgversion="Buildroot" \
> +		--without-cvs \
> +		--disable-profile \
> +		--without-gd \
> +		--enable-obsolete-rpc)
> +	$(HOST_LOCALEDEF_ADD_MISSING_STUB_H)
> +endef

With all this, aren't we building a full glibc for the host ?

Can you use the locale/others and locale/install-others make targets
instead ?

See how they are doing it in PTXdist:
https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make.
Most likely using the locale/others make target will help cut the build
time significantly.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [RFC PATCH] host-localedef: Compile against glibc

[Samuel Mendoza-Jonas]

On Tue, 2018-07-31 at 22:55 +0200, Thomas Petazzoni wrote:
> Hello Samuel,
> 
> Thanks for taking a look into this!
> 
> On Mon, 30 Jul 2018 15:39:36 +1000, Samuel Mendoza-Jonas wrote:
> > In glibc 2.27 the following change occured:
> > "Statically compiled applications attempting to load locales compiled
> > for the GNU C Library version 2.27 will fail and fall back to the
> > builtin C/POSIX locale."
> 
> I don't really understand this explanation. Isn't the problem that
> glibc 2.27 doesn't read properly locale files generated by localedef
> from glibc 2.24 ?
> 
> I kind of fail to parse the explanation given here :-/

I think it's a bit of both; 2.27 introduces new locale descriptions which
localedef can't handle, eg:

Generating locale ru_RU.UTF-8
/scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error
/scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error

And we fall back to the C locale at runtime (at least in my environment).
I'll stare at the glibc release notes / bug report a bit more and see if
I can make a more legible explanation.

> 
> > This impacts us since upstream buildroot uses a localdef built against
> > an older eglibc release [0].
> > Update the host-localedef package to build against glibc 2.27.
> > Unfortunately we can't compile just the localedef utility, but even so
> > this does not increase the build time too significantly. eg:
> > 
> > With glibc localedef:
> > real    33m11.278s
> > user    171m51.684s
> > sys     19m46.528s
> > 
> > With eglibc localedef:
> > real    31m12.715s
> > user    155m13.616s
> > sys     16m23.504s
> 
> Meh, two more minutes just to build some not so complicated user-space
> program :-/
> 
> > -HOST_LOCALEDEF_CONF_ENV = CFLAGS="$(HOST_CFLAGS) -fgnu89-inline"
> > +# glibc is part of the toolchain so disable the toolchain dependency
> > +HOST_LOCALEDEF_ADD_TOOLCHAIN_DEPENDENCY = NO
> 
> Not needed for host-localedef.
> 

Thanks for noting this and the others below; I was sure a lot of this was
target-only but I figured someone would be able to notice a few of them
better than me :)

> > -# The makefile does not implement an install target
> > +HOST_LOCALEDEF_SUBDIR = build
> > +
> > +# Thumb build is broken, build in ARM mode
> > +ifeq ($(BR2_ARM_INSTRUCTIONS_THUMB),y)
> 
> This is a target related option...
> 
> > +HOST_LOCALEDEF_EXTRA_CFLAGS += -marm
> 
> ... so it is not relevant to observe it when building a host program.
> 
> > +endif
> > +
> > +# MIPS64 defaults to n32 so pass the correct -mabi if
> > +# we are using a different ABI. OABI32 is also used
> > +# in MIPS so we pass -mabi=32 in this case as well
> > +# even though it's not strictly necessary.
> > +ifeq ($(BR2_MIPS_NABI64),y)
> > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=64
> > +else ifeq ($(BR2_MIPS_OABI32),y)
> > +HOST_LOCALEDEF_EXTRA_CFLAGS += -mabi=32
> > +endif
> 
> Same comment.
> 
> > +ifeq ($(BR2_ENABLE_DEBUG),y)
> > +HOST_LOCALEDEF_EXTRA_CFLAGS += -g
> > +endif
> 
> Same comment.
> 
> > +
> > +# The stubs.h header is not installed by install-headers, but is
> > +# needed for the gcc build. An empty stubs.h will work, as explained
> > +# in http://gcc.gnu.org/ml/gcc/2002-01/msg00900.html. The same trick
> > +# is used by Crosstool-NG.
> > +ifeq ($(BR2_TOOLCHAIN_BUILDROOT_HOST_LOCALEDEF),y)
> > +define HOST_LOCALEDEF_ADD_MISSING_STUB_H
> > +	mkdir -p $(STAGING_DIR)/usr/include/gnu
> > +	touch $(STAGING_DIR)/usr/include/gnu/stubs.h
> 
> STAGING_DIR is observed when building target packages, not host
> packages, so this seems useless.
> 
> > +endef
> > +endif
> > +
> > +# Even though we use the autotools-package infrastructure, we have to
> > +# override the default configure commands for several reasons:
> > +#
> > +#  1. We have to build out-of-tree, but we can't use the same
> > +#     'symbolic link to configure' used with the gcc packages.
> > +#
> > +#  2. We have to execute the configure script with bash and not sh.
> > +#
> > +# Note that as mentionned in
> > +# http://patches.openembedded.org/patch/38849/, glibc must be
> > +# built with -O2, so we pass our own CFLAGS and CXXFLAGS below.
> > +define HOST_LOCALEDEF_CONFIGURE_CMDS
> > +	mkdir -p $(@D)/build
> > +	# Do the configuration
> > +	(cd $(@D)/build; \
> > +		$(HOST_CONFIGURE_OPTS) \
> > +		CFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" CPPFLAGS="" \
> > +		CXXFLAGS="-O2 $(HOST_LOCALEDEF_EXTRA_CFLAGS)" \
> > +		$(SHELL) $(@D)/configure \
> > +		ac_cv_path_BASH_SHELL=/bin/bash \
> > +		libc_cv_forced_unwind=yes \
> > +		libc_cv_ssp=no \
> > +		--target=$(GNU_HOST_NAME) \
> > +		--host=$(GNU_HOST_NAME) \
> > +		--build=$(GNU_HOST_NAME) \
> > +		--prefix=/usr \
> 
> Should be --prefix=$(HOST_DIR)

Oh yes, will fix.

> 
> > +		$(if $(BR2_SOFT_FLOAT),--without-fp,--with-fp) \
> > +		$(if $(BR2_x86_64),--enable-lock-elision) \
> 
> Both BR2_SOFT_FLOAT and BR2_x86_64 are target options, so observing
> them when building a host package is almost always wrong.
> 
> > +		--with-pkgversion="Buildroot" \
> > +		--without-cvs \
> > +		--disable-profile \
> > +		--without-gd \
> > +		--enable-obsolete-rpc)
> > +	$(HOST_LOCALEDEF_ADD_MISSING_STUB_H)
> > +endef
> 
> With all this, aren't we building a full glibc for the host ?
> 
> Can you use the locale/others and locale/install-others make targets
> instead ?
> 
> See how they are doing it in PTXdist:
> https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make.
> Most likely using the locale/others make target will help cut the build
> time significantly.

Yes this is the contentious part, we're building a whole glibc. Unfortunately
if we want to build just the locale/others target like PTXdist we also need
their bonus patch to glibc:
https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch

Otherwise the build fails with, for example:

make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'.  Stop.
Makefile:215: recipe for target 'locale/others' failed
make[3]: *** [locale/others] Error 2
Makefile:9: recipe for target 'locale/others' failed
make[2]: *** [locale/others] Error 2

So the tradeoff is adding a few minutes of extra build time versus carrying
a bit of a hack to enable just the localedef/others target.

> 
> Thanks!
> 
> Thomas

[Buildroot] [RFC PATCH] host-localedef: Compile against glibc

[Thomas Petazzoni]

Hello,

On Wed, 01 Aug 2018 11:31:16 +1000, Samuel Mendoza-Jonas wrote:

> I think it's a bit of both; 2.27 introduces new locale descriptions which
> localedef can't handle, eg:
> 
> Generating locale ru_RU.UTF-8
> /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error
> /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error

Ah, so I guess this update of localedef is going to fix
https://bugs.busybox.net/show_bug.cgi?id=11096.

> And we fall back to the C locale at runtime (at least in my environment).
> I'll stare at the glibc release notes / bug report a bit more and see if
> I can make a more legible explanation.

Would be nice, though it generally makes sense to have a localedef
implementation that is in sync with glibc.

> > See how they are doing it in PTXdist:
> > https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make.
> > Most likely using the locale/others make target will help cut the build
> > time significantly.  
> 
> Yes this is the contentious part, we're building a whole glibc. Unfortunately
> if we want to build just the locale/others target like PTXdist we also need
> their bonus patch to glibc:
> https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch
> 
> Otherwise the build fails with, for example:
> 
> make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'.  Stop.
> Makefile:215: recipe for target 'locale/others' failed
> make[3]: *** [locale/others] Error 2
> Makefile:9: recipe for target 'locale/others' failed
> make[2]: *** [locale/others] Error 2
> 
> So the tradeoff is adding a few minutes of extra build time versus carrying
> a bit of a hack to enable just the localedef/others target.

Their patch is not that crazy IMO, it should be too complicated to
maintain. Perhaps we could even improve it a bit further to make it
acceptable upstream ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [RFC PATCH] host-localedef: Compile against glibc

[Peter Seiderer]

Hello *,

On Wed, 1 Aug 2018 08:39:38 +0200, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote:

> Hello,
> 
> On Wed, 01 Aug 2018 11:31:16 +1000, Samuel Mendoza-Jonas wrote:
> 
> > I think it's a bit of both; 2.27 introduces new locale descriptions which
> > localedef can't handle, eg:
> > 
> > Generating locale ru_RU.UTF-8
> > /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:125: LC_TIME:syntax error
> > /scratch/jenkins-workspace/op-build-upstream/output/host/powerpc64le-buildroot-linux-gnu/sysroot/usr/share/i18n/locales/ru_RU:149: LC_TIME:syntax error  
> 
> Ah, so I guess this update of localedef is going to fix
> https://bugs.busybox.net/show_bug.cgi?id=11096.
> 
> > And we fall back to the C locale at runtime (at least in my environment).
> > I'll stare at the glibc release notes / bug report a bit more and see if
> > I can make a more legible explanation.  
> 
> Would be nice, though it generally makes sense to have a localedef
> implementation that is in sync with glibc.
> 
> > > See how they are doing it in PTXdist:
> > > https://git.pengutronix.de/cgit/ptxdist/tree/rules/host-localedef.make.
> > > Most likely using the locale/others make target will help cut the build
> > > time significantly.    
> > 
> > Yes this is the contentious part, we're building a whole glibc. Unfortunately
> > if we want to build just the locale/others target like PTXdist we also need
> > their bonus patch to glibc:
> > https://git.pengutronix.de/cgit/ptxdist/tree/patches/localedef-glibc-2.27/0001-HACK-only-build-and-install-localedef.patch
> > 
> > Otherwise the build fails with, for example:
> > 
> > make[4]: *** No rule to make target '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/elf/soinit.os', needed by '/scratch/builds/locales/build/host-localedef-glibc-2.27-57-g6c99e37f6fb640a50a3113b2dbee5d5389843c1e/build/libc.so'.  Stop.
> > Makefile:215: recipe for target 'locale/others' failed
> > make[3]: *** [locale/others] Error 2
> > Makefile:9: recipe for target 'locale/others' failed
> > make[2]: *** [locale/others] Error 2
> > 
> > So the tradeoff is adding a few minutes of extra build time versus carrying
> > a bit of a hack to enable just the localedef/others target.  
> 
> Their patch is not that crazy IMO, it should be too complicated to
> maintain. Perhaps we could even improve it a bit further to make it
> acceptable upstream ?
> 
> Best regards,
> l
> Thomas

Just stumbled over the same (or similar) locale problem with buildroot-2018.11/glibc-2.28,
fixed by the suggested patch. Just send out a follow up RFC patch with various comments addressed,
fixes and updates, see [1]...

Regards,
Peter

[1] https://patchwork.ozlabs.org/patch/1008473/