[Buildroot] [PATCH] uclibc: deprecate locale support

[Buildroot] [PATCH] uclibc: deprecate locale support

[Waldemar Brodkorb]

The quality of the locale implementation in uClibc seems
to be bad. Recently a discussion on the uClibc mailinglist
showed that the uClibc maintainer has no time to fix
remaining issues in the locale implementation.
http://lists.uclibc.org/pipermail/uclibc/2015-June/049000.html

Because of the bad state of the code I would like to deprecate
it. I never used it myself and see no good reason why I would
need it for an embedded system.

I believe not many people are using it. For good working
locale support there is still GNU libc and musl.

Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
---
 Config.in.legacy         |    6 ++++++
 package/uclibc/Config.in |    8 --------
 2 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/Config.in.legacy b/Config.in.legacy
index d86d585..1814acb 100644
--- a/Config.in.legacy
+++ b/Config.in.legacy
@@ -107,6 +107,12 @@ endif
 ###############################################################################
 comment "Legacy options removed in 2015.08"
 
+config BR2_TOOLCHAIN_BUILDROOT_LOCALE
+	bool "Toolchain locale/i18n setting removed"
+	select BR2_LEGACY
+	help
+	  uClibc locale support have been removed.
+
 config BR2_TARGET_UBOOT_NETWORK
 	bool "U-Boot custom network settings removed"
 	select BR2_LEGACY
diff --git a/package/uclibc/Config.in b/package/uclibc/Config.in
index 15a590f..03d7171 100644
--- a/package/uclibc/Config.in
+++ b/package/uclibc/Config.in
@@ -100,14 +100,6 @@ config BR2_TOOLCHAIN_BUILDROOT_WCHAR
 	  wide characters (i.e characters longer than 8 bits, needed
 	  for locale support).
 
-config BR2_TOOLCHAIN_BUILDROOT_LOCALE
-	bool "Enable toolchain locale/i18n support"
-	select BR2_TOOLCHAIN_BUILDROOT_WCHAR
-	select BR2_ENABLE_LOCALE
-	help
-	  Enable this option if you want your toolchain to support
-	  localization and internationalization.
-
 choice
 	prompt "Thread library implementation"
 	default BR2_PTHREADS_NATIVE if BR2_UCLIBC_VERSION_SUPPORTS_NPTL
-- 
1.7.10.4

[Buildroot] [PATCH] uclibc: deprecate locale support

[Yann E. MORIN]

Waldemar, All,

On 2015-06-28 09:14 +0200, Waldemar Brodkorb spake thusly:
> The quality of the locale implementation in uClibc seems
> to be bad. Recently a discussion on the uClibc mailinglist
> showed that the uClibc maintainer has no time to fix
> remaining issues in the locale implementation.
> http://lists.uclibc.org/pipermail/uclibc/2015-June/049000.html
> 
> Because of the bad state of the code I would like to deprecate
> it. I never used it myself and see no good reason why I would
> need it for an embedded system.
> 
> I believe not many people are using it. For good working
> locale support there is still GNU libc and musl.
> 
> Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>

While on principle I'm not opposed to deprecating locale support in
uClibc altogether.

However, we've had locales working in uClinc so far, it's only with
uClibc-ng that we started to have build failures (AFAICS).

So, I find it a bit cavalier that we remove support for locales when
uClibc-ng / uClibc is used, like we'd hide the dirt under the rug.

Also, when we add something to the legacy entries, this is not
deprecating, this is removing. Deprecating is done by adding a
dependency to a symbol, like is done now for the blackbox package.

But for what it's worth:

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
>  Config.in.legacy         |    6 ++++++
>  package/uclibc/Config.in |    8 --------
>  2 files changed, 6 insertions(+), 8 deletions(-)
> 
> diff --git a/Config.in.legacy b/Config.in.legacy
> index d86d585..1814acb 100644
> --- a/Config.in.legacy
> +++ b/Config.in.legacy
> @@ -107,6 +107,12 @@ endif
>  ###############################################################################
>  comment "Legacy options removed in 2015.08"
>  
> +config BR2_TOOLCHAIN_BUILDROOT_LOCALE
> +	bool "Toolchain locale/i18n setting removed"
> +	select BR2_LEGACY
> +	help
> +	  uClibc locale support have been removed.
> +
>  config BR2_TARGET_UBOOT_NETWORK
>  	bool "U-Boot custom network settings removed"
>  	select BR2_LEGACY
> diff --git a/package/uclibc/Config.in b/package/uclibc/Config.in
> index 15a590f..03d7171 100644
> --- a/package/uclibc/Config.in
> +++ b/package/uclibc/Config.in
> @@ -100,14 +100,6 @@ config BR2_TOOLCHAIN_BUILDROOT_WCHAR
>  	  wide characters (i.e characters longer than 8 bits, needed
>  	  for locale support).
>  
> -config BR2_TOOLCHAIN_BUILDROOT_LOCALE
> -	bool "Enable toolchain locale/i18n support"
> -	select BR2_TOOLCHAIN_BUILDROOT_WCHAR
> -	select BR2_ENABLE_LOCALE
> -	help
> -	  Enable this option if you want your toolchain to support
> -	  localization and internationalization.
> -
>  choice
>  	prompt "Thread library implementation"
>  	default BR2_PTHREADS_NATIVE if BR2_UCLIBC_VERSION_SUPPORTS_NPTL
> -- 
> 1.7.10.4
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH] uclibc: deprecate locale support

[Waldemar Brodkorb]

Hi,
Yann E. MORIN wrote,

> Waldemar, All,
> 
> On 2015-06-28 09:14 +0200, Waldemar Brodkorb spake thusly:
> > The quality of the locale implementation in uClibc seems
> > to be bad. Recently a discussion on the uClibc mailinglist
> > showed that the uClibc maintainer has no time to fix
> > remaining issues in the locale implementation.
> > http://lists.uclibc.org/pipermail/uclibc/2015-June/049000.html
> > 
> > Because of the bad state of the code I would like to deprecate
> > it. I never used it myself and see no good reason why I would
> > need it for an embedded system.
> > 
> > I believe not many people are using it. For good working
> > locale support there is still GNU libc and musl.
> > 
> > Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
> 
> While on principle I'm not opposed to deprecating locale support in
> uClibc altogether.
> 
> However, we've had locales working in uClinc so far, it's only with
> uClibc-ng that we started to have build failures (AFAICS).

I am really sure this also happens with uClibc git master.
It worked for 0.9.33.2 in some way, but it is broken in latest code.
So uclibc-ng inherited that.
 
> So, I find it a bit cavalier that we remove support for locales when
> uClibc-ng / uClibc is used, like we'd hide the dirt under the rug.

I planned to remove/deprecate support for locales before the
autobuilders started to show the bad situation. Just didn't had the
time to prepare a patch.
 
May be it is just time to get rid of this unmaintained
functionality.

If someone have a usecase for locales and prepares a patch, I will
not reject it.

best regards
 Waldemar

[Buildroot] [PATCH] uclibc: deprecate locale support

[Thomas Petazzoni]

Dear Waldemar Brodkorb,

On Sun, 28 Jun 2015 09:14:17 +0200, Waldemar Brodkorb wrote:
> The quality of the locale implementation in uClibc seems
> to be bad. Recently a discussion on the uClibc mailinglist
> showed that the uClibc maintainer has no time to fix
> remaining issues in the locale implementation.
> http://lists.uclibc.org/pipermail/uclibc/2015-June/049000.html
> 
> Because of the bad state of the code I would like to deprecate
> it. I never used it myself and see no good reason why I would
> need it for an embedded system.
> 
> I believe not many people are using it. For good working
> locale support there is still GNU libc and musl.
> 
> Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>

Can you assess the impact on the packages we currently have in
Buildroot ? I.e, which packages can only work with locale support?

If we remove BR2_TOOLCHAIN_BUILDROOT_LOCALE, then there's a lot of
possible further cleanup: BR2_ENABLE_LOCALE should be removed, all
packages doing select BR2_PACKAGE_LIBICONV if !BR2_ENABLE_LOCALE should
be changed, etc.

Also, do you have an idea of what is the state of the locale support in
musl ? Does musl provide the functions provided by libiconv ?

It would be good to take this opportunity to make sure the locale
support is OK.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH] uclibc: deprecate locale support

[Waldemar Brodkorb]

Hi Thomas,
Thomas Petazzoni wrote,

> Dear Waldemar Brodkorb,
> 
> On Sun, 28 Jun 2015 09:14:17 +0200, Waldemar Brodkorb wrote:
> > The quality of the locale implementation in uClibc seems
> > to be bad. Recently a discussion on the uClibc mailinglist
> > showed that the uClibc maintainer has no time to fix
> > remaining issues in the locale implementation.
> > http://lists.uclibc.org/pipermail/uclibc/2015-June/049000.html
> > 
> > Because of the bad state of the code I would like to deprecate
> > it. I never used it myself and see no good reason why I would
> > need it for an embedded system.
> > 
> > I believe not many people are using it. For good working
> > locale support there is still GNU libc and musl.
> > 
> > Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
> 
> Can you assess the impact on the packages we currently have in
> Buildroot ? I.e, which packages can only work with locale support?

I think no package really needs locale support in the C library.

May be there is some misunderstanding about the locale stuff.
Here is how I understand the stuff.

I think we have mixed up two different things here:
- iconv: perform character set conversion, needed for a lot of
  packages, f.e. Kodi. Is included in Glibc and Musl. Libiconv can
  be used with uClibc-ng/uClibc.
- locale: defines the user's language, country, ..
  If you want f.e. german error messages on your embedded system
  you would need locale support and locale data.
  env LC_ALL=de_DE ls foo

Who is really using the locale stuff?
I think the iconv functions, which are automatically enabled
when UCLIBC_HAS_LOCALE is set are neither complete not used
externally. If libiconv is to big, libiconv-tiny for simple
UTF-8 <-> ASCII conversion could be used.
 
> If we remove BR2_TOOLCHAIN_BUILDROOT_LOCALE, then there's a lot of
> possible further cleanup: BR2_ENABLE_LOCALE should be removed, all
> packages doing select BR2_PACKAGE_LIBICONV if !BR2_ENABLE_LOCALE should
> be changed, etc.

May be something like this:
select BR2_PACKAGE_LIBICONV if BR2_LIBC_WITHOUT_ICONV
 
> Also, do you have an idea of what is the state of the locale support in
> musl ? Does musl provide the functions provided by libiconv ?

Yes. Like GNU libc, musl already provides iconv functions out of the
box.
 
> It would be good to take this opportunity to make sure the locale
> support is OK.

Yes, that would be good.

best regards
 Waldemar

[Buildroot] [PATCH] uclibc: deprecate locale support

[Thomas Petazzoni]

Hello,

On Sun, 28 Jun 2015 14:58:37 +0200, Waldemar Brodkorb wrote:

> I think no package really needs locale support in the C library.
> 
> May be there is some misunderstanding about the locale stuff.
> Here is how I understand the stuff.
> 
> I think we have mixed up two different things here:
> - iconv: perform character set conversion, needed for a lot of
>   packages, f.e. Kodi. Is included in Glibc and Musl. Libiconv can
>   be used with uClibc-ng/uClibc.
> - locale: defines the user's language, country, ..
>   If you want f.e. german error messages on your embedded system
>   you would need locale support and locale data.
>   env LC_ALL=de_DE ls foo

Fully agreed here. But in uClibc, enabling locale support actually
gives both: the iconv functions *and* the locale support itself.

> Who is really using the locale stuff?
> I think the iconv functions, which are automatically enabled
> when UCLIBC_HAS_LOCALE is set are neither complete not used
> externally.

Of course they are used externally: many of our packages do "select
BR2_PACKAGE_LIBICONV if !BR2_ENABLE_LOCALE" because they
unconditionally use the iconv function, and uClibc only provide these
functions if locale support is enabled. From
libc/misc/wchar/Makefile.in (in original uClibc, I haven't looked at
uClibc-ng) :

CSRC-$(UCLIBC_HAS_LOCALE) += iconv.c

> If libiconv is to big, libiconv-tiny for simple UTF-8 <-> ASCII
> conversion could be used. 

This could be useful, but this would be a separate matter.

> > If we remove BR2_TOOLCHAIN_BUILDROOT_LOCALE, then there's a lot of
> > possible further cleanup: BR2_ENABLE_LOCALE should be removed, all
> > packages doing select BR2_PACKAGE_LIBICONV if !BR2_ENABLE_LOCALE
> > should be changed, etc.
> 
> May be something like this:
> select BR2_PACKAGE_LIBICONV if BR2_LIBC_WITHOUT_ICONV

Yes, correct.

> > Also, do you have an idea of what is the state of the locale
> > support in musl ? Does musl provide the functions provided by
> > libiconv ?
> 
> Yes. Like GNU libc, musl already provides iconv functions out of the
> box.

Ok, so BR2_LIBC_WITHOUT_ICONV would be simple: it's y if uClibc, and n
otherwise.

Or should we instead have in uClibc the iconv functions enabled by
default (by having a UCLIBC_HAS_ICONV option separate from
UCLIBC_HAS_LOCALE), so that uClibc is like all other C libraries from
an iconv() point of view ? And of course, get rid completely of locale
support in uClibc ?

There are still lots of packages that use BR2_ENABLE_LOCALE in
Buildroot, we will have to check what they do exactly with it.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH] uclibc: deprecate locale support

[Waldemar Brodkorb]

Hi,
Thomas Petazzoni wrote,

> Hello,
> 
> On Sun, 28 Jun 2015 14:58:37 +0200, Waldemar Brodkorb wrote:
> 
> > I think no package really needs locale support in the C library.
> > 
> > May be there is some misunderstanding about the locale stuff.
> > Here is how I understand the stuff.
> > 
> > I think we have mixed up two different things here:
> > - iconv: perform character set conversion, needed for a lot of
> >   packages, f.e. Kodi. Is included in Glibc and Musl. Libiconv can
> >   be used with uClibc-ng/uClibc.
> > - locale: defines the user's language, country, ..
> >   If you want f.e. german error messages on your embedded system
> >   you would need locale support and locale data.
> >   env LC_ALL=de_DE ls foo
> 
> Fully agreed here. But in uClibc, enabling locale support actually
> gives both: the iconv functions *and* the locale support itself.

Yeah, unfortunately both is coupled together as another research has
proofed.
 
So the best option for 1.0.x to fix the build issue.
I tracked it down to a parallel build issue. When number of jobs
is set to 1, the error does not happen.
I thought using the make target pregen instead of header would 
be a good workaround, but the problem still exist in Xtensa builds.
The ARM builds work after this change.

Any parallel make gurus available?
It only happens hen LOCALE support is activated.

best regards
 Waldemar