[Buildroot] buildroot uClibc-ng build failure

[Buildroot] buildroot uClibc-ng build failure

[Waldemar Brodkorb]

Hi Andr?,

we are seeing a build error on your machine when
locales in uClibc-ng are enabled:
http://autobuild.buildroot.net/results/d5b/d5ba81eea9223569ba5b363551c4a2f7044ff8ec/

Can you show us locale -a from your machine?

For a working tolower(), the uClibc-ng code needs to use setlocale
on the host.

But it seems only these are tried to use:
en_US en_US en_GB

best regards
 Waldemar

[Buildroot] buildroot uClibc-ng build failure

[Thomas Petazzoni]

Hello,

On Thu, 3 Aug 2017 07:48:28 +0200, Waldemar Brodkorb wrote:

> we are seeing a build error on your machine when
> locales in uClibc-ng are enabled:
> http://autobuild.buildroot.net/results/d5b/d5ba81eea9223569ba5b363551c4a2f7044ff8ec/
> 
> Can you show us locale -a from your machine?
> 
> For a working tolower(), the uClibc-ng code needs to use setlocale
> on the host.
> 
> But it seems only these are tried to use:
> en_US en_US en_GB

It is worth mentioning that Buildroot is already checking that at least
one UTF-8 locale is available on the host machine, for the needs of
uClibc locale support. From support/dependencies/dependencies.sh:

if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
        if ! which locale > /dev/null ; then
                echo
                echo "You need locale support on your build machine to build a toolchain supporting locales"
                exit 1 ;
        fi
        if ! locale -a | grep -q -i utf8$ ; then
                echo
                echo "You need at least one UTF8 locale to build a toolchain supporting locales"
                exit 1 ;
        fi
fi

And the only package selecting BR2_NEEDS_HOST_UTF8_LOCALE is:

package/uclibc/Config.in:       select BR2_NEEDS_HOST_UTF8_LOCALE

Is this check insufficient now ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] Fwd: buildroot uClibc-ng build failure

[Eugene Yudin]

I resent the mail.
Previous was rejected by mailing list.
Sorry for duplication.

---------- Forwarded message ----------
From: Eugene Yudin <e.yudin@ndmsystems.com>
Date: Thu, Aug 3, 2017 at 11:11 AM
Subject: Re: [Buildroot] buildroot uClibc-ng build failure
To: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Waldemar Brodkorb <wbx@openadk.org>, Andr? Hentschel <nerv@dawncrow.de>,
devel at uclibc-ng.org, buildroot at buildroot.org


Hi Thomas,

> Is this check insufficient now ?
I think it's sufficient.

Please post output of the following commands:
$ locale
$ locale -a
$ cat /etc/os-release

It would be very helpful to reproduce build failure.

On Thu, Aug 3, 2017 at 10:23 AM, Thomas Petazzoni <thomas.petazzoni@free-
electrons.com> wrote:

> Hello,
>
> On Thu, 3 Aug 2017 07:48:28 +0200, Waldemar Brodkorb wrote:
>
> > we are seeing a build error on your machine when
> > locales in uClibc-ng are enabled:
> > http://autobuild.buildroot.net/results/d5b/d5ba81eea9223569b
> a5b363551c4a2f7044ff8ec/
> >
> > Can you show us locale -a from your machine?
> >
> > For a working tolower(), the uClibc-ng code needs to use setlocale
> > on the host.
> >
> > But it seems only these are tried to use:
> > en_US en_US en_GB
>
> It is worth mentioning that Buildroot is already checking that at least
> one UTF-8 locale is available on the host machine, for the needs of
> uClibc locale support. From support/dependencies/dependencies.sh:
>
> if grep ^BR2_NEEDS_HOST_UTF8_LOCALE=y $BR2_CONFIG > /dev/null; then
>         if ! which locale > /dev/null ; then
>                 echo
>                 echo "You need locale support on your build machine to
> build a toolchain supporting locales"
>                 exit 1 ;
>         fi
>         if ! locale -a | grep -q -i utf8$ ; then
>                 echo
>                 echo "You need at least one UTF8 locale to build a
> toolchain supporting locales"
>                 exit 1 ;
>         fi
> fi
>
> And the only package selecting BR2_NEEDS_HOST_UTF8_LOCALE is:
>
> package/uclibc/Config.in:       select BR2_NEEDS_HOST_UTF8_LOCALE
>
> Is this check insufficient now ?
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
>



-- 
Best regards,
Eugene



-- 
Best regards,
Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170803/7b16e1e1/attachment.html>

[Buildroot] buildroot uClibc-ng build failure

[André Hentschel]

Am 03.08.2017 um 10:11 schrieb Eugene Yudin:
> Hi Thomas,
> 
>> Is this check insufficient now ?
> I think it's sufficient.
> 
> Please post output of the following commands:
> $ locale
> $ locale -a
> $ cat /etc/os-release
$ locale
LANG=de_DE.utf8
LANGUAGE=de_DE.utf8
LC_CTYPE="de_DE.utf8"
LC_NUMERIC="de_DE.utf8"
LC_TIME="de_DE.utf8"
LC_COLLATE="de_DE.utf8"
LC_MONETARY="de_DE.utf8"
LC_MESSAGES="de_DE.utf8"
LC_PAPER="de_DE.utf8"
LC_NAME="de_DE.utf8"
LC_ADDRESS="de_DE.utf8"
LC_TELEPHONE="de_DE.utf8"
LC_MEASUREMENT="de_DE.utf8"
LC_IDENTIFICATION="de_DE.utf8"
LC_ALL=de_DE.utf8

$ locale -a
C
C.UTF-8
de_DE.utf8
POSIX

OS is Ubuntu 14.04.5 LTS
Should I install the english locale?

[Buildroot] buildroot uClibc-ng build failure

[Eugene Yudin]

Yes. Please install the locale "en_US.UTF-8".
It's used in buildroot by default:
package/uclibc/uclibc.mk:46:UCLIBC_LOCALES = en_US

On Tue, Aug 8, 2017 at 7:52 PM, Andr? Hentschel <nerv@dawncrow.de> wrote:

> Am 03.08.2017 um 10:11 schrieb Eugene Yudin:
> > Hi Thomas,
> >
> >> Is this check insufficient now ?
> > I think it's sufficient.
> >
> > Please post output of the following commands:
> > $ locale
> > $ locale -a
> > $ cat /etc/os-release
> $ locale
> LANG=de_DE.utf8
> LANGUAGE=de_DE.utf8
> LC_CTYPE="de_DE.utf8"
> LC_NUMERIC="de_DE.utf8"
> LC_TIME="de_DE.utf8"
> LC_COLLATE="de_DE.utf8"
> LC_MONETARY="de_DE.utf8"
> LC_MESSAGES="de_DE.utf8"
> LC_PAPER="de_DE.utf8"
> LC_NAME="de_DE.utf8"
> LC_ADDRESS="de_DE.utf8"
> LC_TELEPHONE="de_DE.utf8"
> LC_MEASUREMENT="de_DE.utf8"
> LC_IDENTIFICATION="de_DE.utf8"
> LC_ALL=de_DE.utf8
>
> $ locale -a
> C
> C.UTF-8
> de_DE.utf8
> POSIX
>
> OS is Ubuntu 14.04.5 LTS
> Should I install the english locale?
>



-- 
Best regards,
Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170808/0197ea4b/attachment.html>

[Buildroot] buildroot uClibc-ng build failure

[André Hentschel]

done:

$ locale -a
C
C.UTF-8
de_DE.utf8
en_US
en_US.iso88591
en_US.utf8
POSIX


Am 08.08.2017 um 21:07 schrieb Eugene Yudin:
> Yes. Please install the locale "en_US.UTF-8".
> It's used in buildroot by default:
> package/uclibc/uclibc.mk:46:UCLIBC_LOCALES = en_US
> 
> On Tue, Aug 8, 2017 at 7:52 PM, Andr? Hentschel <nerv at dawncrow.de <mailto:nerv@dawncrow.de>> wrote:
> 
>     Am 03.08.2017 um 10:11 schrieb Eugene Yudin:
>     > Hi Thomas,
>     >
>     >> Is this check insufficient now ?
>     > I think it's sufficient.
>     >
>     > Please post output of the following commands:
>     > $ locale
>     > $ locale -a
>     > $ cat /etc/os-release
>     $ locale
>     LANG=de_DE.utf8
>     LANGUAGE=de_DE.utf8
>     LC_CTYPE="de_DE.utf8"
>     LC_NUMERIC="de_DE.utf8"
>     LC_TIME="de_DE.utf8"
>     LC_COLLATE="de_DE.utf8"
>     LC_MONETARY="de_DE.utf8"
>     LC_MESSAGES="de_DE.utf8"
>     LC_PAPER="de_DE.utf8"
>     LC_NAME="de_DE.utf8"
>     LC_ADDRESS="de_DE.utf8"
>     LC_TELEPHONE="de_DE.utf8"
>     LC_MEASUREMENT="de_DE.utf8"
>     LC_IDENTIFICATION="de_DE.utf8"
>     LC_ALL=de_DE.utf8
> 
>     $ locale -a
>     C
>     C.UTF-8
>     de_DE.utf8
>     POSIX
> 
>     OS is Ubuntu 14.04.5 LTS
>     Should I install the english locale?
> 
> 
> 
> 
> -- 
> Best regards,
> Eugene

[Buildroot] [uclibc-ng-devel] buildroot uClibc-ng build failure

[Max Filippov]

On Tue, Aug 8, 2017 at 12:07 PM, Eugene Yudin <e.yudin@ndmsystems.com> wrote:
> Yes. Please install the locale "en_US.UTF-8".
> It's used in buildroot by default:
> package/uclibc/uclibc.mk:46:UCLIBC_LOCALES = en_US

Just curious: why not defaulting to C (or C.UTF-8) locale?

-- 
Thanks.
-- Max

[Buildroot] buildroot uClibc-ng build failure

[Thomas Petazzoni]

Hello,

On Tue, 8 Aug 2017 18:52:14 +0200, Andr? Hentschel wrote:

> > Please post output of the following commands:
> > $ locale
> > $ locale -a
> > $ cat /etc/os-release  
> $ locale
> LANG=de_DE.utf8
> LANGUAGE=de_DE.utf8
> LC_CTYPE="de_DE.utf8"
> LC_NUMERIC="de_DE.utf8"
> LC_TIME="de_DE.utf8"
> LC_COLLATE="de_DE.utf8"
> LC_MONETARY="de_DE.utf8"
> LC_MESSAGES="de_DE.utf8"
> LC_PAPER="de_DE.utf8"
> LC_NAME="de_DE.utf8"
> LC_ADDRESS="de_DE.utf8"
> LC_TELEPHONE="de_DE.utf8"
> LC_MEASUREMENT="de_DE.utf8"
> LC_IDENTIFICATION="de_DE.utf8"
> LC_ALL=de_DE.utf8
> 
> $ locale -a
> C
> C.UTF-8
> de_DE.utf8
> POSIX
> 
> OS is Ubuntu 14.04.5 LTS
> Should I install the english locale?

As per the current Buildroot requirements, it shouldn't be needed. We
require one UTF-8 locale to be installed for uClibc to build its locale
data, but not specifically the en_US.UTF-8 locale or any other english
locale. If that has changed in uClibc, then we should adjust the
dependency in Buildroot accordingly, but it would be a bit annoying.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] buildroot uClibc-ng build failure

[Eugene Yudin]

I describe the current situation:
1) user enables locale support (BR2_TOOLCHAIN_BUILDROOT_LOCALE);
2) user optionally can set custom list of locales (BR2_GENERATE_LOCALE).

The option "BR2_GENERATE_LOCALE" is handled in "package/uclibc/uclibc.mk":
 UCLIBC_GENERATE_LOCALES = $(call qstrip,$(BR2_GENERATE_LOCALE))

 ifeq ($(UCLIBC_GENERATE_LOCALES),)
 # We need at least one locale
 UCLIBC_LOCALES = en_US
 else
 # Strip out the encoding part of locale names, if any
 UCLIBC_LOCALES = \
         $(foreach locale,$(UCLIBC_GENERATE_LOCALES),\
         $(firstword $(subst .,$(space),$(locale))))
 endif
 ...
 define UCLIBC_LOCALE_CONFIG
         ...
         $(call
KCONFIG_ENABLE_OPT,UCLIBC_BUILD_MINIMAL_LOCALE,$(@D)/.config)
         $(call
KCONFIG_SET_OPT,UCLIBC_BUILD_MINIMAL_LOCALES,"$(UCLIBC_LOCALES)",$(@D)/.config)
         ...
 endef

We can see that with empty list buildroot passes "en_US" to uClibc-ng.

The system library makefile handles option the following way
(extra/locale/Makefile.in):
 $(locale_OUT)/wctables.h: $(locale_OUT)/gen_wctype
         @$(disp_gen)
         $(Q)for locale in $(call qstrip,$(UCLIBC_BUILD_MINIMAL_LOCALES))
en_US en_GB; do \
                  $< $(FLAG-locale-verbose) $$locale > $@ || \
                 $< $(FLAG-locale-verbose) $$locale.UTF-8 > $@ || \
                 $< $(FLAG-locale-verbose) $$locale.iso8859-1 > $@ &&
break; \
         done

The program "gen_wctype" receives locale name as command line parameter.
That application call the function setlocale() inside it.
Some time ago that call was disabled to workaround bug with locales.
That's the reason for the absence of problems in the past.
Changes in the last release return this call.

On Wed, Aug 9, 2017 at 12:00 AM, Thomas Petazzoni <
thomas.petazzoni@free-electrons.com> wrote:

> Hello,
>
> On Tue, 8 Aug 2017 18:52:14 +0200, Andr? Hentschel wrote:
>
> > > Please post output of the following commands:
> > > $ locale
> > > $ locale -a
> > > $ cat /etc/os-release
> > $ locale
> > LANG=de_DE.utf8
> > LANGUAGE=de_DE.utf8
> > LC_CTYPE="de_DE.utf8"
> > LC_NUMERIC="de_DE.utf8"
> > LC_TIME="de_DE.utf8"
> > LC_COLLATE="de_DE.utf8"
> > LC_MONETARY="de_DE.utf8"
> > LC_MESSAGES="de_DE.utf8"
> > LC_PAPER="de_DE.utf8"
> > LC_NAME="de_DE.utf8"
> > LC_ADDRESS="de_DE.utf8"
> > LC_TELEPHONE="de_DE.utf8"
> > LC_MEASUREMENT="de_DE.utf8"
> > LC_IDENTIFICATION="de_DE.utf8"
> > LC_ALL=de_DE.utf8
> >
> > $ locale -a
> > C
> > C.UTF-8
> > de_DE.utf8
> > POSIX
> >
> > OS is Ubuntu 14.04.5 LTS
> > Should I install the english locale?
>
> As per the current Buildroot requirements, it shouldn't be needed. We
> require one UTF-8 locale to be installed for uClibc to build its locale
> data, but not specifically the en_US.UTF-8 locale or any other english
> locale. If that has changed in uClibc, then we should adjust the
> dependency in Buildroot accordingly, but it would be a bit annoying.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
>



-- 
Best regards,
Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170810/ec243a68/attachment.html>

[Buildroot] buildroot uClibc-ng build failure

[Arnout Vandecappelle]

On 10-08-17 11:09, Eugene Yudin wrote:
> The system library makefile handles option the following way
> (extra/locale/Makefile.in):
>  $(locale_OUT)/wctables.h: $(locale_OUT)/gen_wctype
>          @$(disp_gen)
>          $(Q)for locale in $(call qstrip,$(UCLIBC_BUILD_MINIMAL_LOCALES)) en_US
> en_GB; do \
>                   $< $(FLAG-locale-verbose) $$locale > $@ || \
>                  $< $(FLAG-locale-verbose) $$locale.UTF-8 > $@ || \
>                  $< $(FLAG-locale-verbose) $$locale.iso8859-1 > $@ && break; \
>          done
> 
> The program "gen_wctype" receives locale name as command line parameter.
> That application call the function setlocale() inside it.
> Some time ago that call was disabled to workaround bug with locales.
> That's the reason for the absence of problems in the past.
> Changes in the last release return this call.

 So the problem really is that this command tries a number of locales, but not
any of the locales actually present on the system?

 I would propose to:
- replace the "en_US en_GB" with "C C.UTF-8" since the latter is much more
likely to be present on the host;
- use '$(sort $(patsubst %.utf8,%,$(shell locale -a)))' as an ultimate fallback.

 Does that sound reasonable? I didn't look at the history if this particular
piece of code, I guess it must have gone through several detours already...

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] buildroot uClibc-ng build failure

[Eugene Yudin]

Yes. In that particular case loop iterates through en_US (from buildroot),
en_US and en_GB.
Of cause, that list can be patched but I don't know if that's correct.
It's look a bit strange if user passes locale which actually absent.

Waldemar, what you think about it?

I made several tests with different locales (en_US, en_GB, ru_RU, C.UTF-8
and C).
The generated header is the same for all the above locale except pure C.

Comment in the same makefile tells about problem with some locales.
I think it would be better to use C locale for fallback.

I attached the tested patch to modify fallback behavior based on Arnout
draft.
I also changed the order of calls to raise priority of C.UTF-8 locale.

On Thu, Aug 10, 2017 at 12:36 PM, Arnout Vandecappelle <arnout@mind.be>
wrote:

>
>
> On 10-08-17 11:09, Eugene Yudin wrote:
> > The system library makefile handles option the following way
> > (extra/locale/Makefile.in):
> >  $(locale_OUT)/wctables.h: $(locale_OUT)/gen_wctype
> >          @$(disp_gen)
> >          $(Q)for locale in $(call qstrip,$(UCLIBC_BUILD_MINIMAL_LOCALES))
> en_US
> > en_GB; do \
> >                   $< $(FLAG-locale-verbose) $$locale > $@ || \
> >                  $< $(FLAG-locale-verbose) $$locale.UTF-8 > $@ || \
> >                  $< $(FLAG-locale-verbose) $$locale.iso8859-1 > $@ &&
> break; \
> >          done
> >
> > The program "gen_wctype" receives locale name as command line parameter.
> > That application call the function setlocale() inside it.
> > Some time ago that call was disabled to workaround bug with locales.
> > That's the reason for the absence of problems in the past.
> > Changes in the last release return this call.
>
>  So the problem really is that this command tries a number of locales, but
> not
> any of the locales actually present on the system?
>
>  I would propose to:
> - replace the "en_US en_GB" with "C C.UTF-8" since the latter is much more
> likely to be present on the host;
> - use '$(sort $(patsubst %.utf8,%,$(shell locale -a)))' as an ultimate
> fallback.
>
>  Does that sound reasonable? I didn't look at the history if this
> particular
> piece of code, I guess it must have gone through several detours already...
>
>  Regards,
>  Arnout
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
>



-- 
Best regards,
Eugene
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170810/c236c8ab/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: uclibc_gen_wctables_fallback.patch
Type: text/x-patch
Size: 759 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20170810/c236c8ab/attachment.bin>

[Buildroot] buildroot uClibc-ng build failure

[Waldemar Brodkorb]

Hi Eugene,
Eugene Yudin wrote,

> Yes. In that particular case loop iterates through en_US (from buildroot),
> en_US and en_GB.
> Of cause, that list can be patched but I don't know if that's correct.
> It's look a bit strange if user passes locale which actually absent.
> 
> Waldemar, what you think about it?
> 
> I made several tests with different locales (en_US, en_GB, ru_RU, C.UTF-8 and
> C).
> The generated header is the same for all the above locale except pure C.
> 
> Comment in the same makefile tells about problem with some locales.
> I think it would be better to use C locale for fallback.
> 
> I attached the tested patch to modify fallback behavior based on Arnout draft.
> I also changed the order of calls to raise priority of C.UTF-8 locale.

Looks good to me. First trying the user configured locale (.UTF-8)
and if it does not exist fallback to C.UTF-8.

Not sure if we should do configured locale (en_US.UTF-8) then
C.UTF_8 and then even C, when C doesn't build workable output?

best regards
 Waldemar