[Buildroot] musl/gettext issue

[Buildroot] musl/gettext issue

[Thomas Petazzoni]

Hello,

Today, I started looking at:

   http://autobuild.buildroot.org/results/ede/ede5ee3316ea2b6790dcb930a6bc71adc8922bfc/build-end.log

Which, in appearance, seems to be the traditional missing -lintl when
linking. But further investigation has revealed a slightly more
complicated problem: how do we handle GNU gettext vs. musl.

Until, we were handling GNU gettext for glibc and uClibc:

 * With glibc, the gettext handling is built into the C library. So the
   separate GNU gettext package (for the target) is not needed, and if
   it ever gets built, it detects that the C library has the necessary
   features, and therefore it doesn't build/install a libintl.so
   library and its header files.

 * With uclibc, the gettext handling is not part of the C library, so
   we have to build the GNU gettext package when gettext support is
   needed. In this case, the GNU gettext package detects that the C
   library does not provide the gettext functionality, and it will
   build/install libintl.so and its header files.

This is why we have BR2_NEEDS_GETTEXT which is true when we use uClibc
and false when we use uClibc, and BR2_NEEDS_GETTEXT_IF_LOCALE which is
true when BR2_NEEDS_GETTEXT is true (i.e uClibc) *and* locale support
is enabled.

So far, so good.

Now, enter musl. It does have an internal gettext implementation.
However, it is not recognized by GNU gettext has a correct
implementation, so when you build GNU gettext in a musl system, it does
build/install libintl.so and its header files.

So for httping, two scenarios are possible:

 1/ httping is built alone against musl. No problem: the gettext
    functions are part of the C library, everything works fine.

 2/ httping is built *after* GNU gettext has been built. Since GNU
    gettext will replace the libintl.h of musl by its own one, the
    symbols from the GNU gettext libintl.so will be used, so we must
    link with -lintl explicitly. Which we are not doing, since
    htting.mk does:

       $(if $(BR2_NEEDS_GETTEXT),-lintl)

    And BR2_NEEDS_GETTEXT is false for musl.

So, I initially tried:

-       $(if $(BR2_NEEDS_GETTEXT),-lintl)
+       $(if $(BR2_PACKAGE_GETTEXT),-lintl)

With the reasoning that if GNU gettext is available, we want to use it,
and if it's not available, then we'll not use it.

But that doesn't work with a glibc configuration: BR2_PACKAGE_GETTEXT
can be enabled, but we don't have a libintl library, because GNU
gettext doesn't build one in a glibc configuration. We try to link
against libintl, but it's not there, and the build fails.

So, I see two possible options here:

 1/ Simply do not allow the GNU gettext package to be built with glibc
    and musl since they provide the gettext functionality internally.

    The only problem with this approach is that while httping is happy
    with the POSIX compliant gettext functionality of musl, some other
    programs such as Bison
    (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786885) will
    really need GNU gettext functionality.

 2/ Allows force to use GNU gettext in musl configurations. This simply
    consists in:

 config BR2_NEEDS_GETTEXT
        bool
-       default y if BR2_TOOLCHAIN_USES_UCLIBC
+       default y if BR2_TOOLCHAIN_USES_UCLIBC || BR2_TOOLCHAIN_USES_MUSL

     I have tested this solution and it does work (obviously).

     The drawback is obviously that we are going to build/install GNU
     gettext even for cases where the internal gettext implementation
     of musl would have been sufficient.

Do you see some other options? Any opinion between the two proposed
options?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] musl/gettext issue

[Arnout Vandecappelle]

On 16-09-15 22:16, Thomas Petazzoni wrote:
[snip]
> But that doesn't work with a glibc configuration: BR2_PACKAGE_GETTEXT
> can be enabled, but we don't have a libintl library, because GNU
> gettext doesn't build one in a glibc configuration. We try to link
> against libintl, but it's not there, and the build fails.
> 
> So, I see two possible options here:
> 
>  1/ Simply do not allow the GNU gettext package to be built with glibc
>     and musl since they provide the gettext functionality internally.

 Problem: something like ecryptfs-utils, which needs the gettext program at
runtime...


>     The only problem with this approach is that while httping is happy
>     with the POSIX compliant gettext functionality of musl, some other
>     programs such as Bison
>     (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786885) will
>     really need GNU gettext functionality.

 This one could perhaps be solved by adding a BR2_HAS_GNU_GETTEXT. But again, of
course, it must be possible to select gettext in that case.


> 
>  2/ Allows force to use GNU gettext in musl configurations. This simply
>     consists in:
> 
>  config BR2_NEEDS_GETTEXT
>         bool
> -       default y if BR2_TOOLCHAIN_USES_UCLIBC
> +       default y if BR2_TOOLCHAIN_USES_UCLIBC || BR2_TOOLCHAIN_USES_MUSL
> 
>      I have tested this solution and it does work (obviously).
> 
>      The drawback is obviously that we are going to build/install GNU
>      gettext even for cases where the internal gettext implementation
>      of musl would have been sufficient.

 That sounds like a bad idea. What's the point of using musl if you're going to
add a bunch of GNU extensions to it?


 Another option is that we add a different flag to indicate the dependency on
libintl:

config BR2_PACKAGE_GETTEXT_WITH_LIBINTL
	bool
	default y
	depends on BR2_PACKAGE_GETTEXT
	depends on BR2_STATIC_LIBS
	depends on !BR2_TOOLCHAIN_USES_GLIBC


 HTTPING_LDFLAGS = $(TARGET_LDFLAGS) \
        $(if $(BR2_PACKAGE_GETTEXT_WITH_LIBINTL),-lintl) \


 There's just about 30 packages that ned to be adapted.

 Regards,
 Arnout

> 
> Do you see some other options? Any opinion between the two proposed
> options?
> 
> Thanks,
> 
> Thomas
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] musl/gettext issue

[Thomas Petazzoni]

Hello,

On Fri, 18 Sep 2015 00:27:21 +0200, Arnout Vandecappelle wrote:

> >  1/ Simply do not allow the GNU gettext package to be built with glibc
> >     and musl since they provide the gettext functionality internally.
> 
>  Problem: something like ecryptfs-utils, which needs the gettext program at
> runtime...

Ah, right, correct.

> >     The only problem with this approach is that while httping is happy
> >     with the POSIX compliant gettext functionality of musl, some other
> >     programs such as Bison
> >     (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786885) will
> >     really need GNU gettext functionality.
> 
>  This one could perhaps be solved by adding a BR2_HAS_GNU_GETTEXT. But again, of
> course, it must be possible to select gettext in that case.

Indeed.

> >  2/ Allows force to use GNU gettext in musl configurations. This simply
> >     consists in:
> > 
> >  config BR2_NEEDS_GETTEXT
> >         bool
> > -       default y if BR2_TOOLCHAIN_USES_UCLIBC
> > +       default y if BR2_TOOLCHAIN_USES_UCLIBC || BR2_TOOLCHAIN_USES_MUSL
> > 
> >      I have tested this solution and it does work (obviously).
> > 
> >      The drawback is obviously that we are going to build/install GNU
> >      gettext even for cases where the internal gettext implementation
> >      of musl would have been sufficient.
> 
>  That sounds like a bad idea. What's the point of using musl if you're going to
> add a bunch of GNU extensions to it?

Absolutely, hence me not wanting to do something like that.

>  Another option is that we add a different flag to indicate the dependency on
> libintl:
> 
> config BR2_PACKAGE_GETTEXT_WITH_LIBINTL
> 	bool
> 	default y
> 	depends on BR2_PACKAGE_GETTEXT
> 	depends on BR2_STATIC_LIBS

Why this depends on BR2_STATIC_LIBS ? Linking against libintl is also
needed in dynamic linking scenarios.

The build failure of
http://autobuild.buildroot.org/results/ede/ede5ee3316ea2b6790dcb930a6bc71adc8922bfc/
requires linking with libintl, even if it's a dynamic linking
configuration.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] musl/gettext issue

[Arnout Vandecappelle]

On 18-09-15 09:41, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 18 Sep 2015 00:27:21 +0200, Arnout Vandecappelle wrote:
[snip]
>>  Another option is that we add a different flag to indicate the dependency on
>> libintl:
>>
>> config BR2_PACKAGE_GETTEXT_WITH_LIBINTL
>> 	bool
>> 	default y
>> 	depends on BR2_PACKAGE_GETTEXT
>> 	depends on BR2_STATIC_LIBS
> 
> Why this depends on BR2_STATIC_LIBS ? Linking against libintl is also
> needed in dynamic linking scenarios.

 My bad, I just thought it was one of these cases where explicit linking was
only needed for static.


 Regards,
 Arnout

-- 
Arnout Vandecappelle      arnout dot vandecappelle at essensium dot com
Senior Embedded Software Architect . . . . . . +32-478-010353 (mobile)
Essensium, Mind division . . . . . . . . . . . . . . http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium . . . . . BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] musl/gettext issue

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >  Another option is that we add a different flag to indicate the dependency on
 > libintl:

 > config BR2_PACKAGE_GETTEXT_WITH_LIBINTL
 > 	bool
 > 	default y
 > 	depends on BR2_PACKAGE_GETTEXT
 > 	depends on BR2_STATIC_LIBS
 > 	depends on !BR2_TOOLCHAIN_USES_GLIBC


 >  HTTPING_LDFLAGS = $(TARGET_LDFLAGS) \
 >         $(if $(BR2_PACKAGE_GETTEXT_WITH_LIBINTL),-lintl) \


 >  There's just about 30 packages that ned to be adapted.

Yes, I guess that's the best solution.

-- 
Bye, Peter Korsgaard

[Buildroot] musl/gettext issue

[Jörg Krause]

On Mi, 2015-09-16 at 22:16 +0200, Thomas Petazzoni wrote:
> Hello,
> 
> Today, I started looking at:
> 
> ???http://autobuild.buildroot.org/results/ede/ede5ee3316ea2b6790dcb93
> 0a6bc71adc8922bfc/build-end.log
> 
> Which, in appearance, seems to be the traditional missing -lintl when
> linking. But further investigation has revealed a slightly more
> complicated problem: how do we handle GNU gettext vs. musl.
> 
> Until, we were handling GNU gettext for glibc and uClibc:
> 
> ?* With glibc, the gettext handling is built into the C library. So
> the
> ???separate GNU gettext package (for the target) is not needed, and
> if
> ???it ever gets built, it detects that the C library has the
> necessary
> ???features, and therefore it doesn't build/install a libintl.so
> ???library and its header files.
> 
> ?* With uclibc, the gettext handling is not part of the C library, so
> ???we have to build the GNU gettext package when gettext support is
> ???needed. In this case, the GNU gettext package detects that the C
> ???library does not provide the gettext functionality, and it will
> ???build/install libintl.so and its header files.
> 
> This is why we have BR2_NEEDS_GETTEXT which is true when we use
> uClibc
> and false when we use uClibc, and BR2_NEEDS_GETTEXT_IF_LOCALE which
> is
> true when BR2_NEEDS_GETTEXT is true (i.e uClibc) *and* locale support
> is enabled.
> 
> So far, so good.
> 
> Now, enter musl. It does have an internal gettext implementation.
> However, it is not recognized by GNU gettext has a correct
> implementation, so when you build GNU gettext in a musl system, it
> does
> build/install libintl.so and its header files.
> 
> So for httping, two scenarios are possible:
> 
> ?1/ httping is built alone against musl. No problem: the gettext
> ????functions are part of the C library, everything works fine.
> 
> ?2/ httping is built *after* GNU gettext has been built. Since GNU
> ????gettext will replace the libintl.h of musl by its own one, the
> ????symbols from the GNU gettext libintl.so will be used, so we must
> ????link with -lintl explicitly. Which we are not doing, since
> ????htting.mk does:
> 
> ???????$(if $(BR2_NEEDS_GETTEXT),-lintl)
> 
> ????And BR2_NEEDS_GETTEXT is false for musl.
> 
> So, I initially tried:
> 
> -???????$(if $(BR2_NEEDS_GETTEXT),-lintl)
> +???????$(if $(BR2_PACKAGE_GETTEXT),-lintl)
> 
> With the reasoning that if GNU gettext is available, we want to use
> it,
> and if it's not available, then we'll not use it.
> 
> But that doesn't work with a glibc configuration: BR2_PACKAGE_GETTEXT
> can be enabled, but we don't have a libintl library, because GNU
> gettext doesn't build one in a glibc configuration. We try to link
> against libintl, but it's not there, and the build fails.
> 
> So, I see two possible options here:
> 
> ?1/ Simply do not allow the GNU gettext package to be built with
> glibc
> ????and musl since they provide the gettext functionality internally.
> 
> ????The only problem with this approach is that while httping is
> happy
> ????with the POSIX compliant gettext functionality of musl, some
> other
> ????programs such as Bison
> ????(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786885) will
> ????really need GNU gettext functionality.
> 
> ?2/ Allows force to use GNU gettext in musl configurations. This
> simply
> ????consists in:
> 
> ?config BR2_NEEDS_GETTEXT
> ????????bool
> -???????default y if BR2_TOOLCHAIN_USES_UCLIBC
> +???????default y if BR2_TOOLCHAIN_USES_UCLIBC ||
> BR2_TOOLCHAIN_USES_MUSL
> 
> ?????I have tested this solution and it does work (obviously).
> 
> ?????The drawback is obviously that we are going to build/install GNU
> ?????gettext even for cases where the internal gettext implementation
> ?????of musl would have been sufficient.
> 
> Do you see some other options? Any opinion between the two proposed
> options?

I have the same problem building the package mtd/host-mtd with musl.
mkfs.ubifs needs to be linked with lintl in this case and host-mtd not.

The option to use?BR2_PACKAGE_GETTEXT_WITH_LIBINTL as suggested by
Arnout looks good to me. Anybody cares about proposing a patch?

Best regards
J?rg Krause?

[Buildroot] musl/gettext issue

[Jörg Krause]

Hello,

On Mi, 2015-09-16 at 22:16 +0200, Thomas Petazzoni wrote:
> Hello,
> 
> Today, I started looking at:
> 
> ???http://autobuild.buildroot.org/results/ede/ede5ee3316ea2b6790dcb93
> 0a6bc71adc8922bfc/build-end.log
> 
> Which, in appearance, seems to be the traditional missing -lintl when
> linking. But further investigation has revealed a slightly more
> complicated problem: how do we handle GNU gettext vs. musl.
> 
> Until, we were handling GNU gettext for glibc and uClibc:
> 
> ?* With glibc, the gettext handling is built into the C library. So
> the
> ???separate GNU gettext package (for the target) is not needed, and
> if
> ???it ever gets built, it detects that the C library has the
> necessary
> ???features, and therefore it doesn't build/install a libintl.so
> ???library and its header files.
> 
> ?* With uclibc, the gettext handling is not part of the C library, so
> ???we have to build the GNU gettext package when gettext support is
> ???needed. In this case, the GNU gettext package detects that the C
> ???library does not provide the gettext functionality, and it will
> ???build/install libintl.so and its header files.
> 
> This is why we have BR2_NEEDS_GETTEXT which is true when we use
> uClibc
> and false when we use uClibc, and BR2_NEEDS_GETTEXT_IF_LOCALE which
> is
> true when BR2_NEEDS_GETTEXT is true (i.e uClibc) *and* locale support
> is enabled.
> 
> So far, so good.
> 
> Now, enter musl. It does have an internal gettext implementation.
> However, it is not recognized by GNU gettext has a correct
> implementation, so when you build GNU gettext in a musl system, it
> does
> build/install libintl.so and its header files.
> 
> So for httping, two scenarios are possible:
> 
> ?1/ httping is built alone against musl. No problem: the gettext
> ????functions are part of the C library, everything works fine.
> 
> ?2/ httping is built *after* GNU gettext has been built. Since GNU
> ????gettext will replace the libintl.h of musl by its own one, the
> ????symbols from the GNU gettext libintl.so will be used, so we must
> ????link with -lintl explicitly. Which we are not doing, since
> ????htting.mk does:
> 
> ???????$(if $(BR2_NEEDS_GETTEXT),-lintl)
> 
> ????And BR2_NEEDS_GETTEXT is false for musl.
> 
> So, I initially tried:
> 
> -???????$(if $(BR2_NEEDS_GETTEXT),-lintl)
> +???????$(if $(BR2_PACKAGE_GETTEXT),-lintl)
> 
> With the reasoning that if GNU gettext is available, we want to use
> it,
> and if it's not available, then we'll not use it.
> 
> But that doesn't work with a glibc configuration: BR2_PACKAGE_GETTEXT
> can be enabled, but we don't have a libintl library, because GNU
> gettext doesn't build one in a glibc configuration. We try to link
> against libintl, but it's not there, and the build fails.
> 
> So, I see two possible options here:
> 
> ?1/ Simply do not allow the GNU gettext package to be built with
> glibc
> ????and musl since they provide the gettext functionality internally.
> 
> ????The only problem with this approach is that while httping is
> happy
> ????with the POSIX compliant gettext functionality of musl, some
> other
> ????programs such as Bison
> ????(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=786885) will
> ????really need GNU gettext functionality.
> 
> ?2/ Allows force to use GNU gettext in musl configurations. This
> simply
> ????consists in:
> 
> ?config BR2_NEEDS_GETTEXT
> ????????bool
> -???????default y if BR2_TOOLCHAIN_USES_UCLIBC
> +???????default y if BR2_TOOLCHAIN_USES_UCLIBC ||
> BR2_TOOLCHAIN_USES_MUSL
> 
> ?????I have tested this solution and it does work (obviously).
> 
> ?????The drawback is obviously that we are going to build/install GNU
> ?????gettext even for cases where the internal gettext implementation
> ?????of musl would have been sufficient.
> 
> Do you see some other options? Any opinion between the two proposed
> options?

How do we deal with this issue?

J?rg

[Buildroot] musl/gettext issue

[Bernd Kuhls]

Am Wed, 16 Sep 2015 22:16:36 +0200 schrieb Thomas Petazzoni:

> Now, enter musl. It does have an internal gettext implementation.
> However, it is not recognized by GNU gettext has a correct
> implementation, so when you build GNU gettext in a musl system, it does
> build/install libintl.so and its header files.

Hi Thomas,

is there already a solution for this problem?
I saw the bug with aumix-2.8 which I could solve by bumping the package 
to 2.9.1. Now I am stuck with http://autobuild.buildroot.net/?reason=gdk-
pixbuf-2.32.3

Is this bug report related to our problem?
http://savannah.gnu.org/bugs/?46436

Regards, Bernd

[Buildroot] musl/gettext issue

[Thomas Petazzoni]

Hello,

On Tue, 26 Jan 2016 22:25:11 +0100, Bernd Kuhls wrote:
> Am Wed, 16 Sep 2015 22:16:36 +0200 schrieb Thomas Petazzoni:
> 
> > Now, enter musl. It does have an internal gettext implementation.
> > However, it is not recognized by GNU gettext has a correct
> > implementation, so when you build GNU gettext in a musl system, it does
> > build/install libintl.so and its header files.
> 
> Hi Thomas,
> 
> is there already a solution for this problem?

No, I personally haven't had the time to work on this topic, and nobody
else did, apparently.

> I saw the bug with aumix-2.8 which I could solve by bumping the package 
> to 2.9.1. Now I am stuck with http://autobuild.buildroot.net/?reason=gdk-
> pixbuf-2.32.3
> 
> Is this bug report related to our problem?
> http://savannah.gnu.org/bugs/?46436

Yes, it seems related, at least it reflects very well the fact that
musl provides a gettext implementation, but gettext.m4 doesn't
recognize it as being a valid implementation.

I see
http://lists.gnu.org/archive/html/bug-gettext/2015-04/msg00002.html
hasn't got much feedback on the gettext mailing list, so we'll probably
have to dig into the topic ourselves, and find some solution.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] musl/gettext issue

[Bernd Kuhls]

Am Tue, 26 Jan 2016 22:29:58 +0100 schrieb Thomas Petazzoni:

> No, I personally haven't had the time to work on this topic, and nobody
> else did, apparently.

Hi,

please test my ugly hack ;) http://patchwork.ozlabs.org/patch/576093/

Regards, Bernd