[Buildroot] [WIP PATCH 0/5] Static linking fixes

[Buildroot] [WIP PATCH 0/5] Static linking fixes

[Gustavo Zacarias]

As discussed on the mailing list here is my current patchset to get
static linking fixed.
This is a WIP (Work In Progress) patchset, that being said i consider
patches 1 and 3 pretty safe and ready for the tree.
Patch 2 and 4 are medium confidence level, how things should be IMHO.
Patch 5 is also how things should be however it's not extensively tested.

Gustavo Zacarias (5):
  uclibc: don't install tools to staging when static
  gcc-final: disable shared build for static
  libiconv: disable preloadable for static builds
  package infra: revert 0a4bd19f
  package infra: add -static to C/CXXFLAGS

 package/Makefile.in                |  4 +++-
 package/gcc/gcc-final/gcc-final.mk | 21 +++++++++++++++++++--
 package/libiconv/libiconv.mk       |  8 ++++++++
 package/uclibc/uclibc.mk           |  4 ++--
 4 files changed, 32 insertions(+), 5 deletions(-)

-- 
1.8.5.5

[Buildroot] [WIP PATCH 1/5] uclibc: don't install tools to staging when static

[Gustavo Zacarias]

Don't try to install ld* tools to staging when it's a static build,
there's no point and it'll be broken once we build a purely-static
uclibc.

Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 package/uclibc/uclibc.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index 717cf53..ee87009 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -522,8 +522,8 @@ define UCLIBC_INSTALL_TARGET_CMDS
 	$(UCLIBC_INSTALL_TEST_SUITE)
 endef
 
-# For FLAT binfmts (static) there are no host utils
-ifeq ($(BR2_BINFMT_FLAT),)
+# STATIC has no ld* tools, only getconf
+ifeq ($(BR2_PREFER_STATIC_LIB),)
 define UCLIBC_INSTALL_UTILS_STAGING
 	$(INSTALL) -D -m 0755 $(@D)/utils/ldd.host $(HOST_DIR)/usr/bin/ldd
 	ln -sf ldd $(HOST_DIR)/usr/bin/$(GNU_TARGET_NAME)-ldd
-- 
1.8.5.5

[Buildroot] [WIP PATCH 1/5] uclibc: don't install tools to staging when static

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Sun, 25 May 2014 19:12:56 -0300, Gustavo Zacarias wrote:
> Don't try to install ld* tools to staging when it's a static build,
> there's no point and it'll be broken once we build a purely-static
> uclibc.
> 
> Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
> ---
>  package/uclibc/uclibc.mk | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Acked-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.co>

(for -next).

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [WIP PATCH 1/5] uclibc: don't install tools to staging when static

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Sun, 25 May 2014 19:12:56 -0300, Gustavo Zacarias wrote:
> Don't try to install ld* tools to staging when it's a static build,
> there's no point and it'll be broken once we build a purely-static
> uclibc.
> 
> Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
> ---
>  package/uclibc/uclibc.mk | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Gustavo Zacarias]

Disable shared build for host-gcc-final when building for static targets.
We really want static or shared, there's no such thing as "preferring static"
since we can't choose with any degree of granularity for which packages.
And it confuses linking scripts having both available at the same time. Fixes:
http://autobuild.buildroot.net/results/c54/c54bdf88eff6d60c7001cb0e2cb6792cc75178db/

Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 package/gcc/gcc-final/gcc-final.mk | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/package/gcc/gcc-final/gcc-final.mk b/package/gcc/gcc-final/gcc-final.mk
index 622dcf2..366dc42 100644
--- a/package/gcc/gcc-final/gcc-final.mk
+++ b/package/gcc/gcc-final/gcc-final.mk
@@ -26,7 +26,7 @@ HOST_GCC_FINAL_SUBDIR = build
 
 HOST_GCC_FINAL_PRE_CONFIGURE_HOOKS += HOST_GCC_CONFIGURE_SYMLINK
 
-define  HOST_GCC_FINAL_CONFIGURE_CMDS
+define HOST_GCC_FINAL_CONFIGURE_CMDS
         (cd $(HOST_GCC_FINAL_SRCDIR) && rm -rf config.cache; \
                 $(HOST_CONFIGURE_OPTS) \
                 CFLAGS="$(HOST_CFLAGS)" \
@@ -35,7 +35,7 @@ define  HOST_GCC_FINAL_CONFIGURE_CMDS
                 ./configure \
                 --prefix="$(HOST_DIR)/usr" \
                 --sysconfdir="$(HOST_DIR)/etc" \
-                --enable-shared --enable-static \
+                --enable-static \
                 $(QUIET) $(HOST_GCC_FINAL_CONF_OPT) \
         )
 endef
@@ -54,6 +54,13 @@ HOST_GCC_FINAL_CONF_OPT = \
 	$(DISABLE_LARGEFILE) \
 	--with-build-time-tools=$(HOST_DIR)/usr/$(GNU_TARGET_NAME)/bin
 
+# Disable shared libs like libstdc++ if we do static since it confuses linking
+ifeq ($(BR2_PREFER_STATIC_LIB),y)
+HOST_GCC_FINAL_CONF_OPT += --disable-shared
+else
+HOST_GCC_FINAL_CONF_OPT += --enable-shared
+endif
+
 ifeq ($(BR2_GCC_ENABLE_OPENMP),y)
 HOST_GCC_FINAL_CONF_OPT += --enable-libgomp
 else
@@ -141,6 +148,15 @@ endif
 endif
 
 ifneq ($(HOST_GCC_FINAL_USR_LIBS),)
+ifeq ($(BR2_PREFER_STATIC_LIB),y)
+define HOST_GCC_FINAL_INSTALL_USR_LIBS
+	mkdir -p $(TARGET_DIR)/usr/lib
+	for i in $(HOST_GCC_FINAL_USR_LIBS) ; do \
+		cp -dpf $(HOST_DIR)/usr/$(GNU_TARGET_NAME)/lib*/$${i}.a \
+			$(STAGING_DIR)/usr/lib/ ; \
+	done
+endef
+else
 define HOST_GCC_FINAL_INSTALL_USR_LIBS
 	mkdir -p $(TARGET_DIR)/usr/lib
 	for i in $(HOST_GCC_FINAL_USR_LIBS) ; do \
@@ -152,6 +168,7 @@ define HOST_GCC_FINAL_INSTALL_USR_LIBS
 			$(TARGET_DIR)/usr/lib/ ; \
 	done
 endef
+endif
 HOST_GCC_FINAL_POST_INSTALL_HOOKS += HOST_GCC_FINAL_INSTALL_USR_LIBS
 endif
 
-- 
1.8.5.5

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Sun, 25 May 2014 19:12:57 -0300, Gustavo Zacarias wrote:
> Disable shared build for host-gcc-final when building for static targets.
> We really want static or shared, there's no such thing as "preferring static"
> since we can't choose with any degree of granularity for which packages.
> And it confuses linking scripts having both available at the same time. Fixes:
> http://autobuild.buildroot.net/results/c54/c54bdf88eff6d60c7001cb0e2cb6792cc75178db/

While on principle I'm probably fine with the patch, I kinda disagree
that it fixes the autobuilder issues. Because the same autobuilder
issue will appear with an external toolchain that has both libstdc++.so
and libstdc++.a.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Gustavo Zacarias]

On 05/26/2014 06:37 AM, Thomas Petazzoni wrote:

> While on principle I'm probably fine with the patch, I kinda disagree
> that it fixes the autobuilder issues. Because the same autobuilder
> issue will appear with an external toolchain that has both libstdc++.so
> and libstdc++.a.

This is a way to fix it for internal toolchains, i can build up to mpd
static with audiofile support with this, but it's not complete, that's
for sure.
Regards.

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Mon, 26 May 2014 07:04:14 -0300, Gustavo Zacarias wrote:

> > While on principle I'm probably fine with the patch, I kinda disagree
> > that it fixes the autobuilder issues. Because the same autobuilder
> > issue will appear with an external toolchain that has both libstdc++.so
> > and libstdc++.a.
> 
> This is a way to fix it for internal toolchains, i can build up to mpd
> static with audiofile support with this, but it's not complete, that's
> for sure.

Yeah, I read on the rest of your patches, and the patch changing
--static to -static indeed makes it clear that it fixes the problem for
external toolchains. IMO, switching from --static to -static is the
real fix: the build process should not be confused by the presence of
libstdc++.so even if we're doing a pure static build.

Also, about reverting back from --static to -static, would it be useful
to contact Andy Kennedy to have more details about why he made the
change? His commit log lacks the details on why he changed from -static
to --static.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Gustavo Zacarias]

On 05/26/2014 07:07 AM, Thomas Petazzoni wrote:

> Yeah, I read on the rest of your patches, and the patch changing
> --static to -static indeed makes it clear that it fixes the problem for
> external toolchains. IMO, switching from --static to -static is the
> real fix: the build process should not be confused by the presence of
> libstdc++.so even if we're doing a pure static build.
> 
> Also, about reverting back from --static to -static, would it be useful
> to contact Andy Kennedy to have more details about why he made the
> change? His commit log lacks the details on why he changed from -static
> to --static.

In tests for the internal toolchain audiofile builds but mpd fails to
link with that change alone (+host-gcc-final static works, actually with
host-gcc-fnial static alone it works).
For an external toolchain it works with the -static change.
Regards.

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Sun, 25 May 2014 19:12:57 -0300, Gustavo Zacarias wrote:

>  ifneq ($(HOST_GCC_FINAL_USR_LIBS),)
> +ifeq ($(BR2_PREFER_STATIC_LIB),y)
> +define HOST_GCC_FINAL_INSTALL_USR_LIBS
> +	mkdir -p $(TARGET_DIR)/usr/lib
> +	for i in $(HOST_GCC_FINAL_USR_LIBS) ; do \
> +		cp -dpf $(HOST_DIR)/usr/$(GNU_TARGET_NAME)/lib*/$${i}.a \
> +			$(STAGING_DIR)/usr/lib/ ; \
> +	done
> +endef

No, we want to install .a files as well in my opinion. In fact, the
patch series I have to rename BR2_PREFER_STATIC_LIB makes things even
clearer, with three possibilities: pure static, static and shared, pure
shared.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Gustavo Zacarias]

On 05/26/2014 06:38 AM, Thomas Petazzoni wrote:

> No, we want to install .a files as well in my opinion. In fact, the
> patch series I have to rename BR2_PREFER_STATIC_LIB makes things even
> clearer, with three possibilities: pure static, static and shared, pure
> shared.

Sorry but i didn't see your patchset so relax a bit :)
That being said do we really want a pure dyn scenario? Since we purge .a
files from the target anyway since they're not required for runtime it
wouldn't hurt a bit to keep the dual mode and just static i think.
Regards.

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Mon, 26 May 2014 07:06:08 -0300, Gustavo Zacarias wrote:

> > No, we want to install .a files as well in my opinion. In fact, the
> > patch series I have to rename BR2_PREFER_STATIC_LIB makes things
> > even clearer, with three possibilities: pure static, static and
> > shared, pure shared.
> 
> Sorry but i didn't see your patchset so relax a bit :)

Yeah, no problem :)

> That being said do we really want a pure dyn scenario? Since we
> purge .a files from the target anyway since they're not required for
> runtime it wouldn't hurt a bit to keep the dual mode and just static
> i think. Regards.

There is an advantage with the pure dynamic scenario: build time.

When you have dyn+static, you build all object files (for packages that
do it correctly): once for static without -fPIC, and once for dynamic
with -fPIC. Here is what I have in my commit log about this:

    For example, a static+shared build of libglib2 takes 1 minutes and
    59 seconds, with a final build directory of 96 MB. A shared-only
    build of libglib2 takes only 1 minutes and 31 seconds (almost a 25%
    reduction of the build time), and the final build directory weights
    89 MB (a reduction of almost 8%).

A 25% reduction in build time is certainly nice for users who don't
care about static libraries, no?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Gustavo Zacarias]

On 05/26/2014 07:24 AM, Thomas Petazzoni wrote:
> There is an advantage with the pure dynamic scenario: build time.
> 
> When you have dyn+static, you build all object files (for packages that
> do it correctly): once for static without -fPIC, and once for dynamic
> with -fPIC. Here is what I have in my commit log about this:
> 
>     For example, a static+shared build of libglib2 takes 1 minutes and
>     59 seconds, with a final build directory of 96 MB. A shared-only
>     build of libglib2 takes only 1 minutes and 31 seconds (almost a 25%
>     reduction of the build time), and the final build directory weights
>     89 MB (a reduction of almost 8%).
> 
> A 25% reduction in build time is certainly nice for users who don't
> care about static libraries, no?

Ok, so then we can go for it, i don't think anyone will care about the
space saving though, but time is a whole different matter.
Regards.

[Buildroot] [WIP PATCH 2/5] gcc-final: disable shared build for static

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Mon, 26 May 2014 08:03:48 -0300, Gustavo Zacarias wrote:

> >     For example, a static+shared build of libglib2 takes 1 minutes
> > and 59 seconds, with a final build directory of 96 MB. A shared-only
> >     build of libglib2 takes only 1 minutes and 31 seconds (almost a
> > 25% reduction of the build time), and the final build directory
> > weights 89 MB (a reduction of almost 8%).
> > 
> > A 25% reduction in build time is certainly nice for users who don't
> > care about static libraries, no?
> 
> Ok, so then we can go for it, i don't think anyone will care about the
> space saving though, but time is a whole different matter.

Agreed, the disk space saving nobody really cares, especially if it's
only ~8%. But a build time saving of 25% is quite nice, in my opinion.
Normally, all packages using libtool should be affected, but I haven't
checked.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [WIP PATCH 3/5] libiconv: disable preloadable for static builds

[Gustavo Zacarias]

Disable the preloadable libiconv variant when building for static
targets since that's not possible.

Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 package/libiconv/libiconv.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/package/libiconv/libiconv.mk b/package/libiconv/libiconv.mk
index ee313ce..d2209bc 100644
--- a/package/libiconv/libiconv.mk
+++ b/package/libiconv/libiconv.mk
@@ -8,6 +8,14 @@ LIBICONV_VERSION = 1.14
 LIBICONV_SITE = $(BR2_GNU_MIRROR)/libiconv
 LIBICONV_INSTALL_STAGING = YES
 
+# Preloadable libiconv is for dynamic targets only
+ifeq ($(BR2_PREFER_STATIC_LIB),y)
+define LIBICONV_DISABLE_PRELOAD
+	$(SED) '/preload/d' $(@D)/Makefile.in
+endef
+endif
+LIBICONV_PRE_CONFIGURE_HOOKS += LIBICONV_DISABLE_PRELOAD
+
 # Remove not used preloadable libiconv.so
 define LIBICONV_TARGET_REMOVE_PRELOADABLE_LIBS
 	rm -f $(TARGET_DIR)/usr/lib/preloadable_libiconv.so
-- 
1.8.5.5

[Buildroot] [WIP PATCH 4/5] package infra: revert 0a4bd19f

[Gustavo Zacarias]

Revert commit 0a4bd19f4a136930c611027942ce43124a81c5cb
Using --static is not documented and actually breaks static linking
according to my tests.
Fixes:
http://autobuild.buildroot.net/results/327/327c18db4e5d0ddc2c72a4684e103c19a1405e50/
...for external toolchains

Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 package/Makefile.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 2fc3aa7..ec1f22b 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -373,7 +373,7 @@ endif
 
 ifeq ($(BR2_PREFER_STATIC_LIB),y)
 SHARED_STATIC_LIBS_OPTS=--enable-static --disable-shared
-TARGET_LDFLAGS += --static
+TARGET_LDFLAGS += -static
 else
 SHARED_STATIC_LIBS_OPTS=--enable-static --enable-shared
 endif
-- 
1.8.5.5

[Buildroot] [WIP PATCH 5/5] package infra: add -static to C/CXXFLAGS

[Gustavo Zacarias]

Generic infra packages might not use LDFLAGS at all so add -static for
static builds to CFLAGS and CXXFLAGS too.

Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
---
 package/Makefile.in | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/Makefile.in b/package/Makefile.in
index ec1f22b..ec8fb1b 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -373,6 +373,8 @@ endif
 
 ifeq ($(BR2_PREFER_STATIC_LIB),y)
 SHARED_STATIC_LIBS_OPTS=--enable-static --disable-shared
+TARGET_CFLAGS += -static
+TARGET_CXXFLAGS += -static
 TARGET_LDFLAGS += -static
 else
 SHARED_STATIC_LIBS_OPTS=--enable-static --enable-shared
-- 
1.8.5.5

[Buildroot] [WIP PATCH 0/5] Static linking fixes

[Thomas De Schampheleire]

Hi Gustavo,

On Mon, May 26, 2014 at 12:12 AM, Gustavo Zacarias
<gustavo@zacarias.com.ar> wrote:
> As discussed on the mailing list here is my current patchset to get
> static linking fixed.
> This is a WIP (Work In Progress) patchset, that being said i consider
> patches 1 and 3 pretty safe and ready for the tree.
> Patch 2 and 4 are medium confidence level, how things should be IMHO.
> Patch 5 is also how things should be however it's not extensively tested.
>
> Gustavo Zacarias (5):
>   uclibc: don't install tools to staging when static
>   gcc-final: disable shared build for static
>   libiconv: disable preloadable for static builds
>   package infra: revert 0a4bd19f
>   package infra: add -static to C/CXXFLAGS
>
>  package/Makefile.in                |  4 +++-
>  package/gcc/gcc-final/gcc-final.mk | 21 +++++++++++++++++++--
>  package/libiconv/libiconv.mk       |  8 ++++++++
>  package/uclibc/uclibc.mk           |  4 ++--
>  4 files changed, 32 insertions(+), 5 deletions(-)

Your patches do not yet go as far as touching the
BR2_PREFER_STATIC_LIB variable. We have talked about this in the past
but as far as I know there is no-one that has actually wrote patches
for it.
See also http://www.elinux.org/Buildroot:DeveloperDaysFOSDEM2014#BR2_PREFER_STATIC_LIB_refactoring

It would be awesome if this refactoring could be taken up by someone
in the 2014.08 cycle, but I don't know if anyone has time for this...

Best regards,
Thomas

[Buildroot] [WIP PATCH 0/5] Static linking fixes

[Gustavo Zacarias]

On 05/26/2014 05:13 AM, Thomas De Schampheleire wrote:

Hi.

> Your patches do not yet go as far as touching the
> BR2_PREFER_STATIC_LIB variable. We have talked about this in the past
> but as far as I know there is no-one that has actually wrote patches
> for it.
> See also http://www.elinux.org/Buildroot:DeveloperDaysFOSDEM2014#BR2_PREFER_STATIC_LIB_refactoring
> 
> It would be awesome if this refactoring could be taken up by someone
> in the 2014.08 cycle, but I don't know if anyone has time for this...

Well, that's why it's called a WIP :)
Also Thomas P. told me he has done something about it so i would just be
reinventing the wheel.
And if it's just a rename it's a pretty simple "find . -type f -exec sed
-i -e "s/BR2_PREFER_STATIC_LIB/BR2_SOMETHING/" {} \;" with some hand
editing of toplevel Config.in to adjust the description as well.
Regards.

[Buildroot] [WIP PATCH 0/5] Static linking fixes

[Thomas Petazzoni]

Dear Gustavo Zacarias,

On Sun, 25 May 2014 19:12:55 -0300, Gustavo Zacarias wrote:
> As discussed on the mailing list here is my current patchset to get
> static linking fixed.
> This is a WIP (Work In Progress) patchset, that being said i consider
> patches 1 and 3 pretty safe and ready for the tree.
> Patch 2 and 4 are medium confidence level, how things should be IMHO.
> Patch 5 is also how things should be however it's not extensively tested.
> 
> Gustavo Zacarias (5):
>   uclibc: don't install tools to staging when static

This one was already applied.

>   gcc-final: disable shared build for static
>   libiconv: disable preloadable for static builds
>   package infra: revert 0a4bd19f
>   package infra: add -static to C/CXXFLAGS

I've applied those four patches. On the gcc-final patch, I did a small
change to factorize the static library installation logic in one place.
On the libiconv patch, I did a little bit more changes: completely
disable the build of the preloadable library, since we were anyway
removing it from staging and target. It allowed to simplify the patch a
bit.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com