[Buildroot] [RFC 0/6] Support 32-bit binaries on a 64-bit architecture

[Buildroot] [RFC 0/6] Support 32-bit binaries on a 64-bit architecture

[Markus Mayer]

This series contains a proposal for supporting 32-bit libraries and
binaries on a 64-bit platform.

There are some limitations and prerequisites as to the scope this has
been tested.

- It has only been tested on ARM/ARM64.
- It has only been tested with an external toolchain
  (https://github.com/Broadcom/stbgcc-6.3/releases).
- It requires that the Aarch64 compiler be compiled with "multi-lib"
  enabled. This ensures that the sysroot doesn't contain a lib64 ->
  lib symlink. Instead, lib and lib64 are separate directories.
- It has only been tested with a fairly limited number of packages.
  There might be other packages that don't correctly pass --libdir,
  similar to bzip2 as mentioned below.

The patches in this series make the following changes, in this order:

- ensure that check-bin-arch allows Aarch32 binaries on Aarch64
- introduce some new BR2 configuration variables (primarily for the
  32-bit sysroot location on the host and where system libraries live
  by default on the target)
- tell the Buildroot core not to create lib32 or lib64 symlink when
  32-bit support is enabled
- tell the Buildroot core to copy the 32-bit sysroot when 32-bit support
  is enabled
- introduce the $(LIBDIR) variable to the bzip2 package, so we can tell
  it that libbz2.so should go in /usr/lib64 when that is desired
- modify the package core to pass /usr/lib (or /usr/lib64) as the library
  location for the system

The result is that all 64-bit shared libraries should be installed into
/usr/lib64. The 32-bit libraries are copied from the 32-bit sysroot on
the host into /usr/lib.

I am quite certain this submission will only be a first step, but it does
look like there are others who are faced with the same issue.

Markus Mayer (6):
  support/scripts/check-bin-arch: improve architecture check
  system/Config.in: add configuration options for 32-bit library support
  core: system and toolchain: 32-bit run-time support on 64-bit platform
  core/pkg-toolchain-external: copy external 32-bit libraries to staging
  bzip2: introduce make variable $(LIBDIR)
  package: use BR2_ROOTFS_LIB_DIR for all libraries we use

 package/bzip2/0003-add-libdir-to-makefile.patch    | 36 ++++++++++++++++++++++
 .../bzip2/0004-add-libdir-to-makefile-libbz2.patch | 31 +++++++++++++++++++
 package/bzip2/bzip2.mk                             |  4 ++-
 package/dmalloc/dmalloc.mk                         |  7 +++--
 package/e2fsprogs/e2fsprogs.mk                     |  2 ++
 package/libzlib/libzlib.mk                         |  3 +-
 package/linux-tools/linux-tool-cpupower.mk.in      |  7 ++++-
 package/pciutils/pciutils.mk                       |  2 ++
 package/pkg-autotools.mk                           |  1 +
 package/readline/readline.mk                       |  2 +-
 support/scripts/check-bin-arch                     | 15 +++++++++
 system/Config.in                                   | 34 ++++++++++++++++++++
 system/system.mk                                   | 11 ++++++-
 .../toolchain-external/pkg-toolchain-external.mk   | 13 +++++++-
 15 files changed, 161 insertions(+), 10 deletions(-)
 create mode 100644 package/bzip2/0003-add-libdir-to-makefile.patch
 create mode 100644 package/bzip2/0004-add-libdir-to-makefile-libbz2.patch

-- 
2.7.4

[Buildroot] [RFC 1/6] support/scripts/check-bin-arch: improve architecture check

[Markus Mayer]

On 64-bit targets, allow 32-bit binaries of the same architecture
(MIPS64 -> MIPS32, AArch64 -> ARM, etc.)

In order for 32-bit binaries to run on the target, the corresponding
32-bit libraries need to be present in the target's root file system
and the kernel needs to support execution of 32-bit binaries.

Signed-off-by: Markus Mayer <mmayer@broadcom.com>
---
 support/scripts/check-bin-arch | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/support/scripts/check-bin-arch b/support/scripts/check-bin-arch
index 887b6613cdb8..31711769eabd 100755
--- a/support/scripts/check-bin-arch
+++ b/support/scripts/check-bin-arch
@@ -58,6 +58,21 @@ while read f; do
 		continue
 	fi
 
+	# Didn't find a straight match. See if we are on a 64-bit architecture
+	# and our binary is a 32-bit binary of the same architecture.
+	if [[ "${arch_name}" =~ 64$ ]]; then
+		if [[ ${arch_name} = "AArch64" ]]; then
+			arch32='ARM'
+		else
+			arch32=${arch_name/64/32}
+		fi
+
+		if [ "${arch}" = "${arch32}" ]; then
+			echo "Accepting ${arch32} binary on ${arch_name} ($f)"
+			continue
+		fi
+	fi
+
 	printf 'ERROR: architecture for "%s" is "%s", should be "%s"\n' \
 	       "${f}" "${arch}" "${arch_name}"
 
-- 
2.7.4

[Buildroot] [RFC 2/6] system/Config.in: add configuration options for 32-bit library support

[Markus Mayer]

In order to support 32-bit shared libraries in a 64-bit root file
system, we need several configuration options that let us tell
buildroot which libraries to put where.

Specifically, we introduce the following:

BR2_ROOTFS_LIB_DIR
------------------

Tells buildroot where to put the main system libraries. On 32-bit
setups, this will mean 32-bit libraries. On 64-bit setups, this
setting will apply to 64-bit libraries. Typical values for
BR2_ROOTFS_LIB_DIR are "lib", "lib32" or "lib64".

The default value for 32-bit setups is "lib". For 64-bit setups it is
"lib64".

BR2_ROOTFS_LIB32_DIR
--------------------

This tells buildroot where to put 32-bit libraries on a 64-bit system.
Typical values for BR2_ROOTFS_LIB32_DIR are "lib" or "lib32". The
default value is "lib".

On 32-bit setups, this variable is not available. On 64-bit systems
with 32-bit support disabled, this variable will not be used.

BR2_ROOTFS_RUNTIME32
--------------------

Tells buildroot whether it should install 32-bit shared libraries into
a 64-bit root file system. If set to false, the root file system will
not be able to support 32-bit binaries.

This variable is not available on 32-bit setups.

BR2_ROOTFS_RUNTIME32_PATH
-------------------------

Points to the sys-root of a 32-bit toolchain for the current 64-bit
architecture (i.e. an ARM sys-root on AArch64 or MIPS32 on MIPS64).

This path is used to copy all 32-bit shared libraries from the 32-bit
sys-root into the buildroot staging area. From there, they will be
included into the root file system. Care is taken that 32-bit libraries
will not interfere with 64-bit libraries already in staging.

We need the ability to specify a separate sys-root for 32-bit, because
the 64-bit sys-root may not include 32-bit libraries.

This variable is not available on 32-bit setups.

Signed-off-by: Markus Mayer <mmayer@broadcom.com>
---
 system/Config.in | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/system/Config.in b/system/Config.in
index d48cf8d8c4ed..1e9415ae6f6d 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -24,6 +24,40 @@ config BR2_ROOTFS_SKELETON_CUSTOM
 
 endchoice
 
+config BR2_ROOTFS_LIB_DIR
+	string "Name of system library directories (e.g. lib or lib64)"
+	default "lib64" if BR2_ARCH_IS_64
+	default "lib"
+	help
+	  The name of the directories where system libraries live. This will
+	  most often be "lib", "lib32" or "lib64". This name will be used in
+	  the root directory (e.g. /lib64) as well as the /usr directory (e.g.
+	  /usr/lib64).
+
+if BR2_ARCH_IS_64
+
+config BR2_ROOTFS_LIB32_DIR
+	string "Name of directory for 32-bit libraries (e.g. lib or lib32)"
+	default "lib"
+	help
+	  The name of the directories where 32-bit libraries live. This will
+	  most often be "lib" or "lib32". This name will be used in the root
+	  directory (e.g. /lib) as well as the /usr directory (e.g. /usr/lib).
+
+config BR2_ROOTFS_RUNTIME32
+	bool "Install 32-bit runtime on 64-bit system"
+	help
+	  Install 32-bit libraries and the 32-bit shared library loader on a
+	  64-bit system, so it is possible to execute 32-bit applications.
+
+config BR2_ROOTFS_RUNTIME32_PATH
+	string "Path to the 32-bit run-time"
+	help
+	  Specifies the path to the 32-bit run-time that should be copied into
+	  the root file system.
+
+endif
+
 if BR2_ROOTFS_SKELETON_CUSTOM
 
 config BR2_ROOTFS_SKELETON_CUSTOM_PATH
-- 
2.7.4

[Buildroot] [RFC 3/6] core: system and toolchain: 32-bit run-time support on 64-bit platform

[Markus Mayer]

Set up library symlinks only if we don't want to support 32-bit
libraries on 64-bit platforms.

We cannot set up sym-links such as "lib64" -> "lib" if we want to
support 32-bit binaries on a 64-bit platform. Instead, we need to keep
32-bit and 64-bit libraries in separate directories, so we must skip
generating sym-links that would prevent this.

Signed-off-by: Markus Mayer <mmayer@broadcom.com>
---
 system/system.mk                                       | 11 ++++++++++-
 toolchain/toolchain-external/pkg-toolchain-external.mk |  6 ++++++
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/system/system.mk b/system/system.mk
index ca6bf1388f39..e0d45356fe66 100644
--- a/system/system.mk
+++ b/system/system.mk
@@ -55,6 +55,14 @@ define SYSTEM_RSYNC
 		$(1)/ $(2)/
 endef
 
+ifeq ($(BR2_ROOTFS_RUNTIME32),y)
+
+define SYSTEM_LIB_SYMLINK
+	@echo "SYSTEM_LIB_SYMLINK: nothing to do"
+endef
+
+else
+
 # Make a symlink lib32->lib or lib64->lib as appropriate.
 # MIPS64/n32 requires lib32 even though it's a 64-bit arch.
 # $(1): base dir (either staging or target)
@@ -68,7 +76,8 @@ define SYSTEM_LIB_SYMLINK
 	ln -snf lib $(1)/lib32
 	ln -snf lib $(1)/usr/lib32
 endef
-endif
+endif # BR2_ARCH_IS_64
+endif # BR2_ROOTFS_RUNTIME32
 
 SYSTEM_GETTY_PORT = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_PORT))
 SYSTEM_GETTY_BAUDRATE = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_BAUDRATE))
diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index 3bf9fac4121c..d7cf97d170ca 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -448,6 +448,7 @@ define TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS
 	$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR})
 endef
 
+ifeq ($(BR2_ROOTFS_RUNTIME32),)
 # Create a symlink from (usr/)$(ARCH_LIB_DIR) to lib.
 # Note: the skeleton package additionally creates lib32->lib or lib64->lib
 # (as appropriate)
@@ -462,6 +463,11 @@ create_lib_symlinks = \
 		ln -snf $${relpath}lib "$${DESTDIR}/usr/$${ARCH_LIB_DIR}" ; \
 	fi
 
+else
+create_lib_symlinks = \
+	@echo "BR2_ROOTFS_RUNTIME32 is set, skipping lib symlinks creation"
+endif
+
 define TOOLCHAIN_EXTERNAL_CREATE_STAGING_LIB_SYMLINK
 	$(call create_lib_symlinks,$(STAGING_DIR))
 endef
-- 
2.7.4

[Buildroot] [RFC 4/6] core/pkg-toolchain-external: copy external 32-bit libraries to staging

[Markus Mayer]

We use rsync to copy the 32-bit libraries from the 32-bit sysroot into
staging.

Signed-off-by: Markus Mayer <mmayer@broadcom.com>
---
 toolchain/toolchain-external/pkg-toolchain-external.mk | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index d7cf97d170ca..0c5cd4b9f6d7 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -445,7 +445,12 @@ define TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS
 		ARCH_SUBDIR=`echo $${ARCH_SYSROOT_DIR} | sed -r -e "s:^$${SYSROOT_DIR}(.*)/$$:\1:"` ; \
 	fi ; \
 	$(call MESSAGE,"Copying external toolchain sysroot to staging...") ; \
-	$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR})
+	$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR},$${ARCH_LIB_DIR},$${SUPPORT_LIB_DIR}); \
+	if [ "$(BR2_ROOTFS_RUNTIME32)" = "y" ]; then \
+		$(call MESSAGE,"Copying external toolchain 32-bit libraries to staging...") ; \
+		mkdir -p "$${STAGING_DIR}/$(BR2_ROOTFS_LIB32_DIR)"; \
+		rsync -a --exclude '*.a' "$(BR2_ROOTFS_RUNTIME32_PATH)/$(BR2_ROOTFS_LIB32_DIR)" "$${STAGING_DIR}"; \
+	fi
 endef
 
 ifeq ($(BR2_ROOTFS_RUNTIME32),)
-- 
2.7.4

[Buildroot] [RFC 5/6] bzip2: introduce make variable $(LIBDIR)

[Markus Mayer]

In order to allow us to install libz2 into a location other than
/usr/lib, we introduce a variable called $(LIBDIR) that can be set by
the build system to an alternate location.

$(LIBDIR) defaults to /usr/lib, so the behaviour doesn't change if
$(LIBDIR) is never set.

Signed-off-by: Markus Mayer <mmayer@broadcom.com>
---
 package/bzip2/0003-add-libdir-to-makefile.patch    | 36 ++++++++++++++++++++++
 .../bzip2/0004-add-libdir-to-makefile-libbz2.patch | 31 +++++++++++++++++++
 2 files changed, 67 insertions(+)
 create mode 100644 package/bzip2/0003-add-libdir-to-makefile.patch
 create mode 100644 package/bzip2/0004-add-libdir-to-makefile-libbz2.patch

diff --git a/package/bzip2/0003-add-libdir-to-makefile.patch b/package/bzip2/0003-add-libdir-to-makefile.patch
new file mode 100644
index 000000000000..8a724484bd1e
--- /dev/null
+++ b/package/bzip2/0003-add-libdir-to-makefile.patch
@@ -0,0 +1,36 @@
+Introduce $(LIBDIR) to Makefile
+
+The variable $(LIBDIR) allows us to override where libbz2 is installed.
+
+Index: bzip2-1.0.6/Makefile
+===================================================================
+--- bzip2-1.0.6.orig/Makefile	2010-09-10 15:46:02.000000000 -0700
++++ bzip2-1.0.6/Makefile	2017-11-23 12:51:53.318896836 -0800
+@@ -19,6 +19,7 @@
+ AR=ar
+ RANLIB=ranlib
+ LDFLAGS=
++LIBDIR ?= lib
+ 
+ override CFLAGS += -Wall
+ 
+@@ -71,7 +73,7 @@
+ 
+ install: bzip2 bzip2recover
+ 	if ( test ! -d $(PREFIX)/bin ) ; then mkdir -p $(PREFIX)/bin ; fi
+-	if ( test ! -d $(PREFIX)/lib ) ; then mkdir -p $(PREFIX)/lib ; fi
++	if ( test ! -d $(PREFIX)/$(LIBDIR) ) ; then mkdir -p $(PREFIX)/$(LIBDIR) ; fi
+ 	if ( test ! -d $(PREFIX)/man ) ; then mkdir -p $(PREFIX)/man ; fi
+ 	if ( test ! -d $(PREFIX)/man/man1 ) ; then mkdir -p $(PREFIX)/man/man1 ; fi
+ 	if ( test ! -d $(PREFIX)/include ) ; then mkdir -p $(PREFIX)/include ; fi
+@@ -87,8 +89,8 @@
+ 	chmod a+r $(PREFIX)/man/man1/bzip2.1
+ 	cp -f bzlib.h $(PREFIX)/include
+ 	chmod a+r $(PREFIX)/include/bzlib.h
+-	cp -f libbz2.a $(PREFIX)/lib
+-	chmod a+r $(PREFIX)/lib/libbz2.a
++	cp -f libbz2.a $(PREFIX)/$(LIBDIR)
++	chmod a+r $(PREFIX)/$(LIBDIR)/libbz2.a
+ 	cp -f bzgrep $(PREFIX)/bin/bzgrep
+ 	ln -s -f $(PREFIX)/bin/bzgrep $(PREFIX)/bin/bzegrep
+ 	ln -s -f $(PREFIX)/bin/bzgrep $(PREFIX)/bin/bzfgrep
diff --git a/package/bzip2/0004-add-libdir-to-makefile-libbz2.patch b/package/bzip2/0004-add-libdir-to-makefile-libbz2.patch
new file mode 100644
index 000000000000..9b2f6b114fc6
--- /dev/null
+++ b/package/bzip2/0004-add-libdir-to-makefile-libbz2.patch
@@ -0,0 +1,31 @@
+Introduce $(LIBDIR) to Makefile-libbz2_so
+
+The variable $(LIBDIR) allows us to override where libbz2 is installed.
+
+Index: bzip2-1.0.6/Makefile
+===================================================================
+--- bzip2-1.0.6/Makefile-libbz2_so.orig	2017-11-23 11:08:33.067637637 -0800
++++ bzip2-1.0.6/Makefile-libbz2_so	2017-11-23 12:51:38.694728036 -0800
+@@ -23,6 +23,9 @@
+ 
+ SHELL=/bin/sh
+ CC=gcc
++
++LIBDIR ?= lib
++
+ override CFLAGS += -fpic -fPIC -Wall
+ 
+ OBJS= blocksort.sho  \
+@@ -38,9 +41,9 @@
+ 	$(CC) $(CFLAGS) -o bzip2-shared bzip2.c libbz2.so.1.0.6
+ 
+ install:
+-	install -m 0755 -D libbz2.so.1.0.6 $(PREFIX)/lib/libbz2.so.1.0.6
+-	ln -sf libbz2.so.1.0.6 $(PREFIX)/lib/libbz2.so
+-	ln -sf libbz2.so.1.0.6 $(PREFIX)/lib/libbz2.so.1.0
++	install -m 0755 -D libbz2.so.1.0.6 $(PREFIX)/$(LIBDIR)/libbz2.so.1.0.6
++	ln -sf libbz2.so.1.0.6 $(PREFIX)/$(LIBDIR)/libbz2.so
++	ln -sf libbz2.so.1.0.6 $(PREFIX)/$(LIBDIR)/libbz2.so.1.0
+ 
+ clean: 
+ 	rm -f $(OBJS) bzip2.o libbz2.so.1.0.6 libbz2.so.1.0 bzip2-shared
-- 
2.7.4

[Buildroot] [RFC 6/6] package: use BR2_ROOTFS_LIB_DIR for all libraries we use

[Markus Mayer]

Make sure to pass $(BR2_ROOTFS_LIB_DIR) as the library directory name
to all library packages we use.

Autoconf packages are easy. They only require a single added line in
pkg-autotools.mk. It is the generic packages not using autoconf that
need to be tweaked individually.

Signed-off-by: Markus Mayer <mmayer@broadcom.com>
---
 package/bzip2/bzip2.mk                        | 4 +++-
 package/dmalloc/dmalloc.mk                    | 7 ++++---
 package/e2fsprogs/e2fsprogs.mk                | 2 ++
 package/libzlib/libzlib.mk                    | 3 ++-
 package/linux-tools/linux-tool-cpupower.mk.in | 7 ++++++-
 package/pciutils/pciutils.mk                  | 2 ++
 package/pkg-autotools.mk                      | 1 +
 package/readline/readline.mk                  | 2 +-
 9 files changed, 23 insertions(+), 8 deletions(-)

diff --git a/package/bzip2/bzip2.mk b/package/bzip2/bzip2.mk
index e43533072b2d..67f0239944ca 100644
--- a/package/bzip2/bzip2.mk
+++ b/package/bzip2/bzip2.mk
@@ -39,6 +39,7 @@ endef
 ifeq ($(BR2_STATIC_LIBS),)
 define BZIP2_INSTALL_TARGET_SHARED_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) \
 		-f Makefile-libbz2_so PREFIX=$(TARGET_DIR)/usr -C $(@D) install
 endef
 endif
@@ -47,7 +48,8 @@ endif
 define BZIP2_INSTALL_TARGET_CMDS
 	rm -f $(addprefix $(TARGET_DIR)/usr/bin/,bzip2 bunzip2 bzcat)
 	$(TARGET_MAKE_ENV) $(MAKE) \
-		PREFIX=$(TARGET_DIR)/usr -C $(@D) install
+		PREFIX=$(TARGET_DIR)/usr \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) -C $(@D) install
 	$(BZIP2_INSTALL_TARGET_SHARED_CMDS)
 endef
 
diff --git a/package/dmalloc/dmalloc.mk b/package/dmalloc/dmalloc.mk
index 6ebb44c5c068..d54950fec5a1 100644
--- a/package/dmalloc/dmalloc.mk
+++ b/package/dmalloc/dmalloc.mk
@@ -50,15 +50,16 @@ DMALLOC_POST_PATCH_HOOKS += DMALLOC_POST_PATCH
 define DMALLOC_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) includedir="$(STAGING_DIR)/usr/include" \
 		bindir="$(STAGING_DIR)/usr/bin" \
-		libdir="$(STAGING_DIR)/usr/lib" \
-		shlibdir="$(STAGING_DIR)/usr/lib" \
+		libdir="$(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)" \
+		shlibdir="$(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)" \
 		infodir="$(STAGING_DIR)/usr/share/info/" \
 		-C $(@D) install
 endef
 
 ifeq ($(BR2_STATIC_LIBS),)
 define DMALLOC_INSTALL_SHARED_LIB
-	cp -dpf $(STAGING_DIR)/usr/lib/libdmalloc*.so $(TARGET_DIR)/usr/lib
+	cp -dpf $(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/libdmalloc*.so \
+		$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)
 endef
 endif
 
diff --git a/package/e2fsprogs/e2fsprogs.mk b/package/e2fsprogs/e2fsprogs.mk
index e77b10f26dae..5cf2445562c9 100644
--- a/package/e2fsprogs/e2fsprogs.mk
+++ b/package/e2fsprogs/e2fsprogs.mk
@@ -65,6 +65,8 @@ ifeq ($(BR2_nios2),y)
 E2FSPROGS_CONF_ENV += ac_cv_func_fallocate=no
 endif
 
+E2FSPROGS_CONF_ENV += PKG_CONFIG_PATH="$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/pkgconfig"
+
 # Some programs are built for the host, but use definitions guessed by
 # the configure script (i.e with the cross-compiler). Help them by
 # saying that <sys/stat.h> is available on the host, which is needed
diff --git a/package/libzlib/libzlib.mk b/package/libzlib/libzlib.mk
index eea0c12f2209..7ef79bac21b7 100644
--- a/package/libzlib/libzlib.mk
+++ b/package/libzlib/libzlib.mk
@@ -31,6 +31,7 @@ define LIBZLIB_CONFIGURE_CMDS
 		./configure \
 		$(LIBZLIB_SHARED) \
 		--prefix=/usr \
+		--libdir='$${exec_prefix}/$(BR2_ROOTFS_LIB_DIR)' \
 	)
 endef
 
@@ -65,7 +66,7 @@ endef
 # assembling the filesystem images anyway.
 ifeq ($(BR2_SHARED_LIBS),y)
 define LIBZLIB_RM_STATIC_STAGING
-	rm -f $(STAGING_DIR)/usr/lib/libz.a
+	rm -f $(STAGING_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/libz.a
 endef
 LIBZLIB_POST_INSTALL_STAGING_HOOKS += LIBZLIB_RM_STATIC_STAGING
 endif
diff --git a/package/linux-tools/linux-tool-cpupower.mk.in b/package/linux-tools/linux-tool-cpupower.mk.in
index 2a2d3e01dd5a..9a5a32efb2de 100644
--- a/package/linux-tools/linux-tool-cpupower.mk.in
+++ b/package/linux-tools/linux-tool-cpupower.mk.in
@@ -8,10 +8,13 @@ LINUX_TOOLS += cpupower
 
 CPUPOWER_DEPENDENCIES = pciutils $(TARGET_NLS_DEPENDENCIES)
 
+CPUPOWER_LDFLAGS = -L$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)
+CPUPOWER_LDFLAGS += $(TARGET_NLS_LIBS)
+
 CPUPOWER_MAKE_OPTS = CROSS=$(TARGET_CROSS) \
 	CPUFREQ_BENCH=false \
 	NLS=false \
-	LDFLAGS=$(TARGET_NLS_LIBS) \
+	LDFLAGS=$(CPUPOWER_LDFLAGS) \
 	DEBUG=false
 
 define CPUPOWER_BUILD_CMDS
@@ -30,6 +33,7 @@ define CPUPOWER_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(LINUX_DIR)/tools \
 		$(CPUPOWER_MAKE_OPTS) \
 		DESTDIR=$(STAGING_DIR) \
+		libdir=/usr/$(BR2_ROOTFS_LIB_DIR) \
 		cpupower_install
 endef
 
@@ -37,5 +41,6 @@ define CPUPOWER_INSTALL_TARGET_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(LINUX_DIR)/tools \
 		$(CPUPOWER_MAKE_OPTS) \
 		DESTDIR=$(TARGET_DIR) \
+		libdir=/usr/$(BR2_ROOTFS_LIB_DIR) \
 		cpupower_install
 endef
diff --git a/package/pciutils/pciutils.mk b/package/pciutils/pciutils.mk
index 2dd5771f2193..775425c1d5a4 100644
--- a/package/pciutils/pciutils.mk
+++ b/package/pciutils/pciutils.mk
@@ -67,12 +67,14 @@ endef
 define PCIUTILS_INSTALL_TARGET_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(PCIUTILS_MAKE_OPTS) \
 		PREFIX=$(TARGET_DIR)/usr SBINDIR=$(TARGET_DIR)/usr/bin \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) \
 		install install-lib install-pcilib
 endef
 
 define PCIUTILS_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(PCIUTILS_MAKE_OPTS) \
 		PREFIX=$(STAGING_DIR)/usr SBINDIR=$(STAGING_DIR)/usr/bin \
+		LIBDIR=$(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR) \
 		install install-lib install-pcilib
 endef
 
diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index 45de99356f5f..95f423737074 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -189,6 +189,7 @@ define $(2)_CONFIGURE_CMDS
 		--build=$$(GNU_HOST_NAME) \
 		--prefix=/usr \
 		--exec-prefix=/usr \
+		--libdir='$$$${exec_prefix}/$(BR2_ROOTFS_LIB_DIR)' \
 		--sysconfdir=/etc \
 		--localstatedir=/var \
 		--program-prefix="" \
diff --git a/package/readline/readline.mk b/package/readline/readline.mk
index cc5d4f29201e..decb88cfa73c 100644
--- a/package/readline/readline.mk
+++ b/package/readline/readline.mk
@@ -27,7 +27,7 @@ READLINE_POST_INSTALL_TARGET_HOOKS += READLINE_INSTALL_INPUTRC
 ifneq ($(BR2_STATIC_LIBS),y)
 # libraries get installed read only, so strip fails
 define READLINE_INSTALL_FIXUPS_SHARED
-	chmod +w $(addprefix $(TARGET_DIR)/usr/lib/,libhistory.so.* libreadline.so.*)
+	chmod +w $(addprefix $(TARGET_DIR)/usr/$(BR2_ROOTFS_LIB_DIR)/,libhistory.so.* libreadline.so.*)
 endef
 READLINE_POST_INSTALL_TARGET_HOOKS += READLINE_INSTALL_FIXUPS_SHARED
 endif
-- 
2.7.4

[Buildroot] [RFC 0/6] Support 32-bit binaries on a 64-bit architecture

[Baruch Siach]

Hi Markus,

On Thu, Feb 15, 2018 at 04:56:06PM -0800, Markus Mayer wrote:
> This series contains a proposal for supporting 32-bit libraries and
> binaries on a 64-bit platform.
> 
> There are some limitations and prerequisites as to the scope this has
> been tested.
> 
> - It has only been tested on ARM/ARM64.
> - It has only been tested with an external toolchain
>   (https://github.com/Broadcom/stbgcc-6.3/releases).
> - It requires that the Aarch64 compiler be compiled with "multi-lib"
>   enabled. This ensures that the sysroot doesn't contain a lib64 ->
>   lib symlink. Instead, lib and lib64 are separate directories.
> - It has only been tested with a fairly limited number of packages.
>   There might be other packages that don't correctly pass --libdir,
>   similar to bzip2 as mentioned below.

There is one thing I am missing here. AFAICS, this series does not add support 
for building 32-bit libraries in addition to their 64-bit versions. So the 
32-bit binaries that are to run on target must have their dependencies 
satisfied from the toolchain provided libraries. Is that correct?

baruch

> The patches in this series make the following changes, in this order:
> 
> - ensure that check-bin-arch allows Aarch32 binaries on Aarch64
> - introduce some new BR2 configuration variables (primarily for the
>   32-bit sysroot location on the host and where system libraries live
>   by default on the target)
> - tell the Buildroot core not to create lib32 or lib64 symlink when
>   32-bit support is enabled
> - tell the Buildroot core to copy the 32-bit sysroot when 32-bit support
>   is enabled
> - introduce the $(LIBDIR) variable to the bzip2 package, so we can tell
>   it that libbz2.so should go in /usr/lib64 when that is desired
> - modify the package core to pass /usr/lib (or /usr/lib64) as the library
>   location for the system
> 
> The result is that all 64-bit shared libraries should be installed into
> /usr/lib64. The 32-bit libraries are copied from the 32-bit sysroot on
> the host into /usr/lib.
> 
> I am quite certain this submission will only be a first step, but it does
> look like there are others who are faced with the same issue.
> 
> Markus Mayer (6):
>   support/scripts/check-bin-arch: improve architecture check
>   system/Config.in: add configuration options for 32-bit library support
>   core: system and toolchain: 32-bit run-time support on 64-bit platform
>   core/pkg-toolchain-external: copy external 32-bit libraries to staging
>   bzip2: introduce make variable $(LIBDIR)
>   package: use BR2_ROOTFS_LIB_DIR for all libraries we use
> 
>  package/bzip2/0003-add-libdir-to-makefile.patch    | 36 ++++++++++++++++++++++
>  .../bzip2/0004-add-libdir-to-makefile-libbz2.patch | 31 +++++++++++++++++++
>  package/bzip2/bzip2.mk                             |  4 ++-
>  package/dmalloc/dmalloc.mk                         |  7 +++--
>  package/e2fsprogs/e2fsprogs.mk                     |  2 ++
>  package/libzlib/libzlib.mk                         |  3 +-
>  package/linux-tools/linux-tool-cpupower.mk.in      |  7 ++++-
>  package/pciutils/pciutils.mk                       |  2 ++
>  package/pkg-autotools.mk                           |  1 +
>  package/readline/readline.mk                       |  2 +-
>  support/scripts/check-bin-arch                     | 15 +++++++++
>  system/Config.in                                   | 34 ++++++++++++++++++++
>  system/system.mk                                   | 11 ++++++-
>  .../toolchain-external/pkg-toolchain-external.mk   | 13 +++++++-
>  15 files changed, 161 insertions(+), 10 deletions(-)
>  create mode 100644 package/bzip2/0003-add-libdir-to-makefile.patch
>  create mode 100644 package/bzip2/0004-add-libdir-to-makefile-libbz2.patch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.52.368.4656, http://www.tkos.co.il -

[Buildroot] [RFC 0/6] Support 32-bit binaries on a 64-bit architecture

[Thomas Petazzoni]

Hello,

On Fri, 16 Feb 2018 08:45:57 +0200, Baruch Siach wrote:

> On Thu, Feb 15, 2018 at 04:56:06PM -0800, Markus Mayer wrote:
> > This series contains a proposal for supporting 32-bit libraries and
> > binaries on a 64-bit platform.
> > 
> > There are some limitations and prerequisites as to the scope this has
> > been tested.
> > 
> > - It has only been tested on ARM/ARM64.
> > - It has only been tested with an external toolchain
> >   (https://github.com/Broadcom/stbgcc-6.3/releases).
> > - It requires that the Aarch64 compiler be compiled with "multi-lib"
> >   enabled. This ensures that the sysroot doesn't contain a lib64 ->
> >   lib symlink. Instead, lib and lib64 are separate directories.
> > - It has only been tested with a fairly limited number of packages.
> >   There might be other packages that don't correctly pass --libdir,
> >   similar to bzip2 as mentioned below.  
> 
> There is one thing I am missing here. AFAICS, this series does not add support 
> for building 32-bit libraries in addition to their 64-bit versions. So the 
> 32-bit binaries that are to run on target must have their dependencies 
> satisfied from the toolchain provided libraries. Is that correct?

Yes, indeed, I'm interested in understanding that part as well. It
seems like the assumption is that the external toolchain has a 32-bit
sysroot, and only the libraries from this sysroot are necessary.

This feels very limited/restricted to very specific use cases and
therefore not very usable in a broader context.

And extending that to building packages is really difficult with
Buildroot:

 (1) Kconfig has no way to easily allow to select for each and every
     package whether it should be built 32-bit, 64-bit or both.

 (2) Our package infrastructure is entirely based on the fact that a
     given package is built only once. So if you assume that you
     solved problem (1) by adding gazillions of Config.in options to
     each and every package, there is still no way the zlib package
     will be built two times, once 32-bit and 64-bit. No without major
     changes to the package infrastructure.

So supporting multilib has already been discussed several times, but
nobody ever came up with a proposal that could reasonably solve
problems (1) and (2). Unfortunately, unless there are some proposals to
solve those problems, I don't think we want to have some very limited
and highly specific multilib support, that only works with external
toolchains and that only works for 32-bit libraries provided by the
external toolchain.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

[Buildroot] [RFC 0/6] Support 32-bit binaries on a 64-bit architecture

[Florian Fainelli]

On 02/16/2018 12:59 PM, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 16 Feb 2018 08:45:57 +0200, Baruch Siach wrote:
> 
>> On Thu, Feb 15, 2018 at 04:56:06PM -0800, Markus Mayer wrote:
>>> This series contains a proposal for supporting 32-bit libraries and
>>> binaries on a 64-bit platform.
>>>
>>> There are some limitations and prerequisites as to the scope this has
>>> been tested.
>>>
>>> - It has only been tested on ARM/ARM64.
>>> - It has only been tested with an external toolchain
>>>   (https://github.com/Broadcom/stbgcc-6.3/releases).
>>> - It requires that the Aarch64 compiler be compiled with "multi-lib"
>>>   enabled. This ensures that the sysroot doesn't contain a lib64 ->
>>>   lib symlink. Instead, lib and lib64 are separate directories.
>>> - It has only been tested with a fairly limited number of packages.
>>>   There might be other packages that don't correctly pass --libdir,
>>>   similar to bzip2 as mentioned below.  
>>
>> There is one thing I am missing here. AFAICS, this series does not add support 
>> for building 32-bit libraries in addition to their 64-bit versions. So the 
>> 32-bit binaries that are to run on target must have their dependencies 
>> satisfied from the toolchain provided libraries. Is that correct?
> 
> Yes, indeed, I'm interested in understanding that part as well. It
> seems like the assumption is that the external toolchain has a 32-bit
> sysroot, and only the libraries from this sysroot are necessary.

Correct, that is what the stbgcc-6.3 toolchain contains.

> 
> This feels very limited/restricted to very specific use cases and
> therefore not very usable in a broader context.

I suppose it could be reasonably easy to extend on that and either
require a second toolchain to be built or just point to a different
location?

> 
> And extending that to building packages is really difficult with
> Buildroot:
> 
>  (1) Kconfig has no way to easily allow to select for each and every
>      package whether it should be built 32-bit, 64-bit or both.
> 
>  (2) Our package infrastructure is entirely based on the fact that a
>      given package is built only once. So if you assume that you
>      solved problem (1) by adding gazillions of Config.in options to
>      each and every package, there is still no way the zlib package
>      will be built two times, once 32-bit and 64-bit. No without major
>      changes to the package infrastructure.
> 
> So supporting multilib has already been discussed several times, but
> nobody ever came up with a proposal that could reasonably solve
> problems (1) and (2). Unfortunately, unless there are some proposals to
> solve those problems, I don't think we want to have some very limited
> and highly specific multilib support, that only works with external
> toolchains and that only works for 32-bit libraries provided by the
> external toolchain.

Would you recommend that this logic be sort of done using a two pass
build, by that I mean something like an external top-level Makefile
would be controlling the buildroot build process and do something like:

- build all relevant packages for the 32-bit architecture (that could
include the toolchain as well)

- build all relevant packages for the 64-bit architecture, and copy the
32-bit binaries and libraries of interest from the previously build
32-bit environment?

I suppose such a thing could be done, it has the advantage of not
tweaking buildroot too much, although we still need to teach the 64-bit
variant to accept seeing "foreign" binaries.
-- 
Florian