[Buildroot] [PATCH v6 00/13] Relocatable SDK / build machine leaks

[Buildroot] [PATCH v6 00/13] Relocatable SDK / build machine leaks

[Samuel Martin]

(Series formerly named: "Host-package: RPATH fixup")


Hi all,


Here is yet another round of the series aiming improving the relocatability
of the SDK built by Buildroot.

Compare to the previous submissions [1,2], the biggest change in this version
is the rewritting in shell of the fix-rpath script.

In details:

Patches 1 to 3:
  Clean up and preparatory work for the relocatable SDK series and also helping
  to track build machine leaks into host, target and staging trees.

Patches 4 to 6:
  Make some pieces of the Buildroot SDK relocatable (limited to some symlinks
  and *-config scripts).

Patches 7 to 9:
  Add the fix-rpath script, then run it on the Buildroot host, target and
  staging trees.

Patch 10:
  Clean-up speex package WRT its RPATH hook.

Patch 11:
  Update the check-host-rpath to re-use the shell helpers.

Patches 12 and 13:
  Add means to track build machine leaks into the host, target and staging
  trees. These patches will allow to identify what remains to be fixed WRT
  build machine leaks.


Per-patch changelog is appended to the commit message of each patch.


[1] http://lists.busybox.net/pipermail/buildroot/2015-July/132440.html
[2] http://lists.busybox.net/pipermail/buildroot/2015-July/132659.html


Regards,

Samuel

Samuel Martin (13):
  package/linux-headers: cleanup installation
  core: use $(CURDIR) to set TOPDIR
  core: re-enter make if $(CURDIR) or $(O) are not absolute canonical
    path
  core: staging symlink uses a relative path when possible
  core: make staging *-config scripts relocatable
  core: make host *-config scripts relocatable
  support/scripts: add fix-rpath script + a bunch of helpers
  core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS
  core: add {TARGET,STAGING}_SANITIZE_RPATH_HOOK to
    TARGET_FINALIZE_HOOKS
  package/speex: remove no longer needed hook
  support/scripts: update check-host-rpath to use the shell helpers
  support/scripts: add check-host-leaks script + all needed helpers
  core: add check-leaks-in-{target,host,staging} targets

 Makefile                               |  88 +++++++++++++++++-
 package/linux-headers/linux-headers.mk |   1 +
 package/pkg-generic.mk                 |  14 ++-
 package/pkg-utils.mk                   |   5 +
 package/speex/speex.mk                 |   5 -
 support/scripts/check-host-leaks       |  63 +++++++++++++
 support/scripts/check-host-rpath       |  67 +++++---------
 support/scripts/fix-rpath              | 101 ++++++++++++++++++++
 support/scripts/shell/log.sh           |  57 ++++++++++++
 support/scripts/shell/patchelf.sh      | 163 +++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh       | 151 ++++++++++++++++++++++++++++++
 support/scripts/shell/sdk.sh           | 141 ++++++++++++++++++++++++++++
 support/scripts/shell/source.sh        |  73 +++++++++++++++
 support/scripts/shell/utils.sh         | 142 ++++++++++++++++++++++++++++
 14 files changed, 1017 insertions(+), 54 deletions(-)
 create mode 100755 support/scripts/check-host-leaks
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

--
2.7.0

[Buildroot] [PATCH v6 01/13] package/linux-headers: cleanup installation

[Samuel Martin]

This patch removes unnecessary files generated during installation.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5 -> v6:
- new patch
---
 package/linux-headers/linux-headers.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
index 1b8b81b..872ef8f 100644
--- a/package/linux-headers/linux-headers.mk
+++ b/package/linux-headers/linux-headers.mk
@@ -53,6 +53,7 @@ define LINUX_HEADERS_INSTALL_STAGING_CMDS
 			HOSTCXX="$(HOSTCXX)" \
 			INSTALL_HDR_PATH=$(STAGING_DIR)/usr \
 			headers_install)
+	find $(STAGING_DIR)/usr \( -name .install -o -name ..install.cmd \) -delete
 endef
 
 ifeq ($(BR2_KERNEL_HEADERS_VERSION),y)
-- 
2.7.0

[Buildroot] [PATCH v6 02/13] core: use $(CURDIR) to set TOPDIR

[Samuel Martin]

This changes saves a shell call and uses a variable automatically set
by make [1].

[1] http://www.gnu.org/software/make/manual/make.html#Quick-Reference

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5 -> v6:
- new patch
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 17c181b..13c16af 100644
--- a/Makefile
+++ b/Makefile
@@ -69,7 +69,7 @@ endif
 .NOTPARALLEL:
 
 # absolute path
-TOPDIR := $(shell pwd)
+TOPDIR := $(CURDIR)
 CONFIG_CONFIG_IN = Config.in
 CONFIG = support/kconfig
 DATE := $(shell date +%Y%m%d)
-- 
2.7.0

[Buildroot] [PATCH v6 03/13] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path

[Samuel Martin]

When $(CURDIR) or $(O) contain symlinks (or mount-bind) in their path,
they can be resolved differently, depending on each package build-system
(whether it uses the given paths or get the absolute canonical ones).

Thus, to make easier tracking down host machine paths leaking into the
host, target or staging trees, the CURDIR and O variables are set to
their absolute canonical paths.

Note that this change takes care of the makefile wrapper installed in
$(O) to avoid unneeded make recursion.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5 -> v6:
- new patch
---
 Makefile | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 13c16af..10193ba 100644
--- a/Makefile
+++ b/Makefile
@@ -26,16 +26,30 @@
 
 # Trick for always running with a fixed umask
 UMASK = 0022
-ifneq ($(shell umask),$(UMASK))
+
+# Check if we need to re-enter make for one or several of the following reasons:
+# 1- Wrong (too restrictive) umask:
+#    This prevents Buildroot and packages from creating files and directories.
+# 2- CWD (i.e. $(CURDIR)) not being the absolute canonical path:
+#    This makes harder tracking and fixing host machine path leaks.
+# 3- Output location (i.e. $(O)) not being the absolute canonical path:
+#    This makes harder tracking and fixing host machine path leaks.
+#
+# Note:
+# - remove the trailing '/.' from $(O) as it can be added by the makefile
+#   wrapper installed in the $(O).
+ifneq ($(shell umask):$(CURDIR):$(patsubst %/.,%,$(O)),$(UMASK):$(realpath $(CURDIR)):$(realpath $(O)))
 .PHONY: _all $(MAKECMDGOALS)
 
 $(MAKECMDGOALS): _all
 	@:
 
 _all:
-	@umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
+	umask $(UMASK) && \
+		$(MAKE) -C $(realpath $(CURDIR)) --no-print-directory \
+			$(MAKECMDGOALS) O=$(realpath $(O))
 
-else # umask
+else # umask / $(CURDIR) / $(O)
 
 # This is our default rule, so must come first
 all:
@@ -1001,4 +1015,4 @@ include docs/manual/manual.mk
 
 .PHONY: $(noconfig_targets)
 
-endif #umask
+endif #umask / $(CURDIR) / $(O)
-- 
2.7.0

[Buildroot] [PATCH v6 04/13] core: staging symlink uses a relative path when possible

[Samuel Martin]

A step forward toward a relocatable SDK.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- no changes

changes v4->v5:
- add support when BR2_HOST_DIR is set somewhere out of BASE_DIR

changes v3->v4:
- no change

changes v2->v3:
- no change
---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Makefile b/Makefile
index 10193ba..7c87177 100644
--- a/Makefile
+++ b/Makefile
@@ -518,7 +518,11 @@ $(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIS
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):
 	@mkdir -p $(STAGING_DIR)
+ifeq ($(BASE_DIR)/host/$(STAGING_SUBDIR),$(STAGING_DIR))
+	@ln -snf host/$(STAGING_SUBDIR) $(BASE_DIR)/staging
+else
 	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
+endif
 
 RSYNC_VCS_EXCLUSIONS = \
 	--exclude .svn --exclude .git --exclude .hg --exclude .bzr \
-- 
2.7.0

[Buildroot] [PATCH v6 05/13] core: make staging *-config scripts relocatable

[Samuel Martin]

This change adjusts the _CONFIG_SCRIPTS hook to set add {exec_,}prefix computed
relatively to the script location.

This patch hook only fixes *-config scripts located in the staging area,
the target ones are already removed. A follow-up change will fix those
from the HOST_DIR location.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5 -> v6:
- new patch
---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1e024d3..1bf613e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -240,7 +240,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 		$(call MESSAGE,"Fixing package configuration files") ;\
 			$(SED)  "s,$(BASE_DIR), at BASE_DIR@,g" \
 				-e "s,$(STAGING_DIR), at STAGING_DIR@,g" \
-				-e "s,^\(exec_\)\?prefix=.*,\1prefix=@STAGING_DIR@/usr,g" \
+				-e "s,^\(exec_\)\?prefix=.*,\1prefix=\`dirname \$$0\`/../../usr,g" \
 				-e "s,-I/usr/,-I at STAGING_DIR@/usr/,g" \
 				-e "s,-L/usr/,-L at STAGING_DIR@/usr/,g" \
 				-e "s, at STAGING_DIR@,$(STAGING_DIR),g" \
-- 
2.7.0

[Buildroot] [PATCH v6 06/13] core: make host *-config scripts relocatable

[Samuel Martin]

This change adjusts the _CONFIG_SCRIPTS hook to set add {exec_,}prefix computed
relatively to the script location.

This patch only fixes *-config scripts located in the host area, those
from the target and staging area are already handled.

This hook is a kind of brute-force sed on all files matching '*-config'
in the $(HOST_DIR)/usr/bin directory. This allows correctly handle
script whose name may depends on the host system, not the target
configuration (e.g. ncurses can be configured for the target to be
ncurses6w, so ncurses6w-config ; whereas for the host it will be
ncurses5, so ncurses5-config).
Since there way less host packages than target ones, it is reasonable to
assume that processing all *-config scripts found when a host package
(whose target package set the *_CONFIG_SCRIPTS variable) is fairly short
and does not add its overhead not even noticeable.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5 -> v6:
- new patch
---
 package/pkg-generic.mk | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1bf613e..ca79450 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -207,6 +207,18 @@ $(BUILD_DIR)/%/.stamp_host_installed:
 	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
 	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
+	$(Q)if test -n "$($(RAWNAME)_CONFIG_SCRIPTS)" ; then \
+		$(call MESSAGE,"Fixing package configuration files") ;\
+			ls $(HOST_DIR)/usr/bin/*-config 2>/dev/null |\
+			xargs --no-run-if-empty \
+			$(SED)  "s,$(BASE_DIR), at BASE_DIR@,g" \
+				-e "s,$(HOST_DIR), at HOST_DIR@,g" \
+				-e "s,^\(exec_\)\?prefix=.*,\1prefix=\`dirname \$$0\`/../../usr,g" \
+				-e "s,-I/usr/,-I at HOST_DIR@/usr/,g" \
+				-e "s,-L/usr/,-L at HOST_DIR@/usr/,g" \
+				-e "s, at HOST_DIR@,$(HOST_DIR),g" \
+				-e "s, at BASE_DIR@,$(BASE_DIR),g" ;\
+	fi
 	@$(call step_end,install-host)
 	$(Q)touch $@
 
-- 
2.7.0

[Buildroot] [PATCH v6 07/13] support/scripts: add fix-rpath script + a bunch of helpers

[Samuel Martin]

This commit introduces a fix-rpath shell-script able to scan a tree,
detect ELF files, check their RPATH and fix it in a proper way.

Along to the fix-rpath script, it also adds a bunch of shell helper
functions grouped into modules. This will help writing scripts handling
RPATH and other things, while allowing to share and reuse these
functions between scripts.

These helpers are namespaced within the filename of the module in which
they are gathered.

This change adds 6 modules:
- source.sh: provides simple helper to easily source another module, taking
    care of not sourcing again an already-sourced one;
- log.sh: provides logging helpers;
- utils.sh: provides simple functions to filter ELF files in a list;
- readelf.sh: provides functions retrieving ELF details from a file;
- patchelf.sh: provides function updating ELF files;
- sdk.sh: provides RPATH computation functions.

These 6 modules are used by the fix-rpath script.
Follow-up patches will make some scripts leveraging these modules.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- fully rewritten in shell

changes v4->v5:
- add verbose support
- rename shrink_rpath -> clear_rpath
- add sanitize_rpath function

changes v3->v4:
- fix typos and license (Baruch)

changes v2->v3:
- no change
---
 support/scripts/fix-rpath         | 101 +++++++++++++++++++++++
 support/scripts/shell/log.sh      |  57 +++++++++++++
 support/scripts/shell/patchelf.sh | 163 ++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh  |  52 ++++++++++++
 support/scripts/shell/sdk.sh      |  70 ++++++++++++++++
 support/scripts/shell/source.sh   |  73 +++++++++++++++++
 support/scripts/shell/utils.sh    |  60 ++++++++++++++
 7 files changed, 576 insertions(+)
 create mode 100755 support/scripts/fix-rpath
 create mode 100644 support/scripts/shell/log.sh
 create mode 100644 support/scripts/shell/patchelf.sh
 create mode 100644 support/scripts/shell/readelf.sh
 create mode 100644 support/scripts/shell/sdk.sh
 create mode 100644 support/scripts/shell/source.sh
 create mode 100644 support/scripts/shell/utils.sh

diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
new file mode 100755
index 0000000..938e599
--- /dev/null
+++ b/support/scripts/fix-rpath
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  cat <<EOF >&2
+Usage:  ${0} TREE_KIND TREE_ROOT
+
+Description:
+
+        This script scans a tree and sanitize ELF files' RPATH found in there.
+
+        Sanitization behaves the same whatever the kindd of the processed tree, but
+        the resulting RPATH differs.
+
+        Sanitization action:
+        - remove RPATH pointing outside of the tree
+        - for RPATH pointing in the tree:
+          - if they point to standard location (/lib, /usr/lib): remove them
+          - otherwise: make them relative using \$ORIGIN
+
+        For the target tree:
+        - scan the whole tree for sanitization
+
+        For the staging tree :
+        - scan the whole tree for sanitization
+
+        For the host tree:
+        - skip the staging tree for sanitization
+        - add \$HOST_DIR/{lib,usr/lib} to RPATH (as relative pathes)
+
+Arguments:
+
+        TREE_KIND   Kind of tree to be processed.
+                    Allowed values: host, target, staging
+
+        TREE_ROOT   Path to the root of the tree to be scaned
+
+EOF
+}
+
+source "${0%/*}/shell/source.sh"
+
+source.load_module utils
+source.load_module readelf
+source.load_module patchelf
+
+main() {
+    local tree="${1}"
+    local basedir="$( readlink -f "${2}" )"
+
+    local find_args=( "${basedir}" )
+    local sanitize_extra_args=()
+    local readelf
+
+    case "${tree}" in
+        host)
+            # do not process the sysroot (only contains target binaries)
+            find_args+=( "-name" "sysroot" "-prune" "-o" )
+
+            # do not process the external toolchain installation directory to
+            # to avoid breaking it.
+            find_args+=( "-path" "*/opt/ext-toolchain" "-prune" "-o" )
+
+            # make sure RPATH will point to ${hostdir}/lib and ${hostdir}/usr/lib
+            sanitize_extra_args+=( "keep_lib_and_usr_lib" )
+
+            readelf="${HOST_READELF}"
+            ;;
+        staging|target)
+            readelf="${TARGET_READELF}"
+            ;;
+        *)
+            usage
+            exit 1
+            ;;
+    esac
+
+    find_args+=( "-type" "f" "-print" )
+
+    while read file; do
+        READELF="${READELF}" PATCHELF="${PATCHELF}" \
+            patchelf.sanitize_rpath "${basedir}" "${file}" ${sanitize_extra_args[@]}
+    done < <( find ${find_args[@]} | utils.filter_elf )
+}
+
+main ${@}
diff --git a/support/scripts/shell/log.sh b/support/scripts/shell/log.sh
new file mode 100644
index 0000000..ffa19e8
--- /dev/null
+++ b/support/scripts/shell/log.sh
@@ -0,0 +1,57 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Logging helpers
+#
+# This module defines the following functions:
+#   log.trace
+#   log.debug
+#   log.info
+#   log.warn
+#   log.errorN
+#   log.error
+#
+# This module sets the following variables:
+#   my_name
+#
+# This module is sensitive to the following environment variables:
+#   DEBUG
+
+source.declare_module log
+
+# Debug level:
+# - 0 or empty: only show errors
+# - 1         : show errors and warnings
+# - 2         : show errors, warnings, and info
+# - 3         : show errors, warnings, info and debug
+: ${DEBUG:=0}
+
+# Low level utility function
+log.trace()  { local msg="${1}"; shift; printf "%s: ${msg}" "${my_name}" "${@}"; }
+
+# Public logging functions
+log.debug()  { :; }
+[ ${DEBUG} -lt 3 ] || log.debug() { log.trace "${@}" >&2; }
+log.info()   { :; }
+[ ${DEBUG} -lt 2 ] || log.info()  { log.trace "${@}" >&2; }
+log.warn()   { :; }
+[ ${DEBUG} -lt 1 ] || log.warn()  { log.trace "${@}" >&2; }
+log.errorN() { local ret="${1}"; shift; log.warn "${@}"; exit ${ret}; }
+log.error()  { log.errorN 1 "${@}"; }
+
+# Program name
+my_name="${0##*/}"
+
diff --git a/support/scripts/shell/patchelf.sh b/support/scripts/shell/patchelf.sh
new file mode 100644
index 0000000..d1eb590
--- /dev/null
+++ b/support/scripts/shell/patchelf.sh
@@ -0,0 +1,163 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Patchelf helpers
+#
+# This module defines the following functions:
+#   patchelf.set_xrpath
+#   patchelf.update_rpath
+#   patchelf.sanitize_rpath
+#
+# This module is sensitive to the following environment variables:
+#   PATCHELF
+#   READELF
+
+source.declare_module patchelf
+
+source.load_module log
+source.load_module sdk
+source.load_module utils
+source.load_module readelf
+
+: ${PATCHELF:=patchelf}
+
+# patchelf.set_xrpath file rpath...
+#
+# Set RPATH in $file.
+# Automatically join all RPATH with the correct separator, and also handle
+# XRPATH (replacing "XORIGIN" with "$ORIGIN").
+#
+# file  : ELF file path
+# rpath : RPATH element
+#
+# environment:
+#   PATCHELF: patchelf program path
+patchelf.set_xrpath() {
+    local file="${1}"
+    shift
+    local xrpath="$(sed -e 's/ +/:/g' <<<"${@}")"
+    "${PATCHELF}" --set-rpath "${xrpath//XORIGIN/\$ORIGIN}" "${file}"
+}
+
+# patchelf.update_rpath basedir binary libdirs...
+#
+# Set RPATH in $binary computing them from the paths $libdirs (and $basedir).
+# Existing RPATH in $file will be overwritten if any.
+#
+# basedir : absolute path of the tree in which the $bindir and $libdirs must be
+# binary  : ELF file absolute path
+# libdirs : list of library location (absolute paths)
+#
+# environment:
+#   PATCHELF: patchelf program path
+patchelf.update_rpath() {
+    local basedir="${1}"
+    local binary="${2}"
+    shift 2
+    local libdirs=( ${@} )
+    log.debug "  basedir: %s\n" "${basedir}"
+    log.debug "      elf: %s\n" "${binary}"
+    log.debug "  libdirs: %s\n" "${libdirs[*]}"
+    log.info  "    rpath: %s\n" \
+        "$( sdk.compute_xrpath "${basedir}" "${binary%/*}" ${libdirs[@]} |
+            sed -e 's/XORIGIN/\$ORIGIN/g' )"
+    PATCHELF="${PATCHELF}" patchelf.set_xrpath "${binary}" \
+        $( sdk.compute_xrpath "${basedir}" "${binary%/*}" ${libdirs[@]} )
+}
+
+# patchelf.sanitize_rpath basedir binary [keep_lib_usr_lib]
+#
+# Scan $binary's RPATH, remove any of them pointing outside of $basedir.
+# If $keep_lib_usr_lib in not empty, the library directories $basedir/lib and
+# $basedir/usr/lib will be added to the RPATH.
+#
+# Note:
+#     Absolute paths is needed to correctly handle symlinks and or mount-bind in
+#     the $basedir path.
+#
+# basedir          : absolute path of the tree in which the $bindir and $libdirs
+#                    must be
+# binary           : ELF file absolute path
+# keep_lib_usr_lib : add to RPATH $basedir/lib and $basedir/usr/lib
+#
+# environment:
+#   PATCHELF: patchelf program path
+#   READELF : readelf program path
+patchelf.sanitize_rpath() {
+    local basedir="$( readlink -f "${1}" )"
+    local binary="${2}"
+    local keep_lib_usr_lib="${3}"
+
+    local path abspath rpath
+    local libdirs=()
+
+    if test -n "${keep_lib_usr_lib}" ; then
+        libdirs+=( "${basedir}/lib" "${basedir}/usr/lib" )
+    fi
+
+    log.info "ELF: %s\n" "${binary}"
+
+    local rpaths="$( READELF="${READELF}" readelf.get_rpath "${binary}" )"
+
+    for rpath in ${rpaths//:/ } ; do
+        # figure out if we should keep or discard the path; there are several
+        # cases to handled:
+        # - $path starts with "$ORIGIN":
+        #     The original build-system already took care of setting a relative
+        #     RPATH, resolve it and test if it is worthwhile to keep it;
+        # - $basedir/$path exists:
+        #     The original build-system already took care of setting an absolute
+        #     RPATH (absolute in the final rootfs), resolve it and test if it is
+        #     worthwhile to keep it;
+        # - $path start with $basedir:
+        #     The original build-system added some absolute RPATH (absolute on
+        #     the build machine). While this is wrong, it can still be fixed; so
+        #     test if it is worthwhile to keep it;
+        # - $path points somewhere else:
+        #     (can be anywhere: build trees, staging tree, host location,
+        #     non-existing location, etc.)
+        #     Just discard such a path.
+        if grep -q '^$ORIGIN/' <<<"${rpath}" ; then
+            path="${binary%/*}/${rpath#*ORIGIN/}"
+        elif test -e "${basedir}/${rpath}" ; then
+            path="${basedir}/${rpath}"
+        elif grep -q "^${basedir}/" <<<"$( readlink -f "${rpath}" )" ; then
+            path="${rpath}"
+        else
+            log.debug "\tDROPPED [out-of-tree]: %s\n" "${rpath}"
+            continue
+        fi
+
+        abspath="$( readlink -f "${path}" )"
+
+        # discard path pointing to default locations handled by ld-linux
+        if grep -qE "^${basedir}/(lib|usr/lib)$" <<<"${abspath}" ; then
+            log.debug \
+                "\tDROPPED [std libdirs]: %s (%s)\n" "${rpath}" "${abspath}"
+            continue
+        fi
+
+        log.debug "\tKEPT %s (%s)\n" "${rpath}" "${abspath}"
+
+        libdirs+=( "${abspath}" )
+
+    done
+
+    libdirs=( $( utils.list_reduce ${libdirs[@]} ) )
+
+    PATCHELF="${PATCHELF}" \
+        patchelf.update_rpath "${basedir}" "${binary}" ${libdirs[@]}
+}
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
new file mode 100644
index 0000000..82968a2
--- /dev/null
+++ b/support/scripts/shell/readelf.sh
@@ -0,0 +1,52 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Readelf helpers
+#
+# This module defines the following functions:
+#   readelf.get_rpath
+#
+# This module is sensitive to the following environment variables:
+#   READELF
+#
+# This module sets the following environment variables:
+#   LC_ALL=C
+
+source.declare_module readelf
+
+# Override the user's locale so we are sure we can parse the output of
+# readelf(1) and file(1)
+export LC_ALL=C
+
+: ${READELF:=readelf}
+
+# readelf.get_rpath file
+#
+# Return the unsplitted RPATH/RUNPATH of $file.
+#
+# To split the returned RPATH string and store them in an array, do:
+#
+#     paths=( $(readelf.get_rpath "${file}" | sed -e 's/:/ /g') )
+#
+# file : ELF file path
+#
+# environment:
+#   READELF: readelf program path
+readelf.get_rpath() {
+    local file="${1}"
+    "${READELF}" --dynamic "${file}" |
+        sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d ; s//\3/'
+}
diff --git a/support/scripts/shell/sdk.sh b/support/scripts/shell/sdk.sh
new file mode 100644
index 0000000..3188516
--- /dev/null
+++ b/support/scripts/shell/sdk.sh
@@ -0,0 +1,70 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# SDK helpers
+#
+# This module defines the following functions:
+#   sdk.compute_relative_path
+#   sdk.compute_xrpath
+
+source.declare_module sdk
+
+# sdk.compute_relative_path basedir path start
+#
+# Compute and return the relative path between $start and $path within $basedir.
+#
+# basedir : absolute path of the tree in which the $path and $start must be
+# path    : destination absolute path
+# start   : origin absolute path
+sdk.compute_relative_path() {
+    local basedir="${1}"
+    local path="${2}"
+    local start="${3}"
+    # sanity checks: make sure $path and $start starts with $basedir
+    grep -q "^${basedir}" <<<"${path}" || return 1
+    grep -q "^${basedir}" <<<"${start}" || return 1
+    local i backward="${start#${basedir}}"
+    backward=( ${backward//\// } )
+    local relative=()
+    for i in ${backward[@]} ; do
+        test -n ${i} || continue
+        relative+=( ".." )
+    done
+    relative+=( ${path#${basedir}} )
+    sed -r -e 's:[ /]+:/:g' <<<"${relative[@]}"
+}
+
+# sdk.compute_xrpath basedir bindir libdirs...
+#
+# Compute and return the list of XRPATH.
+# XRPATH is RPATH with "$ORIGIN" replaced by "XORIGIN"; this trick is needed to
+# prevent mis-expansion by the shell interpreter.
+# The reverse reokacement is done right in the patchelf command.
+#
+# basedir : absolute path of the tree in which the $bindir and $libdirs must be
+# bindir  : binary directory absolute path
+# libdirs : list of library directories (absolute paths)
+sdk.compute_xrpath() {
+    local basedir="${1}"
+    local bindir="${2}"
+    shift 2
+    local libdirs=( ${@} )
+    local rpath=()
+    for libdir in ${libdirs[@]} ; do
+        rpath+=( "XORIGIN/$(sdk.compute_relative_path "${basedir}" "${libdir}" "${bindir}")" )
+    done
+    sed -e 's/ /:/g' <<<"${rpath[@]}"
+}
diff --git a/support/scripts/shell/source.sh b/support/scripts/shell/source.sh
new file mode 100644
index 0000000..1c70344
--- /dev/null
+++ b/support/scripts/shell/source.sh
@@ -0,0 +1,73 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Source helpers
+#
+# This module defines the following functions:
+#   source.declare_module
+#   source.load_module
+#
+# This module is sensitive to the following environment variables:
+#   TOPDIR
+
+# Assuming the script sourcing this file is in support/scripts/
+: ${TOPDIR:=$( readlink -f "${0}" | sed -re 's:(/[^/]+){3}$::' )}
+
+if [ ! -f "${TOPDIR}/support/scripts/shell/source.sh" ] ; then
+    cat <<EOF >&2
+error: TOPDIR does not point to Buildroot's '\$(TOPDIR)'.
+
+       This script '${0##*/}' is most likely not installed in Buildroot's
+       '\$(TOPDIR)/support/scripts' directory.
+
+       You can fix this by:
+       - either installing '${0##*/}' in the support/scripts/ directory;
+       - or setting the TOPDIR variable in the '${0##*/}' script, before
+         sourcing anything.
+EOF
+    exit 1
+fi
+
+# source.declare_module module_name
+#
+# Declare a shell module.
+# Set the variable '_source_${module_name}'.
+# Should be called once per module, in the global scope.
+#
+# module_name : Module name (allowed char.: [_a-zA-Z0-9])
+source.declare_module() {
+    local module_name="${1}"
+    # use printf from bash to set the variable in the environment:
+    printf -v "_source_${module_name}" "%s" "${module_name}"
+}
+
+# source.load_module module_name
+#
+# Load the given shell module, making available all functions declared
+# in it, ensuring it is not reloaded if it already is.
+# Should be called in the global scope.
+# Need the TOPDIR environment variable.
+#
+# param module_name: Module name
+source.load_module() {
+    local module_name="${1}"
+    local loaded="loaded=\${_source_${module_name}}"
+    eval "${loaded}"
+    local module_file="${TOPDIR}/support/scripts/shell/${module_name}.sh"
+    test -n "${loaded}" || source "${module_file}"
+}
+
+source.declare_module source
diff --git a/support/scripts/shell/utils.sh b/support/scripts/shell/utils.sh
new file mode 100644
index 0000000..6b071fc
--- /dev/null
+++ b/support/scripts/shell/utils.sh
@@ -0,0 +1,60 @@
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+# Miscellaneous helpers
+#
+# This module defines the following functions:
+#   utils.list_reduce
+#   utils.filter_elf
+
+source.declare_module utils
+
+# utils.list_reduce input_list
+#
+# Return the $input_list list with duplicated items removed.
+# Order is preserved, wrt the first occurence of duplicated items.
+#
+# input_list : list of items
+utils.list_reduce() {
+    local -a lout # out list
+    local -A dout # associative array, used to check if an item is duplicated
+    local i
+
+    for i in ${@} ; do
+        test -z "${dout["${i}"]}" || continue
+        dout["${i}"]=y
+        lout+=( "${i}" )
+    done
+
+    echo ${lout[@]}
+}
+
+# utils.filter_elf filepath...
+#
+# Filter ELF files; if $file is an ELF file, $file is print, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+utils.filter_elf() {
+    local in file
+    test ${#} -gt 0 && in='printf "%s\n" ${@}' || in='dd 2>/dev/null'
+    eval "${in}" |
+        while read file ; do
+            file "${file}" 2>/dev/null |
+                sed -r -e '/^([^:]+):.*\<ELF\>.*/!d ; s//\1/'
+        done
+}
-- 
2.7.0

[Buildroot] [PATCH v6 08/13] core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Samuel Martin]

This patch adds host-patchelf as a target-finalize dependency, and
introduces the HOST_SANITIZE_RPATH_HOOK hook to fix the ELF files'
RPATH from the HOST_DIR location (excluding the sysroot).

After running this hook, the RPATH from any host ELF files is relative to
the binary location itself.

Notes:
- we avoid to fix RPATH in the sysroot.
- we do not try to fix RPATH in the external toolchain installation
  location as they may have been built in a way, this is already correct;
  furthermore, fixing RPATH in those programs may result in breaking them.
- the whole host directory is processed because a number of
  host-package install programs that could be useful in places
  different from $(HOST_DIR)/{bin,sbin,usr/bin,usr/sbin}.
- the shared libraries are also processed in case they have a 'main'
  function.

As a step toward a fully relocatable SDK, this change allows to get the
toolchain relocatable, but not yet the whole SDK.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- add verbose support

changes v3->v4:
- add host-patchelf to PACKAGES instead of target-finalize (Baruch)
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)
---
 Makefile | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/Makefile b/Makefile
index 7c87177..b1f4fcf 100644
--- a/Makefile
+++ b/Makefile
@@ -616,6 +616,31 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# RPATH fixing
+# - The host hook sets RPATH in host ELF binaries, using relative paths to the
+#   library locations.
+# - The target hook sanitizes RPATH in target ELF binaries, removing paths
+#   pointing to package's build directories or the sysroot's libdirs.
+PACKAGES += host-patchelf
+
+ifeq ($(BR2_DEBUG_RPATH),)
+BR2_DEBUG_RPATH = 0
+endif
+ifeq ($(BR2_DEBUG_RPATH),0)
+DEBUG_RPATH = 0
+else
+DEBUG_RPATH = $(shell echo $$(($(BR2_DEBUG_RPATH)+1)))
+endif
+
+define HOST_SANITIZE_RPATH_HOOK
+	DEBUG=$(DEBUG_RPATH) \
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=readelf \
+		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
+
 $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(PACKAGES)
-- 
2.7.0

[Buildroot] [PATCH v6 09/13] core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Samuel Martin]

This patch introduces the TARGET_SANITIZE_RPATH_HOOK and
STAGING_SANITIZE_RPATH_HOOK hooks fixing the ELF files' RPATH of
binaries from, respectively, the TARGET_DIR and the STAGING_DIR
locations.

After running this hook, the RPATH from any target ELF files from both
the target and the staging locations won't contain any occurence of the
sysroot or some build locations.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- update for the new script version
- add debug mode support

changes v4->v5:
- target hook now sanitizes the rpath (Baruch)
- add verbose support
- update comment

changes v3->v4:
- rebase
- add comment

changes v2->v3:
- move hook in Makefile (Baruch)
---
 Makefile | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/Makefile b/Makefile
index b1f4fcf..eb8a037 100644
--- a/Makefile
+++ b/Makefile
@@ -641,6 +641,26 @@ endef
 
 TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
 
+# Function sanitizing target/staging ELF files' RPATH.
+# i.e. it removes paths pointing to the staging or build location from the ELF
+# files' RPATH.
+define TARGET_SANITIZE_RPATH_HOOK
+	DEBUG=$(DEBUG_RPATH) \
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+		$(TOPDIR)/support/scripts/fix-rpath target $(TARGET_DIR)
+endef
+
+define STAGING_SANITIZE_RPATH_HOOK
+	DEBUG=$(DEBUG_RPATH) \
+	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
+	READELF=$(TARGET_READELF) \
+		$(TOPDIR)/support/scripts/fix-rpath staging $(STAGING_DIR)
+endef
+
+TARGET_FINALIZE_HOOKS += TARGET_SANITIZE_RPATH_HOOK \
+       STAGING_SANITIZE_RPATH_HOOK
+
 $(TARGETS_ROOTFS): target-finalize
 
 target-finalize: $(PACKAGES)
-- 
2.7.0

[Buildroot] [PATCH v6 10/13] package/speex: remove no longer needed hook

[Samuel Martin]

Remove the LIBTOOL_FIXUP hook since RPATH are now sanitized in
target-finalize hooks.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- no change

changes v4->v5:
- new patch (suggested by Baruch)
---
 package/speex/speex.mk | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/package/speex/speex.mk b/package/speex/speex.mk
index a34bfac..24782ec 100644
--- a/package/speex/speex.mk
+++ b/package/speex/speex.mk
@@ -24,11 +24,6 @@ ifeq ($(BR2_PACKAGE_SPEEX_ARM5E),y)
 SPEEX_CONF_OPTS += --enable-arm5e-asm
 endif
 
-define SPEEX_LIBTOOL_FIXUP
-	$(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
-	$(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
-endef
-
 SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
 
 define SPEEX_BUILD_CMDS
-- 
2.7.0

[Buildroot] [PATCH v6 11/13] support/scripts: update check-host-rpath to use the shell helpers

[Samuel Martin]

Cc: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- new patch
---
 support/scripts/check-host-rpath | 67 ++++++++++++++-------------------------
 support/scripts/shell/readelf.sh | 68 ++++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/utils.sh   | 18 +++++++++++
 3 files changed, 110 insertions(+), 43 deletions(-)

diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath
index 48d69da..55be97e 100755
--- a/support/scripts/check-host-rpath
+++ b/support/scripts/check-host-rpath
@@ -1,12 +1,30 @@
 #!/usr/bin/env bash
 
+# Copyright (C) 2015 Yann E. MORIN <yann.morin.1998@free.fr>
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
 # This script scans $(HOST_DIR)/{bin,sbin} for all ELF files, and checks
 # they have an RPATH to $(HOST_DIR)/usr/lib if they need libraries from
 # there.
 
-# Override the user's locale so we are sure we can parse the output of
-# readelf(1) and file(1)
-export LC_ALL=C
+source "${0%/*}/shell/source.sh"
+
+source.load_module utils
+source.load_module readelf
 
 main() {
     local pkg="${1}"
@@ -18,54 +36,17 @@ main() {
 
     ret=0
     while read file; do
-        elf_needs_rpath "${file}" "${hostdir}" || continue
-        check_elf_has_rpath "${file}" "${hostdir}" && continue
+        READELF=readelf readelf.needs_rpath "${file}" "${hostdir}" || continue
+        READELF=readelf readelf.has_rpath "${file}" "${hostdir}" && continue
         if [ ${ret} -eq 0 ]; then
             ret=1
             printf "***\n"
             printf "*** ERROR: package %s installs executables without proper RPATH:\n" "${pkg}"
         fi
         printf "***   %s\n" "${file}"
-    done < <( find "${hostdir}"/usr/{bin,sbin} -type f -exec file {} + 2>/dev/null \
-              |sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d'                \
-                      -e 's//\1/'                                                  \
-            )
+    done < <( find "${hostdir}"/usr/{bin,sbin} -type f -print | utils.filter_elf_executable )
 
     return ${ret}
 }
 
-elf_needs_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local lib
-
-    while read lib; do
-        [ -e "${hostdir}/usr/lib/${lib}" ] && return 0
-    done < <( readelf -d "${file}"                                         \
-              |sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d;' \
-                     -e 's//\1/;'                                          \
-            )
-
-    return 1
-}
-
-check_elf_has_rpath() {
-    local file="${1}"
-    local hostdir="${2}"
-    local rpath dir
-
-    while read rpath; do
-        for dir in ${rpath//:/ }; do
-            # Remove duplicate and trailing '/' for proper match
-            dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )"
-            [ "${dir}" = "${hostdir}/usr/lib" ] && return 0
-        done
-    done < <( readelf -d "${file}"                                              \
-              |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
-                      -e 's//\3/;'                                              \
-            )
-
-    return 1
-}
-
 main "${@}"
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index 82968a2..e0549d2 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -17,7 +17,10 @@
 # Readelf helpers
 #
 # This module defines the following functions:
+#   readelf.get_neededs
 #   readelf.get_rpath
+#   readelf.needs_rpath
+#   readelf.has_rpath
 #
 # This module is sensitive to the following environment variables:
 #   READELF
@@ -33,6 +36,20 @@ export LC_ALL=C
 
 : ${READELF:=readelf}
 
+# readelf.get_neededs file
+#
+# Return the list of the NEEDED libraries of $file.
+#
+# file : ELF file path
+#
+# environment:
+#   READELF: readelf program path
+readelf.get_neededs() {
+    local file="${1}"
+    "${READELF}" --dynamic "${file}" |
+        sed -r -e '/^.* \(NEEDED\) .*Shared library: \[(.+)\]$/!d ; s//\1/'
+}
+
 # readelf.get_rpath file
 #
 # Return the unsplitted RPATH/RUNPATH of $file.
@@ -50,3 +67,54 @@ readelf.get_rpath() {
     "${READELF}" --dynamic "${file}" |
         sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d ; s//\3/'
 }
+
+# readelf.needs_rpath file basedir
+#
+# Return 0 if $file needs to have RPATH set, 1 otherwise.
+#
+# file    : path of file to be tested
+# basedir : path of the tree in which $basedir/lib and $basedir/usr/lib are
+#           checked for belonging to RPATH
+#
+# environment:
+#   READELF: readelf program path
+readelf.needs_rpath() {
+    local file="${1}"
+    local basedir="${2}"
+    local lib
+
+    while read lib; do
+        [ -e "${basedir}/lib/${lib}" ] && return 0
+        [ -e "${basedir}/usr/lib/${lib}" ] && return 0
+    done < <( READELF="${READELF}" readelf.get_neededs "${file}" )
+    return 1
+}
+
+# readelf.has_rpath file basedir
+#
+# Return 0 if $file has RPATH already set to $basedir/lib or $basedir/usr/lib,
+# 1 otherwise.
+#
+# file    : path of file to be tested
+# basedir : path of the tree in which $basedir/lib and $basedir/usr/lib are
+#           checked for belonging to RPATH
+#
+# environment:
+#   READELF: readelf program path
+readelf.has_rpath() {
+    local file="${1}"
+    local basedir="${2}"
+    local rpath dir
+
+    while read rpath; do
+        for dir in ${rpath//:/ }; do
+            # Remove duplicate and trailing '/' for proper match
+            dir="$( sed -r -e "s:/+:/:g; s:/$::" <<<"${dir}" )"
+            [ "${dir}" = "${basedir}/lib" ] && return 0
+            [ "${dir}" = "${basedir}/usr/lib" ] && return 0
+            grep -q '^\$ORIGIN/' <<<"${dir}" && return 0
+        done
+    done < <( READELF="${READELF}" readelf.get_rpath "${file}" )
+
+    return 1
+}
diff --git a/support/scripts/shell/utils.sh b/support/scripts/shell/utils.sh
index 6b071fc..b49a541 100644
--- a/support/scripts/shell/utils.sh
+++ b/support/scripts/shell/utils.sh
@@ -18,6 +18,7 @@
 #
 # This module defines the following functions:
 #   utils.list_reduce
+#   utils.filter_elf_executable
 #   utils.filter_elf
 
 source.declare_module utils
@@ -42,6 +43,23 @@ utils.list_reduce() {
     echo ${lout[@]}
 }
 
+# utils.filter_elf_executable filepath...
+#
+# Filter ELF files; if $file is an ELF file, $file is print, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+utils.filter_elf_executable() {
+    local in file
+    test ${#} -gt 0 && in='printf "%s\n" ${@}' || in='dd 2>/dev/null'
+    eval "${in}" |
+        while read file ; do
+            file "${file}" 2>/dev/null |
+                sed -r -e '/^([^:]+):.*\<ELF\>.*\<executable\>.*/!d ; s//\1/'
+        done
+}
+
 # utils.filter_elf filepath...
 #
 # Filter ELF files; if $file is an ELF file, $file is print, else it is
-- 
2.7.0

[Buildroot] [PATCH v6 12/13] support/scripts: add check-host-leaks script + all needed helpers

[Samuel Martin]

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- new patch
---
 support/scripts/check-host-leaks | 63 +++++++++++++++++++++++++++++++++++
 support/scripts/shell/readelf.sh | 31 ++++++++++++++++++
 support/scripts/shell/sdk.sh     | 71 ++++++++++++++++++++++++++++++++++++++++
 support/scripts/shell/utils.sh   | 64 ++++++++++++++++++++++++++++++++++++
 4 files changed, 229 insertions(+)
 create mode 100755 support/scripts/check-host-leaks

diff --git a/support/scripts/check-host-leaks b/support/scripts/check-host-leaks
new file mode 100755
index 0000000..42da272
--- /dev/null
+++ b/support/scripts/check-host-leaks
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  cat <<EOF >&2
+Usage:  ${0} TREE_DIR TOPDIR BASE_DIR HOST_DIR STAGING_DIR
+
+Description:
+
+        This script scans a tree for host paths leaks into it.
+        Print out leaks along side to their kind.
+
+Arguments:
+
+        TREE_DIR        Path to the root of the tree to be scaned
+
+        TOPDIR          Buildroot's TOPDIR
+
+        BASE_DIR        Buildroot's build output base location
+
+        HOST_DIR        Buildroot's host directory
+
+        STAGING_DIR     Buildroot's staging directory
+
+EOF
+}
+
+source "${0%/*}/shell/source.sh"
+
+READELF=readelf
+
+source.load_module utils
+source.load_module sdk
+
+main() {
+    if test ${#} -ne 5 ; then
+        usage
+        exit 1
+    fi
+    local to_scan="${1}"
+    local topdir="${2}" basedir="${3}" hostdir="${4}" stagingdir="${5}"
+    local gnu_target_name=$( utils.guess_gnu_target_name "${stagingdir}" )
+    local tree path
+    sdk.check_host_leaks "${gnu_target_name}" "${to_scan}" \
+        "${topdir}" "${basedir}" "${hostdir}"
+}
+
+main "${@}"
diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
index e0549d2..81ebbb8 100644
--- a/support/scripts/shell/readelf.sh
+++ b/support/scripts/shell/readelf.sh
@@ -21,6 +21,8 @@
 #   readelf.get_rpath
 #   readelf.needs_rpath
 #   readelf.has_rpath
+#   readelf.list_sections
+#   readelf.string_section
 #
 # This module is sensitive to the following environment variables:
 #   READELF
@@ -118,3 +120,32 @@ readelf.has_rpath() {
 
     return 1
 }
+
+# readelf.list_sections file
+#
+# Return the ELF section list of $file.
+#
+# file    : ELF file path
+#
+# environment:
+#   READELF: readelf program path
+readelf.list_sections() {
+    local file="${1}"
+    "${READELF}" --sections "${file}" |
+        sed -re '/^  \[ *[0-9]+\] (\S+).*/!d ; s//\1/' |
+        sort
+}
+
+# readelf.string_section file section
+#
+# Return the given $section of $file.
+#
+# file    : ELF file path
+# section : ELF section name
+#
+# environment:
+#   READELF: readelf program path
+readelf.string_section() {
+    local file="${1}" section="${2}"
+    "${READELF}" --string-dump "${section}" "${file}" 2>/dev/null
+}
diff --git a/support/scripts/shell/sdk.sh b/support/scripts/shell/sdk.sh
index 3188516..629f917 100644
--- a/support/scripts/shell/sdk.sh
+++ b/support/scripts/shell/sdk.sh
@@ -19,9 +19,16 @@
 # This module defines the following functions:
 #   sdk.compute_relative_path
 #   sdk.compute_xrpath
+#   sdk.check_host_leaks
+#
+# This module is sensitive to the following environment variables:
+#   READELF
 
 source.declare_module sdk
 
+source.load_module utils
+source.load_module readelf
+
 # sdk.compute_relative_path basedir path start
 #
 # Compute and return the relative path between $start and $path within $basedir.
@@ -68,3 +75,67 @@ sdk.compute_xrpath() {
     done
     sed -e 's/ /:/g' <<<"${rpath[@]}"
 }
+
+# sdk.check_host_leaks gnu_target_name root pattern_basedir...
+#
+# Scan the $root tree for the given $pattern_basedir pattern leaks.
+# The $gnu_target_name is used to skip the sysroot location when
+# scanning the host tree.
+# Categorize the type of leaks.
+# Print the leaks categories and and the matching file path.
+#
+# gnu_target_name : GNU target name (e.g. arm-buildroot-linux-gnueabihf)
+# root            : path to the root of the tree to be scanned
+# pattern_basedir : list of patterns to be searched
+sdk.check_host_leaks() {
+    local target="${1}" root="${2}"
+    local patterns=()
+    local pattern
+    shift 2
+    for pattern in ${@} ; do
+        patterns+=( "${pattern}" "$( readlink -f "${pattern}" )" )
+    done
+    patterns=( $( utils.list_reduce ${patterns[@]} ) )
+
+    local regexp="$( sed -re 's/^/(/ ; s/$/)/ ; s/ +/|/g' <<<"${patterns[*]}" )"
+    ( cd "${root}"
+        local f leak
+        while read f ; do
+            leak=
+            if test -h "${f}" ; then leak="symlink"
+            elif utils.is_elf "${f}" ;then
+                if utils.is_elf_executable "${f}" ; then leak="ELF/exe"
+                elif utils.is_elf_shared_object "${f}" ; then leak="ELF/*.so"
+                else
+                    case "${f}" in
+                        *.ko) leak="ELF/*.ko" ;;
+                        *) leak="ELF/*.o" ;;
+                    esac
+                fi
+                local section
+                local sections=()
+                for section in $( READELF="${READELF}" readelf.list_sections "${f}" ) ; do
+                    if READELF="${READELF}" readelf.string_section "${f}" "${section}" |
+                            grep -qaE "${regexp}" ; then
+                        sections+=( "${section}" )
+                    fi
+                done
+                leak="${leak} [${sections[*]}]"
+            else
+                case "${f}" in
+                    *.a)
+                        if utils.is_elf_static_lib "${f}" ; then leak="ELF/*.a" ; fi
+                        ;;
+                    *.la) leak="*.la" ;;
+                    *.pc) leak="*.pc" ;;
+                    *.py) leak="*.py" ;;
+                    *.pyc) leak="*.pyc" ;;
+                esac
+            fi
+            if test -z "${leak}" ; then
+                leak="? [$( file -z "${f}" | sed -e 's/.*: //' )]"
+            fi
+            printf "%-70s : %-120s\n" "${leak}" "${f}"
+        done < <( grep -raEl "${regexp}" . | sed -re "/${target}\/sysroot\//d ; s:^\.:${root}:" )
+    ) | sort
+}
diff --git a/support/scripts/shell/utils.sh b/support/scripts/shell/utils.sh
index b49a541..15c75f0 100644
--- a/support/scripts/shell/utils.sh
+++ b/support/scripts/shell/utils.sh
@@ -18,8 +18,13 @@
 #
 # This module defines the following functions:
 #   utils.list_reduce
+#   utils.filter_elf_shared_object
 #   utils.filter_elf_executable
 #   utils.filter_elf
+#   utils.is_elf_shared_object
+#   utils.is_elf_executable
+#   utils.is_elf
+#   utils.guess_gnu_target_name
 
 source.declare_module utils
 
@@ -43,6 +48,23 @@ utils.list_reduce() {
     echo ${lout[@]}
 }
 
+# utils.filter_elf_shared_object filepath...
+#
+# Filter ELF files; if $file is an ELF file, $file is print, else it is
+# discarded.
+# This funtion can take one or several arguments, or read them from stdin.
+#
+# file : path of file to be filtered
+utils.filter_elf_shared_object() {
+    local in file
+    test ${#} -gt 0 && in='printf "%s\n" ${@}' || in='dd 2>/dev/null'
+    eval "${in}" |
+        while read file ; do
+            file "${file}" 2>/dev/null |
+                sed -r -e '/^([^:]+):.*\<ELF\>.*\<shared object\>.*/!d ; s//\1/'
+        done
+}
+
 # utils.filter_elf_executable filepath...
 #
 # Filter ELF files; if $file is an ELF file, $file is print, else it is
@@ -76,3 +98,45 @@ utils.filter_elf() {
                 sed -r -e '/^([^:]+):.*\<ELF\>.*/!d ; s//\1/'
         done
 }
+
+# utils.is_elf_shared_object file
+#
+# Return 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+utils.is_elf_shared_object() {
+    test "$(utils.filter_elf_shared_object "${1}")" != ""
+}
+
+# utils.is_elf_executable file
+#
+# Return 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+utils.is_elf_executable() {
+    test "$(utils.filter_elf_executable "${1}")" != ""
+}
+
+# utils.is_elf file
+#
+# Return 0 if $file is an ELF file, non-0 otherwise.
+#
+# file : path of file to be tested
+utils.is_elf() {
+    test "$(utils.filter_elf "${1}")" != ""
+}
+
+# utils.guess_gnu_target_name sysroot
+#
+# Guess the GNU target name from the given sysroot location.
+# This assumes the sysroot is located in:
+#   <somewhere>/<GNU_target_name>/sysroot
+#
+# sysroot : path to the sysroot
+utils.guess_gnu_target_name() {
+    local sysroot="${1}"
+    sysroot="$( readlink -f "${sysroot}" )"
+    local gnu_target_name="${sysroot%/sysroot*}"
+    gnu_target_name="${gnu_target_name##*/}"
+    printf "%s" "${gnu_target_name}"
+}
-- 
2.7.0

[Buildroot] [PATCH v6 13/13] core: add check-leaks-in-{target, host, staging} targets

[Samuel Martin]

This new targets allow identifying what need to be fixed to fulfill the
ultimate goal of removing any trace of the host in the target filesystem
or the SDK.

Signed-off-by: Samuel Martin <s.martin49@gmail.com>

---
changes v5->v6:
- new patch
---
 Makefile             | 15 +++++++++++++++
 package/pkg-utils.mk |  5 +++++
 2 files changed, 20 insertions(+)

diff --git a/Makefile b/Makefile
index eb8a037..a74a016 100644
--- a/Makefile
+++ b/Makefile
@@ -616,6 +616,21 @@ endef
 TARGET_FINALIZE_HOOKS += PURGE_LOCALES
 endif
 
+# Checks for build machine leaks into target filesystem, SDK
+check-leaks-in-target:
+	@$(call MESSAGE,Checking leaks in the target)
+	$(Q)$(call check-for-build-machine-leaks-in,$(TARGET_DIR))
+
+check-leaks-in-host:
+	@$(call MESSAGE,Checking leaks in the host)
+	$(Q)$(call check-for-build-machine-leaks-in,$(HOST_DIR))
+
+check-leaks-in-staging:
+	@$(call MESSAGE,Checking leaks in the staging)
+	$(Q)$(call check-for-build-machine-leaks-in,$(STAGING_DIR))
+
+check-leaks:	check-leaks-in-target check-leaks-in-host check-leaks-in-staging
+
 # RPATH fixing
 # - The host hook sets RPATH in host ELF binaries, using relative paths to the
 #   library locations.
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 44bd2c9..21ad5ae 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -150,3 +150,8 @@ define legal-license-file # pkg, filename, file-fullpath, {HOST|TARGET}
 	mkdir -p $(LICENSE_FILES_DIR_$(4))/$(1)/$(dir $(2)) && \
 	cp $(3) $(LICENSE_FILES_DIR_$(4))/$(1)/$(2)
 endef
+
+define check-for-build-machine-leaks-in
+	$(TOPDIR)/support/scripts/check-host-leaks $(1) \
+		$(TOPDIR) $(BASE_DIR) $(HOST_DIR) $(STAGING_DIR)
+endef
-- 
2.7.0

[Buildroot] [PATCH v6 01/13] package/linux-headers: cleanup installation

[Thomas Petazzoni]

Dear Samuel Martin,

On Mon,  1 Feb 2016 16:53:29 +0100, Samuel Martin wrote:
> This patch removes unnecessary files generated during installation.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5 -> v6:
> - new patch
> ---
>  package/linux-headers/linux-headers.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
> index 1b8b81b..872ef8f 100644
> --- a/package/linux-headers/linux-headers.mk
> +++ b/package/linux-headers/linux-headers.mk
> @@ -53,6 +53,7 @@ define LINUX_HEADERS_INSTALL_STAGING_CMDS
>  			HOSTCXX="$(HOSTCXX)" \
>  			INSTALL_HDR_PATH=$(STAGING_DIR)/usr \
>  			headers_install)
> +	find $(STAGING_DIR)/usr \( -name .install -o -name ..install.cmd \) -delete

Why? Are they causing specific problems? Some better justification in
the commit log would be good, because we generally don't add code to
clean up things in staging.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v6 04/13] core: staging symlink uses a relative path when possible

[Thomas Petazzoni]

Dear Samuel Martin,

On Mon,  1 Feb 2016 16:53:32 +0100, Samuel Martin wrote:
> A step forward toward a relocatable SDK.
> 

Better commit log needed.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v6 06/13] core: make host *-config scripts relocatable

[Thomas Petazzoni]

Dear Samuel Martin,

On Mon,  1 Feb 2016 16:53:34 +0100, Samuel Martin wrote:

> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 1bf613e..ca79450 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -207,6 +207,18 @@ $(BUILD_DIR)/%/.stamp_host_installed:
>  	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
>  	+$($(PKG)_INSTALL_CMDS)
>  	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
> +	$(Q)if test -n "$($(RAWNAME)_CONFIG_SCRIPTS)" ; then \

So you're testing if the current package has defined some
<pkg>_CONFIG_SCRIPTS variable....

> +		$(call MESSAGE,"Fixing package configuration files") ;\
> +			ls $(HOST_DIR)/usr/bin/*-config 2>/dev/null |\

But do the replacement on all *-config scripts ?

This doesn't look good, and I think we should be changing just the ones
that are in <pkg>_CONFIG_SCRIPTS.

I read your commit log saying that this is a bit complicated due to
some packages installing config scripts whose name depend on the host
system. Can you give some specific examples to see if we can address
this ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v6 02/13] core: use $(CURDIR) to set TOPDIR

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> This changes saves a shell call and uses a variable automatically set
> by make [1].
> 
> [1] http://www.gnu.org/software/make/manual/make.html#Quick-Reference

 But then we should maybe remove TOPDIR, and replace it with CURDIR? You can't
change directories in make, so CURDIR will always be valid.

 Regards,
 Arnout

> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5 -> v6:
> - new patch
> ---
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 17c181b..13c16af 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -69,7 +69,7 @@ endif
>  .NOTPARALLEL:
>  
>  # absolute path
> -TOPDIR := $(shell pwd)
> +TOPDIR := $(CURDIR)
>  CONFIG_CONFIG_IN = Config.in
>  CONFIG = support/kconfig
>  DATE := $(shell date +%Y%m%d)
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 03/13] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> When $(CURDIR) or $(O) contain symlinks (or mount-bind) in their path,
> they can be resolved differently, depending on each package build-system
> (whether it uses the given paths or get the absolute canonical ones).
> 
> Thus, to make easier tracking down host machine paths leaking into the
> host, target or staging trees, the CURDIR and O variables are set to
> their absolute canonical paths.
> 
> Note that this change takes care of the makefile wrapper installed in
> $(O) to avoid unneeded make recursion.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5 -> v6:
> - new patch
> ---
>  Makefile | 22 ++++++++++++++++++----
>  1 file changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 13c16af..10193ba 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -26,16 +26,30 @@
>  
>  # Trick for always running with a fixed umask
>  UMASK = 0022
> -ifneq ($(shell umask),$(UMASK))
> +
> +# Check if we need to re-enter make for one or several of the following reasons:
> +# 1- Wrong (too restrictive) umask:
> +#    This prevents Buildroot and packages from creating files and directories.
> +# 2- CWD (i.e. $(CURDIR)) not being the absolute canonical path:
> +#    This makes harder tracking and fixing host machine path leaks.
> +# 3- Output location (i.e. $(O)) not being the absolute canonical path:
> +#    This makes harder tracking and fixing host machine path leaks.
> +#
> +# Note:
> +# - remove the trailing '/.' from $(O) as it can be added by the makefile
> +#   wrapper installed in the $(O).

 When is the trailing /. added? Can't we avoid that it is added by mkmakefile?

 Otherwise looks good to me.

 Regards,
 Arnout

> +ifneq ($(shell umask):$(CURDIR):$(patsubst %/.,%,$(O)),$(UMASK):$(realpath $(CURDIR)):$(realpath $(O)))
>  .PHONY: _all $(MAKECMDGOALS)
>  
>  $(MAKECMDGOALS): _all
>  	@:
>  
>  _all:
> -	@umask $(UMASK) && $(MAKE) --no-print-directory $(MAKECMDGOALS)
> +	umask $(UMASK) && \
> +		$(MAKE) -C $(realpath $(CURDIR)) --no-print-directory \
> +			$(MAKECMDGOALS) O=$(realpath $(O))
>  
> -else # umask
> +else # umask / $(CURDIR) / $(O)
>  
>  # This is our default rule, so must come first
>  all:
> @@ -1001,4 +1015,4 @@ include docs/manual/manual.mk
>  
>  .PHONY: $(noconfig_targets)
>  
> -endif #umask
> +endif #umask / $(CURDIR) / $(O)
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 06/13] core: make host *-config scripts relocatable

[Arnout Vandecappelle]

On 01-02-16 19:45, Thomas Petazzoni wrote:
> Dear Samuel Martin,
> 
> On Mon,  1 Feb 2016 16:53:34 +0100, Samuel Martin wrote:
> 
>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>> index 1bf613e..ca79450 100644
>> --- a/package/pkg-generic.mk
>> +++ b/package/pkg-generic.mk
>> @@ -207,6 +207,18 @@ $(BUILD_DIR)/%/.stamp_host_installed:
>>  	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
>>  	+$($(PKG)_INSTALL_CMDS)
>>  	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
>> +	$(Q)if test -n "$($(RAWNAME)_CONFIG_SCRIPTS)" ; then \
> 
> So you're testing if the current package has defined some
> <pkg>_CONFIG_SCRIPTS variable....
> 
>> +		$(call MESSAGE,"Fixing package configuration files") ;\
>> +			ls $(HOST_DIR)/usr/bin/*-config 2>/dev/null |\
> 
> But do the replacement on all *-config scripts ?
> 
> This doesn't look good, and I think we should be changing just the ones
> that are in <pkg>_CONFIG_SCRIPTS.
> 
> I read your commit log saying that this is a bit complicated due to
> some packages installing config scripts whose name depend on the host
> system. Can you give some specific examples to see if we can address
> this ?

 Indeed, I would prefer an explicit HOST_PKG_CONFIG_SCRIPTS variable. You can
easily look for the ones which are needed with

git grep -l _CONFIG_SCRIPTS | xargs grep -l 'host-.*-package'

 And since this fixup is completely new, it can be a separate patch per package.


 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 06/13] core: make host *-config scripts relocatable

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> This change adjusts the _CONFIG_SCRIPTS hook to set add {exec_,}prefix computed
> relatively to the script location.
> 
> This patch only fixes *-config scripts located in the host area, those
> from the target and staging area are already handled.
> 
> This hook is a kind of brute-force sed on all files matching '*-config'
> in the $(HOST_DIR)/usr/bin directory. This allows correctly handle
> script whose name may depends on the host system, not the target
> configuration (e.g. ncurses can be configured for the target to be
> ncurses6w, so ncurses6w-config ; whereas for the host it will be
> ncurses5, so ncurses5-config).
> Since there way less host packages than target ones, it is reasonable to
> assume that processing all *-config scripts found when a host package
> (whose target package set the *_CONFIG_SCRIPTS variable) is fairly short
> and does not add its overhead not even noticeable.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5 -> v6:
> - new patch
> ---
>  package/pkg-generic.mk | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 1bf613e..ca79450 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -207,6 +207,18 @@ $(BUILD_DIR)/%/.stamp_host_installed:
>  	$(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep))
>  	+$($(PKG)_INSTALL_CMDS)
>  	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
> +	$(Q)if test -n "$($(RAWNAME)_CONFIG_SCRIPTS)" ; then \
> +		$(call MESSAGE,"Fixing package configuration files") ;\
> +			ls $(HOST_DIR)/usr/bin/*-config 2>/dev/null |\
> +			xargs --no-run-if-empty \
> +			$(SED)  "s,$(BASE_DIR), at BASE_DIR@,g" \
> +				-e "s,$(HOST_DIR), at HOST_DIR@,g" \
> +				-e "s,^\(exec_\)\?prefix=.*,\1prefix=\`dirname \$$0\`/../../usr,g" \
> +				-e "s,-I/usr/,-I at HOST_DIR@/usr/,g" \
> +				-e "s,-L/usr/,-L at HOST_DIR@/usr/,g" \

 Is this correct? Host packages are installed with --prefix=$(HOST_DIR), so the
configure script should already have the correct host dir in there. If it does
have /usr/something/, it's most likely because it links with a system-installed
package in a non-standard path. So I think this change will break things.


 Regards,
 Arnout

> +				-e "s, at HOST_DIR@,$(HOST_DIR),g" \
> +				-e "s, at BASE_DIR@,$(BASE_DIR),g" ;\
> +	fi
>  	@$(call step_end,install-host)
>  	$(Q)touch $@
>  
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 05/13] core: make staging *-config scripts relocatable

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> This change adjusts the _CONFIG_SCRIPTS hook to set add {exec_,}prefix computed
> relatively to the script location.
> 
> This patch hook only fixes *-config scripts located in the staging area,
> the target ones are already removed. A follow-up change will fix those
> from the HOST_DIR location.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5 -> v6:
> - new patch
> ---
>  package/pkg-generic.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 1e024d3..1bf613e 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -240,7 +240,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
>  		$(call MESSAGE,"Fixing package configuration files") ;\
>  			$(SED)  "s,$(BASE_DIR), at BASE_DIR@,g" \
>  				-e "s,$(STAGING_DIR), at STAGING_DIR@,g" \
> -				-e "s,^\(exec_\)\?prefix=.*,\1prefix=@STAGING_DIR@/usr,g" \
> +				-e "s,^\(exec_\)\?prefix=.*,\1prefix=\`dirname \$$0\`/../../usr,g" \
>  				-e "s,-I/usr/,-I at STAGING_DIR@/usr/,g" \
>  				-e "s,-L/usr/,-L at STAGING_DIR@/usr/,g" \
>  				-e "s, at STAGING_DIR@,$(STAGING_DIR),g" \

 Since this is still there, it is not really relocatable yet... To make it
really relocatable, we should probably use $${BR_STAGING_DIR}} and export
BR_STAGING_DIR=$(STAGING_DIR).

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 07/13] support/scripts: add fix-rpath script + a bunch of helpers

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> This commit introduces a fix-rpath shell-script able to scan a tree,
> detect ELF files, check their RPATH and fix it in a proper way.
> 
> Along to the fix-rpath script, it also adds a bunch of shell helper
> functions grouped into modules. This will help writing scripts handling
> RPATH and other things, while allowing to share and reuse these
> functions between scripts.
> 
> These helpers are namespaced within the filename of the module in which
> they are gathered.
> 
> This change adds 6 modules:
> - source.sh: provides simple helper to easily source another module, taking
>     care of not sourcing again an already-sourced one;
> - log.sh: provides logging helpers;
> - utils.sh: provides simple functions to filter ELF files in a list;
> - readelf.sh: provides functions retrieving ELF details from a file;
> - patchelf.sh: provides function updating ELF files;
> - sdk.sh: provides RPATH computation functions.
> 
> These 6 modules are used by the fix-rpath script.
> Follow-up patches will make some scripts leveraging these modules.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5->v6:
> - fully rewritten in shell
> 
> changes v4->v5:
> - add verbose support
> - rename shrink_rpath -> clear_rpath
> - add sanitize_rpath function
> 
> changes v3->v4:
> - fix typos and license (Baruch)
> 
> changes v2->v3:
> - no change
> ---
>  support/scripts/fix-rpath         | 101 +++++++++++++++++++++++
>  support/scripts/shell/log.sh      |  57 +++++++++++++
>  support/scripts/shell/patchelf.sh | 163 ++++++++++++++++++++++++++++++++++++++
>  support/scripts/shell/readelf.sh  |  52 ++++++++++++
>  support/scripts/shell/sdk.sh      |  70 ++++++++++++++++
>  support/scripts/shell/source.sh   |  73 +++++++++++++++++
>  support/scripts/shell/utils.sh    |  60 ++++++++++++++
>  7 files changed, 576 insertions(+)
>  create mode 100755 support/scripts/fix-rpath
>  create mode 100644 support/scripts/shell/log.sh
>  create mode 100644 support/scripts/shell/patchelf.sh
>  create mode 100644 support/scripts/shell/readelf.sh
>  create mode 100644 support/scripts/shell/sdk.sh
>  create mode 100644 support/scripts/shell/source.sh
>  create mode 100644 support/scripts/shell/utils.sh
> 
> diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
> new file mode 100755
> index 0000000..938e599
> --- /dev/null
> +++ b/support/scripts/fix-rpath
> @@ -0,0 +1,101 @@
> +#!/usr/bin/env bash
> +
> +# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +usage() {
> +  cat <<EOF >&2
> +Usage:  ${0} TREE_KIND TREE_ROOT
> +
> +Description:
> +
> +        This script scans a tree and sanitize ELF files' RPATH found in there.

 ... sanitizes the RPATH of ELF files found in there.

> +
> +        Sanitization behaves the same whatever the kindd of the processed tree, but

 Please wrap at 80 columns. But actually, I think 4 spaces is enough for
indentation, that's what you use below as well.

 kindd -> kind

 But actually I don't really understand this sentence (well, I do after reading
the rest). So I'd just remove this sentence and add...

> +        the resulting RPATH differs.
> +
> +        Sanitization action:

 Sanitization action (for all trees):

> +        - remove RPATH pointing outside of the tree
> +        - for RPATH pointing in the tree:
> +          - if they point to standard location (/lib, /usr/lib): remove them
> +          - otherwise: make them relative using \$ORIGIN
> +
> +        For the target tree:
> +        - scan the whole tree for sanitization
> +
> +        For the staging tree :
> +        - scan the whole tree for sanitization
> +
> +        For the host tree:
> +        - skip the staging tree for sanitization
> +        - add \$HOST_DIR/{lib,usr/lib} to RPATH (as relative pathes)

 pathes -> paths

> +
> +Arguments:
> +
> +        TREE_KIND   Kind of tree to be processed.
> +                    Allowed values: host, target, staging
> +
> +        TREE_ROOT   Path to the root of the tree to be scaned

 scanned

> +
> +EOF
> +}
> +
> +source "${0%/*}/shell/source.sh"
> +
> +source.load_module utils
> +source.load_module readelf
> +source.load_module patchelf

 What is this, python? :-) I think this source.sh is a bit over the top (it
doesn't really hurt to source those modules a second time), but why not.

> +
> +main() {
> +    local tree="${1}"
> +    local basedir="$( readlink -f "${2}" )"
> +
> +    local find_args=( "${basedir}" )
> +    local sanitize_extra_args=()
> +    local readelf
> +
> +    case "${tree}" in
> +        host)
> +            # do not process the sysroot (only contains target binaries)
> +            find_args+=( "-name" "sysroot" "-prune" "-o" )

 There may be some other random file that is called sysroot, so to be safe I'd
make this

    find_args+=( "-path" "${basedir}/usr/*/sysroot" "-prune" "-o" )

(untested).

> +
> +            # do not process the external toolchain installation directory to
> +            # to avoid breaking it.
> +            find_args+=( "-path" "*/opt/ext-toolchain" "-prune" "-o" )

 Again, to be safe against a opt/ext-toolchain appearing somewhere else in the
tree (admittedly, this is ridiculously unlikely), use . instead of *.

> +
> +            # make sure RPATH will point to ${hostdir}/lib and ${hostdir}/usr/lib
> +            sanitize_extra_args+=( "keep_lib_and_usr_lib" )

 Since sanitize_extra_args only contains a single argument, I don't think it's
worth making it an array. Just assign empty above and assign
keep_lib_and_usr_lib here, and pass unquoted below. But that's just MHO.

> +
> +            readelf="${HOST_READELF}"
> +            ;;
> +        staging|target)
> +            readelf="${TARGET_READELF}"
> +            ;;
> +        *)
> +            usage
> +            exit 1
> +            ;;
> +    esac
> +
> +    find_args+=( "-type" "f" "-print" )
> +
> +    while read file; do
> +        READELF="${READELF}" PATCHELF="${PATCHELF}" \

 Shouldn't that be ${readelf}? And where does ${PATCHELF} come from? If it comes
from the environment, there's no need to export it again here.

 But maybe it's cleaner to just export READELF inside the case statement, above.

> +            patchelf.sanitize_rpath "${basedir}" "${file}" ${sanitize_extra_args[@]}
> +    done < <( find ${find_args[@]} | utils.filter_elf )

 I personally don't like this reverse construct, and would write:

    find ${find_args[@]} | utils.filter_elf | \
        while read file; do
            ...

 Again, MHO.

> +}
> +
> +main ${@}
> diff --git a/support/scripts/shell/log.sh b/support/scripts/shell/log.sh
> new file mode 100644
> index 0000000..ffa19e8
> --- /dev/null
> +++ b/support/scripts/shell/log.sh
> @@ -0,0 +1,57 @@
> +# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +# Logging helpers
> +#
> +# This module defines the following functions:
> +#   log.trace
> +#   log.debug
> +#   log.info
> +#   log.warn
> +#   log.errorN
> +#   log.error
> +#
> +# This module sets the following variables:
> +#   my_name
> +#
> +# This module is sensitive to the following environment variables:
> +#   DEBUG
> +
> +source.declare_module log
> +
> +# Debug level:
> +# - 0 or empty: only show errors
> +# - 1         : show errors and warnings
> +# - 2         : show errors, warnings, and info
> +# - 3         : show errors, warnings, info and debug
> +: ${DEBUG:=0}
> +
> +# Low level utility function
> +log.trace()  { local msg="${1}"; shift; printf "%s: ${msg}" "${my_name}" "${@}"; }

 I'd do the redirect directly here in the printf, rather than below.

 Also, I see no reason to put it all on one line.

> +
> +# Public logging functions
> +log.debug()  { :; }
> +[ ${DEBUG} -lt 3 ] || log.debug() { log.trace "${@}" >&2; }
> +log.info()   { :; }
> +[ ${DEBUG} -lt 2 ] || log.info()  { log.trace "${@}" >&2; }
> +log.warn()   { :; }
> +[ ${DEBUG} -lt 1 ] || log.warn()  { log.trace "${@}" >&2; }
> +log.errorN() { local ret="${1}"; shift; log.warn "${@}"; exit ${ret}; }
> +log.error()  { log.errorN 1 "${@}"; }
> +
> +# Program name
> +my_name="${0##*/}"

 Isn't my_name always going to be log.sh? "source" doesn't propagate the
parent's arguments AFAIK.

> +
> diff --git a/support/scripts/shell/patchelf.sh b/support/scripts/shell/patchelf.sh
> new file mode 100644
> index 0000000..d1eb590
> --- /dev/null
> +++ b/support/scripts/shell/patchelf.sh
> @@ -0,0 +1,163 @@
> +# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +# Patchelf helpers

 AFAICS these helper will only ever be used in the fix-rpaths script (but I
haven't read the entire series yet). Unless you can see a good use case for
using these functions in another script, I would prefer to see them defined
directly in the fix-rpaths script rather than a separate module. IMHO it's
easier to understand related code if it's in one file than if it's in separate
files.

> +#
> +# This module defines the following functions:
> +#   patchelf.set_xrpath
> +#   patchelf.update_rpath
> +#   patchelf.sanitize_rpath
> +#
> +# This module is sensitive to the following environment variables:
> +#   PATCHELF
> +#   READELF
> +
> +source.declare_module patchelf
> +
> +source.load_module log
> +source.load_module sdk
> +source.load_module utils
> +source.load_module readelf
> +
> +: ${PATCHELF:=patchelf}
> +
> +# patchelf.set_xrpath file rpath...
> +#
> +# Set RPATH in $file.
> +# Automatically join all RPATH with the correct separator, and also handle
> +# XRPATH (replacing "XORIGIN" with "$ORIGIN").
> +#
> +# file  : ELF file path
> +# rpath : RPATH element
> +#
> +# environment:
> +#   PATCHELF: patchelf program path
> +patchelf.set_xrpath() {
> +    local file="${1}"
> +    shift
> +    local xrpath="$(sed -e 's/ +/:/g' <<<"${@}")"

 In fix-rpath you put spaces inside the "$( ... )". I think this would be a good
idea here as well.

> +    "${PATCHELF}" --set-rpath "${xrpath//XORIGIN/\$ORIGIN}" "${file}"
> +}
> +
> +# patchelf.update_rpath basedir binary libdirs...
> +#
> +# Set RPATH in $binary computing them from the paths $libdirs (and $basedir).
> +# Existing RPATH in $file will be overwritten if any.

 $file -> $binary ?

> +#
> +# basedir : absolute path of the tree in which the $bindir and $libdirs must be
> +# binary  : ELF file absolute path
> +# libdirs : list of library location (absolute paths)

 location -> locations

> +#
> +# environment:
> +#   PATCHELF: patchelf program path
> +patchelf.update_rpath() {
> +    local basedir="${1}"
> +    local binary="${2}"
> +    shift 2
> +    local libdirs=( ${@} )
> +    log.debug "  basedir: %s\n" "${basedir}"
> +    log.debug "      elf: %s\n" "${binary}"
> +    log.debug "  libdirs: %s\n" "${libdirs[*]}"
> +    log.info  "    rpath: %s\n" \
> +        "$( sdk.compute_xrpath "${basedir}" "${binary%/*}" ${libdirs[@]} |
> +            sed -e 's/XORIGIN/\$ORIGIN/g' )"

 Maybe compute it once and store it in a variable?

 Alternatively, add a helper

log.tracerun() {
    log.info "%s\n" "$*"
    "$@"
}

and use it below.

> +    PATCHELF="${PATCHELF}" patchelf.set_xrpath "${binary}" \

 Again, PATCHELF is already in the environment.

> +        $( sdk.compute_xrpath "${basedir}" "${binary%/*}" ${libdirs[@]} )
> +}
> +
> +# patchelf.sanitize_rpath basedir binary [keep_lib_usr_lib]
> +#
> +# Scan $binary's RPATH, remove any of them pointing outside of $basedir.
> +# If $keep_lib_usr_lib in not empty, the library directories $basedir/lib and

 in -> is

> +# $basedir/usr/lib will be added to the RPATH.
> +#
> +# Note:
> +#     Absolute paths is needed to correctly handle symlinks and or mount-bind in

 is -> are
 and or -> and/or
 mount-bind -> bind-mount (though actually, I don't see how you're going to get
an absolute path through a bind mount)

> +#     the $basedir path.
> +#
> +# basedir          : absolute path of the tree in which the $bindir and $libdirs
> +#                    must be
> +# binary           : ELF file absolute path
> +# keep_lib_usr_lib : add to RPATH $basedir/lib and $basedir/usr/lib
> +#
> +# environment:
> +#   PATCHELF: patchelf program path
> +#   READELF : readelf program path
> +patchelf.sanitize_rpath() {
> +    local basedir="$( readlink -f "${1}" )"
> +    local binary="${2}"
> +    local keep_lib_usr_lib="${3}"
> +
> +    local path abspath rpath
> +    local libdirs=()

 I think new_rpaths is a better name here.

> +
> +    if test -n "${keep_lib_usr_lib}" ; then
> +        libdirs+=( "${basedir}/lib" "${basedir}/usr/lib" )
> +    fi
> +
> +    log.info "ELF: %s\n" "${binary}"
> +
> +    local rpaths="$( READELF="${READELF}" readelf.get_rpath "${binary}" )"

 Again, READELF is already in the environment.

> +
> +    for rpath in ${rpaths//:/ } ; do
> +        # figure out if we should keep or discard the path; there are several
> +        # cases to handled:
> +        # - $path starts with "$ORIGIN":
> +        #     The original build-system already took care of setting a relative
> +        #     RPATH, resolve it and test if it is worthwhile to keep it;
> +        # - $basedir/$path exists:
> +        #     The original build-system already took care of setting an absolute
> +        #     RPATH (absolute in the final rootfs), resolve it and test if it is
> +        #     worthwhile to keep it;
> +        # - $path start with $basedir:

 start -> starts

> +        #     The original build-system added some absolute RPATH (absolute on
> +        #     the build machine). While this is wrong, it can still be fixed; so
> +        #     test if it is worthwhile to keep it;
> +        # - $path points somewhere else:
> +        #     (can be anywhere: build trees, staging tree, host location,
> +        #     non-existing location, etc.)
> +        #     Just discard such a path.

 This is not correct for host packages. If a host package has picked up an RPATH
to e.g. ~/lib/foo through pkg-config, then that RPATH must still be kept. But
only for host packages. For host packages, you should probably just not remove
anything.

 So I think you still need to distinguish between the host and staging/target
cases here. In fact, I think the whole logic of this function is easier to
understand if it is separate for host and target/staging.

> +        if grep -q '^$ORIGIN/' <<<"${rpath}" ; then
> +            path="${binary%/*}/${rpath#*ORIGIN/}"

 Maybe ?ORIGIN instead of *ORIGIN, just in case there's another ORIGIN somewhere
in the path?

> +        elif test -e "${basedir}/${rpath}" ; then

 -e -> -d, it really must be a directory.

> +            path="${basedir}/${rpath}"
> +        elif grep -q "^${basedir}/" <<<"$( readlink -f "${rpath}" )" ; then
> +            path="${rpath}"
> +        else
> +            log.debug "\tDROPPED [out-of-tree]: %s\n" "${rpath}"
> +            continue
> +        fi
> +
> +        abspath="$( readlink -f "${path}" )"
> +
> +        # discard path pointing to default locations handled by ld-linux

 ld-linux -> ld.so

> +        if grep -qE "^${basedir}/(lib|usr/lib)$" <<<"${abspath}" ; then
> +            log.debug \
> +                "\tDROPPED [std libdirs]: %s (%s)\n" "${rpath}" "${abspath}"
> +            continue
> +        fi
> +
> +        log.debug "\tKEPT %s (%s)\n" "${rpath}" "${abspath}"
> +
> +        libdirs+=( "${abspath}" )
> +
> +    done
> +
> +    libdirs=( $( utils.list_reduce ${libdirs[@]} ) )
> +
> +    PATCHELF="${PATCHELF}" \

 Again, already in the env.

> +        patchelf.update_rpath "${basedir}" "${binary}" ${libdirs[@]}
> +}
> diff --git a/support/scripts/shell/readelf.sh b/support/scripts/shell/readelf.sh
> new file mode 100644
> index 0000000..82968a2
> --- /dev/null
> +++ b/support/scripts/shell/readelf.sh
> @@ -0,0 +1,52 @@
> +# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> +# General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +# Readelf helpers
> +#
> +# This module defines the following functions:
> +#   readelf.get_rpath
> +#
> +# This module is sensitive to the following environment variables:
> +#   READELF
> +#
> +# This module sets the following environment variables:
> +#   LC_ALL=C
> +
> +source.declare_module readelf
> +
> +# Override the user's locale so we are sure we can parse the output of
> +# readelf(1) and file(1)
> +export LC_ALL=C
> +
> +: ${READELF:=readelf}
> +
> +# readelf.get_rpath file
> +#
> +# Return the unsplitted RPATH/RUNPATH of $file.

 unsplitted -> unsplit.

 But it's actually not return, it's print. And to make it very explicit, write:

Print the :-separate RPATH/RUNPATH of $file.

> +#
> +# To split the returned RPATH string and store them in an array, do:
> +#
> +#     paths=( $(readelf.get_rpath "${file}" | sed -e 's/:/ /g') )
> +#
> +# file : ELF file path
> +#
> +# environment:
> +#   READELF: readelf program path
> +readelf.get_rpath() {
> +    local file="${1}"
> +    "${READELF}" --dynamic "${file}" |
> +        sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d ; s//\3/'
> +}

 I have to go now, so reviewed until here. The rest I can hopefully review
tomorrow...

 Regards,
 Arnout


[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 08/13] core: add HOST_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> This patch adds host-patchelf as a target-finalize dependency, and
> introduces the HOST_SANITIZE_RPATH_HOOK hook to fix the ELF files'
> RPATH from the HOST_DIR location (excluding the sysroot).
> 
> After running this hook, the RPATH from any host ELF files is relative to
> the binary location itself.
> 
> Notes:
> - we avoid to fix RPATH in the sysroot.
> - we do not try to fix RPATH in the external toolchain installation
>   location as they may have been built in a way, this is already correct;
>   furthermore, fixing RPATH in those programs may result in breaking them.
> - the whole host directory is processed because a number of
>   host-package install programs that could be useful in places
>   different from $(HOST_DIR)/{bin,sbin,usr/bin,usr/sbin}.
> - the shared libraries are also processed in case they have a 'main'
>   function.
> 
> As a step toward a fully relocatable SDK, this change allows to get the
> toolchain relocatable, but not yet the whole SDK.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5->v6:
> - update for the new script version
> - add debug mode support
> 
> changes v4->v5:
> - add verbose support
> 
> changes v3->v4:
> - add host-patchelf to PACKAGES instead of target-finalize (Baruch)
> - add comment
> 
> changes v2->v3:
> - move hook in Makefile (Baruch)
> ---
>  Makefile | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 7c87177..b1f4fcf 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -616,6 +616,31 @@ endef
>  TARGET_FINALIZE_HOOKS += PURGE_LOCALES
>  endif
>  
> +# RPATH fixing
> +# - The host hook sets RPATH in host ELF binaries, using relative paths to the
> +#   library locations.
> +# - The target hook sanitizes RPATH in target ELF binaries, removing paths
> +#   pointing to package's build directories or the sysroot's libdirs.
> +PACKAGES += host-patchelf
> +
> +ifeq ($(BR2_DEBUG_RPATH),)
> +BR2_DEBUG_RPATH = 0
> +endif
> +ifeq ($(BR2_DEBUG_RPATH),0)
> +DEBUG_RPATH = 0
> +else
> +DEBUG_RPATH = $(shell echo $$(($(BR2_DEBUG_RPATH)+1)))
> +endif

 I don't like this debug handling in the Makefile. For the toolchain wrapper,
the BR2_DEBUG_WRAPPER variable is used directly in the wrapper, so nothing
specific has to be done in the Makefile. So here as well I'd prefer to use
BR2_DEBUG_RPATH in the scripts.

 Since the logging stuff is generic, you will indeed have to convert it to the
DEBUG variable in the fix-rpath script, but IMHO that's much more acceptable.

 Also, it should be sufficient to set DEBUG=${BR_DEBUG_RPATH}. If
BR2_DEBUG_RPATH is not set, the ${DEBUG:=0} in log.sh will set it to 0.


> +
> +define HOST_SANITIZE_RPATH_HOOK
> +	DEBUG=$(DEBUG_RPATH) \
> +	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
> +	READELF=readelf \

 That's already set by the scripts, and actually it's probably better if the
user has the possibility to override readelf.


> +		$(TOPDIR)/support/scripts/fix-rpath host $(HOST_DIR)

 I find this indentation a bit weird. But OK.

 Regards,
 Arnout

> +endef
> +
> +TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
> +
>  $(TARGETS_ROOTFS): target-finalize
>  
>  target-finalize: $(PACKAGES)
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 09/13] core: add {TARGET, STAGING}_SANITIZE_RPATH_HOOK to TARGET_FINALIZE_HOOKS

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> This patch introduces the TARGET_SANITIZE_RPATH_HOOK and
> STAGING_SANITIZE_RPATH_HOOK hooks fixing the ELF files' RPATH of
> binaries from, respectively, the TARGET_DIR and the STAGING_DIR
> locations.
> 
> After running this hook, the RPATH from any target ELF files from both
> the target and the staging locations won't contain any occurence of the
> sysroot or some build locations.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> 
> ---
> changes v5->v6:
> - update for the new script version
> - add debug mode support
> 
> changes v4->v5:
> - target hook now sanitizes the rpath (Baruch)
> - add verbose support
> - update comment
> 
> changes v3->v4:
> - rebase
> - add comment
> 
> changes v2->v3:
> - move hook in Makefile (Baruch)
> ---
>  Makefile | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index b1f4fcf..eb8a037 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -641,6 +641,26 @@ endef
>  
>  TARGET_FINALIZE_HOOKS += HOST_SANITIZE_RPATH_HOOK
>  
> +# Function sanitizing target/staging ELF files' RPATH.
> +# i.e. it removes paths pointing to the staging or build location from the ELF
> +# files' RPATH.
> +define TARGET_SANITIZE_RPATH_HOOK
> +	DEBUG=$(DEBUG_RPATH) \
> +	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
> +	READELF=$(TARGET_READELF) \
> +		$(TOPDIR)/support/scripts/fix-rpath target $(TARGET_DIR)
> +endef
> +
> +define STAGING_SANITIZE_RPATH_HOOK
> +	DEBUG=$(DEBUG_RPATH) \
> +	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
> +	READELF=$(TARGET_READELF) \
> +		$(TOPDIR)/support/scripts/fix-rpath staging $(STAGING_DIR)

 Since target and staging are basically the same, I'd just make a single call to
fix-rpath and add a loop over basedirs there. So

	.../fix-rpath target $(TARGET_DIR) $(STAGING_DIR)

 Or actually, I'm not sure if it makes a lot of sense to have the fix-rpath
script the way it is now, because the only commonality between host and target
is the filter_elf step. But that could in fact easily be done inside the
sanitize_rpath function (which would check for an ELF file at start and exit 0
if it's not ELF).

 So you could have

define HOST_SANITIZE_RPATH_HOOK
	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
	find $(HOST_DIR) -path "$(HOST_DIR)/usr/*/sysroot" -prune -o \
		"$(HOST_DIR)/opt/ext-toolchain" -prune -o \
		-type f \
		-exec $(TOPDIR)/support/scripts/sanitize-rpath \
			"$(HOST_DIR)" '{}' keep_lib_and_usr_lib \;
endef

define TARGET_SANITIZE_RPATH_HOOK
	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
	READELF=$(TARGET_READELF) \
	find $(STAGING_DIR) -type f \
		-exec $(TOPDIR)/support/scripts/sanitize-rpath \
			"$(STAGING_DIR)" '{}' \;
	PATCHELF=$(HOST_DIR)/usr/bin/patchelf \
	READELF=$(TARGET_READELF) \
	find $(TARGET_DIR) -type f \
		-exec $(TOPDIR)/support/scripts/sanitize-rpath \
			"$(STAGING_DIR)" '{}' \;
endef

 OK, maybe some factorization should be possible here, but the fix-rpath script
is 100 lines long and those save only 3 lines in the Makefile, so really I don't
think that's a good refactoring...


 Regards,
 Arnout

> +endef
> +
> +TARGET_FINALIZE_HOOKS += TARGET_SANITIZE_RPATH_HOOK \
> +       STAGING_SANITIZE_RPATH_HOOK
> +
>  $(TARGETS_ROOTFS): target-finalize
>  
>  target-finalize: $(PACKAGES)
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 10/13] package/speex: remove no longer needed hook

[Arnout Vandecappelle]

On 01-02-16 16:53, Samuel Martin wrote:
> Remove the LIBTOOL_FIXUP hook since RPATH are now sanitized in
> target-finalize hooks.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


 Regards,
 Arnout

> 
> ---
> changes v5->v6:
> - no change
> 
> changes v4->v5:
> - new patch (suggested by Baruch)
> ---
>  package/speex/speex.mk | 5 -----
>  1 file changed, 5 deletions(-)
> 
> diff --git a/package/speex/speex.mk b/package/speex/speex.mk
> index a34bfac..24782ec 100644
> --- a/package/speex/speex.mk
> +++ b/package/speex/speex.mk
> @@ -24,11 +24,6 @@ ifeq ($(BR2_PACKAGE_SPEEX_ARM5E),y)
>  SPEEX_CONF_OPTS += --enable-arm5e-asm
>  endif
>  
> -define SPEEX_LIBTOOL_FIXUP
> -	$(SED) 's|^hardcode_libdir_flag_spec=.*|hardcode_libdir_flag_spec=""|g' $(@D)/libtool
> -	$(SED) 's|^runpath_var=LD_RUN_PATH|runpath_var=DIE_RPATH_DIE|g' $(@D)/libtool
> -endef
> -
>  SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
>  
>  define SPEEX_BUILD_CMDS
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v6 03/13] core: re-enter make if $(CURDIR) or $(O) are not absolute canonical path

[Thomas Petazzoni]

Hello,

On Mon,  1 Feb 2016 16:53:31 +0100, Samuel Martin wrote:
> When $(CURDIR) or $(O) contain symlinks (or mount-bind) in their path,
> they can be resolved differently, depending on each package build-system
> (whether it uses the given paths or get the absolute canonical ones).
> 
> Thus, to make easier tracking down host machine paths leaking into the
> host, target or staging trees, the CURDIR and O variables are set to
> their absolute canonical paths.
> 
> Note that this change takes care of the makefile wrapper installed in
> $(O) to avoid unneeded make recursion.
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>

I have to say I kind of hate how complicated our Makefile "entry" logic
has become over time. But well, certainly not your fault.

> +# Check if we need to re-enter make for one or several of the following reasons:
> +# 1- Wrong (too restrictive) umask:
> +#    This prevents Buildroot and packages from creating files and directories.
> +# 2- CWD (i.e. $(CURDIR)) not being the absolute canonical path:
> +#    This makes harder tracking and fixing host machine path leaks.
> +# 3- Output location (i.e. $(O)) not being the absolute canonical path:
> +#    This makes harder tracking and fixing host machine path leaks.
> +#
> +# Note:
> +# - remove the trailing '/.' from $(O) as it can be added by the makefile
> +#   wrapper installed in the $(O).
> +ifneq ($(shell umask):$(CURDIR):$(patsubst %/.,%,$(O)),$(UMASK):$(realpath $(CURDIR)):$(realpath $(O)))

Can we simplify this a bit with an intermediate "NEED_RECURSE" or
something like that ?

# umask is too restrictive
ifneq ($(shell umask),$(UMASK))
NEED_RECURSE = YES
endif

...

ifeq ($(NEED_RECURSE),YES)

Maybe "RECURSE" is not the appropriate word. Maybe "NEED_SUBMAKE" is
better. Or other folks might have better suggestions.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com