[Buildroot] [PATCH v8 00/10] Make the SDK relocatable

[Buildroot] [PATCH v8 00/10] Make the SDK relocatable

[Wolfgang Grandegger]

Hello,

this is v8 of my patch series to make the buildroot SDK (HOST_DIR)
relocatable. It sanitizes the RPATH of all ELF files in the "target",
"staging" and "host" tree using "patchelf --make-rpath-relative". We
now use patchelf v0.9 to still support old Debian and RHEL systems.

v5 did RPATH sanitization per package after package installation into
the host, staging or target tree using GLOBAL_INSTRUMENTATION_HOOKS.
This approach got more and more complex and inefficient. Therefore
it was abandoned in favor of global sanititation at the end of the
host, staging and target build (see changes since v5).

Furthermore this patch creates the script "relocate-sdk.sh" in the top
directory of the "host" tree allowing to relocate the SDK after it has
been moved to a new location. It replaces the old path with the new
one in all text files identified by "file --mime-type". The location
is stored in "share/buildroot/sdk-location". This patch is already
upstream.

Unfortunately, "qmake" uses hard-coded pathes compiled into the QT5
libraries. To overcome this problem, "qt5pase" now creates "qt.conf".

v8 now creates a relocatable SDK with "make sdk". The normal "make"
only sanitizes the RPATH in the target tree.

Hope I have not too much pending issues.

Wolfgang.

Changes since v7:
- patchelf patches are now created with git (base on v0.9)
- the target tree is now sanitized before the overlay files are copied
- "make sdk" sanitizes the staging and host tree and adds relocate.sh
- relocate.sh does not use basism an longer
- fix-rpath now uses absolute path without rootdir for staging and
  target dir
- various other minor fixes (doc, indention, etc.)

Changes since v6:
- patchelf patches: update patch header with upstream commit, etc.
- ${HOST_DIR}/usr is gone which required various fixes
- Proper qt.conf.in added

Changes since v5:
- switch back to v4
- patchelf patches are now based on v0.9
- patchelf now calculates minimal relative path to the ELF file
  (and not to the rootdir as before).
- patchelf: neededLibFound is now passed to libFoundInRPath
- Makefile: the "host-finalize" target has been added for SDK relocation
- fix-rpath now uses variables to define the list of dirs to be pruned
- fix-rpath: the staging tree is sanitized like the target tree
- various minor fixes (typos)

Changes since v4:

- RPATH sanitation is now done per package and installation step
  in "pkg-generic.mk".
- After the installation step, the list of the installed files is
  stored in "<package-build-dir>/.br_[host|staging|target]_filelist".
- The GLOBAL_INSTRUMENTATION_HOOKS "step_sanitize_rpath" then calls
  "fix-rpath" to do the sanitation.
- DEPENDENCIES_HOST_PREREQ += host-patchelf is set early in the
  Makefile as we need it for RPATH sanitation. As soon it's available
  sanitation can start (currently missing some packages).
- The patchelf "file busy" issue is now worked-around differently.

Changes since v3:

- The patchelf patch implementing " --make-rpath-relative" now supports
  the option "--relative-to-file" instructing to use "$ORIGIN" in
  RPATHs. Otherwise an absolute path relative to the root directory will
  be used.
- The staging tree is now sanitized as well using the options
  "--relative-to-file" and "--no-standard-libs".
- For the "target" tree, relative RPATHs do not use "$ORIGIN" any
  longer. An absolute path relative to the root directory is used
  instead.

Changes since v2:

- provide "qt.conf" to make "qmake" relocatable
- sed now uses the separator "\" to substitute the directory path.
  It's one of the few characters not allowed in file names. To
  avoid interpreting it as escape character, the "read -r" is used.
- The paranoia substituion check is done before doing the real
  substituion.

Changes since v1:

- The name SDK has been chosen for the relocatabed "HOST_DIR" (instead
  of toolchanin).
- The patchelf version bump and patching are now done by 2 patches
- No more helper functions are used in the Makefile to call "fix-rpath"
  but added directly.
- The staging tree is not touched any more... until we have a good
  reason to do so. 
- The sanitation is now performed by an optimized "fix-rpath" script.
- The relocate-sdk script is now copied for support/misc to the
  top directory of the host tree.

Samuel Martin (1):
  support/scripts: add fix-rpath script to sanitize the rpath

Wolfgang Grandegger (9):
  support/scripts: relocate-sdk.sh now uses a normal pipe to find
    strings
  package/patchelf: add patch for rpath sanitization under a root
    directory
  patchelf: always build this package required for RPATH sanitization
  core: sanitize RPATH in target tree before copying the overlay
  core: introduce "sdk" target to make a relocatable SDK
  core: sanitize RPATH in staging tree to make a relocatable SDK
  external-toolchain: check if a buildroot SDK has already been
    relocated
  core: install relocation script and location to make a relocatable SDK
  package/qt5base: provide "qt.conf" to make "qmake" relocatable

 Makefile                                           |  13 +
 ...move-apparently-incorrect-usage-of-static.patch |  56 ++++
 ...unction-for-splitting-a-colon-separated-s.patch |  63 ++++
 ...to-make-the-rpath-relative-under-a-specif.patch | 332 +++++++++++++++++++++
 package/patchelf/Config.in.host                    |   3 +-
 package/qt5/qt5base/qt.conf.in                     |  19 ++
 package/qt5/qt5base/qt5base.mk                     |   8 +
 support/misc/relocate-sdk.sh                       |   4 +-
 support/scripts/fix-rpath                          | 133 +++++++++
 toolchain/helpers.mk                               |  16 +
 .../toolchain-external/pkg-toolchain-external.mk   |   1 +
 11 files changed, 645 insertions(+), 3 deletions(-)
 create mode 100644 package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
 create mode 100644 package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
 create mode 100644 package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch
 create mode 100644 package/qt5/qt5base/qt.conf.in
 create mode 100755 support/scripts/fix-rpath

-- 
2.7.4

[Buildroot] [PATCH v8 01/10] support/scripts: relocate-sdk.sh now uses a normal pipe to find strings

[Wolfgang Grandegger]

The normal shell does not support the bashism "< <(...)". Therefore
we use a normal pipe to find files containing a specific string.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 support/misc/relocate-sdk.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/support/misc/relocate-sdk.sh b/support/misc/relocate-sdk.sh
index 729353a..caabeaa 100755
--- a/support/misc/relocate-sdk.sh
+++ b/support/misc/relocate-sdk.sh
@@ -35,12 +35,12 @@ echo "Relocating the buildroot SDK from ${OLDPATH} to ${NEWPATH} ..."
 # Make sure file uses the right language
 export LC_ALL=C
 # Replace the old path with the new one in all text files
-while read -r FILE ; do
+grep -lr "${OLDPATH}" . | while read -r FILE ; do
     if file -b --mime-type "${FILE}" | grep -q '^text/' && [ "${FILE}" != "${LOCFILE}" ]
     then
         sed -i "s|${OLDPATH}|${NEWPATH}|g" "${FILE}"
     fi
-done < <(grep -lr "${OLDPATH}" .)
+done
 
 # At the very end, we update the location file to not break the
 # SDK if this script gets interruted.
-- 
2.7.4

[Buildroot] [PATCH v8 01/10] support/scripts: relocate-sdk.sh now uses a normal pipe to find strings

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 16:35:12 +0200, Wolfgang Grandegger wrote:
> The normal shell does not support the bashism "< <(...)". Therefore
> we use a normal pipe to find files containing a specific string.
> 
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> ---
>  support/misc/relocate-sdk.sh | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 02/10] package/patchelf: add patch for rpath sanitization under a root directory

[Wolfgang Grandegger]

The patch allows to use patchelf to sanitize the rpath of the buildroot
libraries and binaries using the option "--make-rpath-relative <rootdir>".
Recent versions of patchelf will not built on old Debian and RHEL systems
due to C++11 constructs. Therefore we stick with v0.9 for the time being.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 ...move-apparently-incorrect-usage-of-static.patch |  56 ++++
 ...unction-for-splitting-a-colon-separated-s.patch |  63 ++++
 ...to-make-the-rpath-relative-under-a-specif.patch | 332 +++++++++++++++++++++
 3 files changed, 451 insertions(+)
 create mode 100644 package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
 create mode 100644 package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
 create mode 100644 package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch

diff --git a/package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch b/package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
new file mode 100644
index 0000000..dc462ef
--- /dev/null
+++ b/package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
@@ -0,0 +1,56 @@
+From 2480efa8411523cf046094492192a5ee451aae5d Mon Sep 17 00:00:00 2001
+From: Eelco Dolstra <eelco.dolstra@logicblox.com>
+Date: Mon, 19 Sep 2016 17:31:37 +0200
+Subject: [PATCH] Remove apparently incorrect usage of "static"
+
+[Upstream-commit: https://github.com/NixOS/patchelf/commit/a365bcb7d7025da51b33165ef7ebc7180199a05e
+This patch also removes the DT_INIT symbols from needed_libs (DT_INIT
+points to library initialisation function, not to needed libraries...)]
+Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
+---
+ src/patchelf.cc | 8 +++-----
+ 1 file changed, 3 insertions(+), 5 deletions(-)
+
+diff --git a/src/patchelf.cc b/src/patchelf.cc
+index 136098f..c870638 100644
+--- a/src/patchelf.cc
++++ b/src/patchelf.cc
+@@ -941,7 +941,6 @@ void ElfFile<ElfFileParamNames>::modifySoname(sonameMode op, const string & newS
+     assert(strTabAddr == rdi(shdrDynStr.sh_addr));
+ 
+     /* Walk through the dynamic section, look for the DT_SONAME entry. */
+-    static vector<string> neededLibs;
+     dyn = (Elf_Dyn *) (contents + rdi(shdrDynamic.sh_offset));
+     Elf_Dyn * dynSoname = 0;
+     char * soname = 0;
+@@ -949,8 +948,7 @@ void ElfFile<ElfFileParamNames>::modifySoname(sonameMode op, const string & newS
+         if (rdi(dyn->d_tag) == DT_SONAME) {
+             dynSoname = dyn;
+             soname = strTab + rdi(dyn->d_un.d_val);
+-        } else if (rdi(dyn->d_tag) == DT_INIT)
+-            neededLibs.push_back(string(strTab + rdi(dyn->d_un.d_val)));
++        }
+     }
+ 
+     if (op == printSoname) {
+@@ -1058,7 +1056,7 @@ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op, string newRPath)
+        unless you use its `--enable-new-dtag' option, in which case it
+        generates a DT_RPATH and DT_RUNPATH pointing at the same
+        string. */
+-    static vector<string> neededLibs;
++    vector<string> neededLibs;
+     dyn = (Elf_Dyn *) (contents + rdi(shdrDynamic.sh_offset));
+     Elf_Dyn * dynRPath = 0, * dynRunPath = 0;
+     char * rpath = 0;
+@@ -1091,7 +1089,7 @@ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op, string newRPath)
+     /* For each directory in the RPATH, check if it contains any
+        needed library. */
+     if (op == rpShrink) {
+-        static vector<bool> neededLibFound(neededLibs.size(), false);
++        vector<bool> neededLibFound(neededLibs.size(), false);
+ 
+         newRPath = "";
+ 
+-- 
+1.9.1
+
diff --git a/package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch b/package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
new file mode 100644
index 0000000..330bea2
--- /dev/null
+++ b/package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
@@ -0,0 +1,63 @@
+From a8452dc7e80eb17572c7458e33a4f4d609e6a3da Mon Sep 17 00:00:00 2001
+From: Tuomas Tynkkynen <tuomas@tuxera.com>
+Date: Fri, 3 Jun 2016 23:03:51 +0300
+Subject: [PATCH] Extract a function for splitting a colon-separated string
+
+We're going to need this logic in another place, so make a function of
+this.
+
+[Upstream-commit: https://github.com/NixOS/patchelf/commit/2e3fdc2030c75c19df6fc2924083cfad53856562]
+Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
+---
+ src/patchelf.cc | 28 +++++++++++++++++++---------
+ 1 file changed, 19 insertions(+), 9 deletions(-)
+
+diff --git a/src/patchelf.cc b/src/patchelf.cc
+index c870638..1d9a772 100644
+--- a/src/patchelf.cc
++++ b/src/patchelf.cc
+@@ -57,6 +57,22 @@ unsigned char * contents = 0;
+ #define ElfFileParamNames Elf_Ehdr, Elf_Phdr, Elf_Shdr, Elf_Addr, Elf_Off, Elf_Dyn, Elf_Sym
+ 
+ 
++static vector<string> splitColonDelimitedString(const char * s){
++    vector<string> parts;
++    const char * pos = s;
++    while (*pos) {
++        const char * end = strchr(pos, ':');
++        if (!end) end = strchr(pos, 0);
++
++        parts.push_back(string(pos, end - pos));
++        if (*end == ':') ++end;
++        pos = end;
++    }
++
++    return parts;
++}
++
++
+ static unsigned int getPageSize(){
+     return pageSize;
+ }
+@@ -1093,15 +1109,9 @@ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op, string newRPath)
+ 
+         newRPath = "";
+ 
+-        char * pos = rpath;
+-        while (*pos) {
+-            char * end = strchr(pos, ':');
+-            if (!end) end = strchr(pos, 0);
+-
+-            /* Get the name of the directory. */
+-            string dirName(pos, end - pos);
+-            if (*end == ':') ++end;
+-            pos = end;
++        vector<string> rpathDirs = splitColonDelimitedString(rpath);
++        for (vector<string>::iterator it = rpathDirs.begin(); it != rpathDirs.end(); ++it) {
++            const string & dirName = *it;
+ 
+             /* Non-absolute entries are allowed (e.g., the special
+                "$ORIGIN" hack). */
+-- 
+1.9.1
+
diff --git a/package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch b/package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch
new file mode 100644
index 0000000..feec627
--- /dev/null
+++ b/package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch
@@ -0,0 +1,332 @@
+From 618220bfb55c875d6a4d197cb24fe632ac93ec85 Mon Sep 17 00:00:00 2001
+From: Wolfgang Grandegger <wg@grandegger.com>
+Date: Mon, 20 Feb 2017 16:29:24 +0100
+Subject: [PATCH] Add option to make the rpath relative under a specified root
+ directory
+
+Running "patchelf" with the option "--make-rpath-relative ROOTDIR" will
+modify or delete the RPATHDIRs according the following rules
+similar to Martin's patches [1] making the Buildroot toolchaing/SDK
+relocatable.
+
+RPATHDIR starts with "$ORIGIN":
+    The original build-system already took care of setting a relative
+    RPATH, resolve it and test if it's valid (does exist)
+
+RPATHDIR starts with ROOTDIR:
+    The original build-system added some absolute RPATH (absolute on
+    the build machine). Test if it's valid (does exist).
+
+ROOTDIR/RPATHDIR exists:
+    The original build-system already took care of setting an absolute
+    RPATH (absolute in the final rootfs), resolve it and test if it's
+    valid (does exist).
+
+RPATHDIR points somewhere else:
+    (can be anywhere: build trees, staging tree, host location,
+    non-existing location, etc.). Just discard such a path.
+
+The option "--no-standard-libs" will discard RPATHDIRs ROOTDIR/lib and
+ROOTDIR/usr/lib. Like "--shrink-rpath", RPATHDIRs are also discarded
+if the directories do not contain a library referenced by the
+DT_NEEDED fields.
+If the option "--relative-to-file" is given, the rpath will start
+with "$ORIGIN" making it relative to the ELF file, otherwise an
+absolute path relative to ROOTDIR will be used.
+
+A pull request for a similar patch [2] for mainline inclusion is
+pending.
+
+[1] http://lists.busybox.net/pipermail/buildroot/2016-April/159422.html
+[2] https://github.com/NixOS/patchelf/pull/118
+
+Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
+---
+ src/patchelf.cc | 196 ++++++++++++++++++++++++++++++++++++++++++++++++++------
+ 1 file changed, 175 insertions(+), 21 deletions(-)
+
+diff --git a/src/patchelf.cc b/src/patchelf.cc
+index 1d9a772..35b4a33 100644
+--- a/src/patchelf.cc
++++ b/src/patchelf.cc
+@@ -46,6 +46,10 @@ static bool debugMode = false;
+ 
+ static bool forceRPath = false;
+ 
++static bool noStandardLibDirs = false;
++
++static bool relativeToFile = false;
++
+ static string fileName;
+ static int pageSize = PAGESIZE;
+ 
+@@ -77,6 +81,49 @@ static unsigned int getPageSize(){
+     return pageSize;
+ }
+ 
++static bool absolutePathExists(const string & path, string & canonicalPath)
++{
++    char *cpath = realpath(path.c_str(), NULL);
++    if (cpath) {
++        canonicalPath = cpath;
++        free(cpath);
++        return true;
++    } else {
++        return false;
++    }
++}
++
++static string makePathRelative(const string & path,
++    const string & refPath)
++{
++    string relPath = "$ORIGIN";
++    string p = path, refP = refPath;
++    size_t pos;
++
++    /* Strip the common part of path and refPath */
++    while (true) {
++        pos = p.find_first_of('/', 1);
++        if (refP.find_first_of('/', 1) != pos)
++            break;
++        if (p.substr(0, pos) != refP.substr(0, pos))
++            break;
++        if (pos == string::npos)
++            break;
++        p = p.substr(pos);
++        refP = refP.substr(pos);
++    }
++    /* Check if both pathes are equal */
++    if (p != refP) {
++        pos = 0;
++        while (pos != string::npos) {
++            pos =refP.find_first_of('/', pos + 1);
++            relPath.append("/..");
++        }
++        relPath.append(p);
++    }
++
++    return relPath;
++}
+ 
+ template<ElfFileParams>
+ class ElfFile
+@@ -183,9 +230,13 @@ public:
+ 
+     void setInterpreter(const string & newInterpreter);
+ 
+-    typedef enum { rpPrint, rpShrink, rpSet, rpRemove } RPathOp;
++    typedef enum { rpPrint, rpShrink, rpMakeRelative, rpSet, rpRemove} RPathOp;
++
++    bool libFoundInRPath(const string & dirName,
++                         const vector<string> neededLibs,
++                         vector<bool> & neededLibFound);
+ 
+-    void modifyRPath(RPathOp op, string newRPath);
++    void modifyRPath(RPathOp op, string rootDir, string newRPath);
+ 
+     void addNeeded(set<string> libs);
+ 
+@@ -1041,7 +1092,27 @@ static void concatToRPath(string & rpath, const string & path)
+ 
+ 
+ template<ElfFileParams>
+-void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op, string newRPath)
++bool ElfFile<ElfFileParamNames>::libFoundInRPath(const string & dirName,
++    const vector<string> neededLibs, vector<bool> & neededLibFound)
++{
++    /* For each library that we haven't found yet, see if it
++       exists in this directory. */
++    bool libFound = false;
++    for (unsigned int j = 0; j < neededLibs.size(); ++j)
++        if (!neededLibFound[j]) {
++            string libName = dirName + "/" + neededLibs[j];
++            struct stat st;
++            if (stat(libName.c_str(), &st) == 0) {
++                neededLibFound[j] = true;
++                libFound = true;
++            }
++        }
++    return libFound;
++}
++
++
++template<ElfFileParams>
++void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op, string rootDir, string newRPath)
+ {
+     Elf_Shdr & shdrDynamic = findSection(".dynamic");
+ 
+@@ -1096,6 +1167,11 @@ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op, string newRPath)
+         return;
+     }
+ 
++    if (op == rpMakeRelative && !rpath) {
++        debug("no RPATH to make relative\n");
++        return;
++    }
++
+     if (op == rpShrink && !rpath) {
+         debug("no RPATH to shrink\n");
+         return;
+@@ -1120,26 +1196,86 @@ void ElfFile<ElfFileParamNames>::modifyRPath(RPathOp op, string newRPath)
+                 continue;
+             }
+ 
+-            /* For each library that we haven't found yet, see if it
+-               exists in this directory. */
+-            bool libFound = false;
+-            for (unsigned int j = 0; j < neededLibs.size(); ++j)
+-                if (!neededLibFound[j]) {
+-                    string libName = dirName + "/" + neededLibs[j];
+-                    struct stat st;
+-                    if (stat(libName.c_str(), &st) == 0) {
+-                        neededLibFound[j] = true;
+-                        libFound = true;
+-                    }
+-                }
+-
+-            if (!libFound)
++            if (!libFoundInRPath(dirName, neededLibs, neededLibFound))
+                 debug("removing directory `%s' from RPATH\n", dirName.c_str());
+             else
+                 concatToRPath(newRPath, dirName);
+         }
+     }
+ 
++    /* Make the the RPATH relative to the specified path */
++    if (op == rpMakeRelative) {
++        vector<bool> neededLibFound(neededLibs.size(), false);
++        string fileDir = fileName.substr(0, fileName.find_last_of("/"));
++
++        newRPath = "";
++
++        vector<string> rpathDirs = splitColonDelimitedString(rpath);
++        for (vector<string>::iterator it = rpathDirs.begin(); it != rpathDirs.end(); ++it) {
++            const string & dirName = *it;
++
++            string canonicalPath;
++
++            /* Figure out if we should keep or discard the path. There are several
++               cases to be handled:
++               "dirName" starts with "$ORIGIN":
++                   The original build-system already took care of setting a relative
++                   RPATH. Resolve it and test if it's valid (does exist).
++               "dirName" start with "rootDir":
++                   The original build-system added some absolute RPATH (absolute on
++                   the build machine). Test if it's valid (does exist).
++               "rootDir"/"dirName" exists:
++                    The original build-system already took care of setting an absolute
++                    RPATH (absolute in the final rootfs). Resolve it and test if it's
++                    valid (does exist).
++               "dirName" points somewhere else:
++                    (can be anywhere: build trees, staging tree, host location,
++                    non-existing location, etc.). Just discard such a path. */
++            if (!dirName.compare(0, 7, "$ORIGIN")) {
++                string path = fileDir + dirName.substr(7);
++                if (!absolutePathExists(path, canonicalPath)) {
++                    debug("removing directory '%s' from RPATH because '%s' doesn't exist\n",
++                          dirName.c_str(), path.c_str());
++                    continue;
++                }
++            } else if (!dirName.compare(0, rootDir.length(), rootDir)) {
++                if (!absolutePathExists(dirName, canonicalPath)) {
++                    debug("removing directory '%s' from RPATH because it doesn't exist\n", dirName.c_str());
++                    continue;
++                }
++            } else {
++                string path = rootDir + dirName;
++                if (!absolutePathExists(path, canonicalPath)) {
++                    debug("removing directory '%s' from RPATH because it's not in rootdir\n",
++                          dirName.c_str());
++                    continue;
++                }
++            }
++
++            if (noStandardLibDirs) {
++                if (!canonicalPath.compare(rootDir + "/lib") ||
++                    !canonicalPath.compare(rootDir + "/usr/lib")) {
++                    debug("removing directory '%s' from RPATH because it's a standard library directory\n",
++                         dirName.c_str());
++                    continue;
++                }
++            }
++
++            if (!libFoundInRPath(canonicalPath, neededLibs, neededLibFound)) {
++                debug("removing directory '%s' from RPATH because it does not contain needed libs\n",
++                      dirName.c_str());
++                continue;
++            }
++
++            /* Finally make "canonicalPath" relative to "filedir" in "rootDir" */
++            if (relativeToFile)
++                concatToRPath(newRPath, makePathRelative(canonicalPath, fileDir));
++            else
++                concatToRPath(newRPath, canonicalPath.substr(rootDir.length()));
++            debug("keeping relative path of %s\n", canonicalPath.c_str());
++        }
++    }
++
+     if (op == rpRemove) {
+         if (!rpath) {
+             debug("no RPATH to delete\n");
+@@ -1413,7 +1549,9 @@ static bool shrinkRPath = false;
+ static bool removeRPath = false;
+ static bool setRPath = false;
+ static bool printRPath = false;
++static bool makeRPathRelative = false;
+ static string newRPath;
++static string rootDir;
+ static set<string> neededLibsToRemove;
+ static map<string, string> neededLibsToReplace;
+ static set<string> neededLibsToAdd;
+@@ -1438,14 +1576,16 @@ static void patchElf2(ElfFile & elfFile)
+         elfFile.setInterpreter(newInterpreter);
+ 
+     if (printRPath)
+-        elfFile.modifyRPath(elfFile.rpPrint, "");
++        elfFile.modifyRPath(elfFile.rpPrint, "", "");
+ 
+     if (shrinkRPath)
+-        elfFile.modifyRPath(elfFile.rpShrink, "");
++        elfFile.modifyRPath(elfFile.rpShrink, "", "");
+     else if (removeRPath)
+-        elfFile.modifyRPath(elfFile.rpRemove, "");
++        elfFile.modifyRPath(elfFile.rpRemove, "", "");
+     else if (setRPath)
+-        elfFile.modifyRPath(elfFile.rpSet, newRPath);
++        elfFile.modifyRPath(elfFile.rpSet, "", newRPath);
++    else if (makeRPathRelative)
++        elfFile.modifyRPath(elfFile.rpMakeRelative, rootDir, "");
+ 
+     if (printNeeded) elfFile.printNeededLibs();
+ 
+@@ -1508,6 +1648,9 @@ void showHelp(const string & progName)
+   [--set-rpath RPATH]\n\
+   [--remove-rpath]\n\
+   [--shrink-rpath]\n\
++  [--make-rpath-relative ROOTDIR]\n\
++  [--no-standard-lib-dirs]\n\
++  [--relative-to-file]\n\
+   [--print-rpath]\n\
+   [--force-rpath]\n\
+   [--add-needed LIBRARY]\n\
+@@ -1564,6 +1707,17 @@ int main(int argc, char * * argv)
+             setRPath = true;
+             newRPath = argv[i];
+         }
++        else if (arg == "--make-rpath-relative") {
++            if (++i == argc) error("missing argument to --make-rpath-relative");
++            makeRPathRelative = true;
++            rootDir = argv[i];
++        }
++        else if (arg == "--no-standard-lib-dirs") {
++            noStandardLibDirs = true;
++        }
++        else if (arg == "--relative-to-file") {
++            relativeToFile = true;
++        }
+         else if (arg == "--print-rpath") {
+             printRPath = true;
+         }
+-- 
+1.9.1
+
-- 
2.7.4

[Buildroot] [PATCH v8 02/10] package/patchelf: add patch for rpath sanitization under a root directory

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 16:35:13 +0200, Wolfgang Grandegger wrote:
> The patch allows to use patchelf to sanitize the rpath of the buildroot
> libraries and binaries using the option "--make-rpath-relative <rootdir>".
> Recent versions of patchelf will not built on old Debian and RHEL systems
> due to C++11 constructs. Therefore we stick with v0.9 for the time being.
> 
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  ...move-apparently-incorrect-usage-of-static.patch |  56 ++++
>  ...unction-for-splitting-a-colon-separated-s.patch |  63 ++++
>  ...to-make-the-rpath-relative-under-a-specif.patch | 332 +++++++++++++++++++++
>  3 files changed, 451 insertions(+)
>  create mode 100644 package/patchelf/0001-Remove-apparently-incorrect-usage-of-static.patch
>  create mode 100644 package/patchelf/0002-Extract-a-function-for-splitting-a-colon-separated-s.patch
>  create mode 100644 package/patchelf/0003-Add-option-to-make-the-rpath-relative-under-a-specif.patch

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Wolfgang Grandegger]

From: Samuel Martin <s.martin49@gmail.com>

This commit introduces the script "fix-rpath" able to scan a tree,
detect ELF files, check their RPATH and fix it in a proper way.
The RPATH fixup is done by the patchelf utility using the option
"--make-rpath-relative <root-directory>".

Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 support/scripts/fix-rpath | 133 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)
 create mode 100755 support/scripts/fix-rpath

diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
new file mode 100755
index 0000000..3d5ee43
--- /dev/null
+++ b/support/scripts/fix-rpath
@@ -0,0 +1,133 @@
+#!/usr/bin/env bash
+
+# Copyright (C) 2016 Samuel Martin <s.martin49@gmail.com>
+# Copyright (C) 2017 Wolfgang Grandegger <wg@grandegger.com>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+usage() {
+  cat <<EOF >&2
+Usage:  ${0} TREE_KIND
+
+Description:
+
+    This script scans a tree and sanitize ELF files' RPATH found in there.
+
+    Sanitization behaves the same whatever the kind of the processed tree,
+    but the resulting RPATH differs. The rpath sanitization is done using
+    "patchelf --make-rpath-relazive".
+
+Arguments:
+
+    TREE_KIND    Kind of tree to be processed.
+                 Allowed values: host, target, staging
+
+Environment:
+
+    PATCHELF     patchelf program to use
+                 (default: HOST_DIR/bin/patchelf)
+
+    HOST_DIR     host directory
+    STAGING_DIR  staging directory
+    TARGET_DIR   target directory
+
+    TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR
+                 (default HOST_DIR/opt/ext-toolchain)              
+EOF
+}
+
+: ${PATCHELF:=${HOST_DIR}/bin/patchelf}
+
+# ELF files should not be in these sub-directories
+HOST_EXCLUDEPATHS="/share/terminfo"
+STAGING_EXCLUDEPATHS="/usr/include /usr/share/terminfo"
+
+main() {
+    local rootdir
+    local tree="${1}"
+    local find_args=( )
+    local sanitize_extra_args=( )
+
+    case "${tree}" in
+        host)
+            rootdir="${HOST_DIR}"
+
+            # do not process the sysroot (only contains target binaries)
+            find_args+=( "-path" "${STAGING_DIR}" "-prune" "-o" )
+
+            # do not process the external toolchain installation directory to
+            # avoid breaking it.
+            test "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" != "" && \
+                find_args+=( "-path" "${TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR}" "-prune" "-o" )
+
+            for excludepath in ${HOST_EXCLUDEPATHS}; do
+                find_args+=( "-path" "${HOST_DIR}""${excludepath}" "-prune" "-o" );
+            done
+
+            # do not process the patchelf binary but a copy to work-around "file in use"
+            find_args+=( "-path" "${PATCHELF}" "-prune" "-o" )
+            cp "${PATCHELF}" "${PATCHELF}.__to_be_patched"
+
+            # we always want $ORIGIN-based rpaths to make it relocatable.
+            sanitize_extra_args+=( "--relative-to-file" )
+            ;;
+
+        staging)
+            rootdir="${STAGING_DIR}"
+
+            # ELF files should not be in these sub-directories
+            for excludepath in ${STAGING_EXCLUDEPATHS}; do
+                find_args+=( "-path" "${STAGING_DIR}""${excludepath}" "-prune" "-o" )
+            done
+
+            # should be like for the target tree below
+            sanitize_extra_args+=( "--no-standard-lib-dirs" )
+            ;;
+
+        target)
+            rootdir="${TARGET_DIR}"
+            # we don't want $ORIGIN-based rpaths but absolute paths without rootdir.
+            # we also want to remove rpaths pointing to /lib or /usr/lib. 
+            sanitize_extra_args+=( "--no-standard-lib-dirs" )
+            ;;
+
+        *)
+            usage
+            exit 1
+            ;;
+    esac
+
+    find_args+=( "-type" "f" "-print" )
+
+    while read file ; do
+        # check if it's an ELF file
+        if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
+            # make files writable if necessary
+            changed=$(chmod -c u+w "${file}")
+            # call patchelf to sanitize the rpath
+            ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}"
+            # restore the original permission
+            test "${changed}" != "" && chmod u-w "${file}"
+        fi
+    done < <(find "${rootdir}" ${find_args[@]})
+
+    # Restore patched patchelf utility
+    test "${tree}" = "host" && mv "${PATCHELF}.__to_be_patched" "${PATCHELF}"
+
+    # ignore errors
+    return 0
+}
+
+main ${@}
-- 
2.7.4

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 16:35:14 +0200, Wolfgang Grandegger wrote:
> From: Samuel Martin <s.martin49@gmail.com>
> 
> This commit introduces the script "fix-rpath" able to scan a tree,
> detect ELF files, check their RPATH and fix it in a proper way.
> The RPATH fixup is done by the patchelf utility using the option
> "--make-rpath-relative <root-directory>".
> 
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  support/scripts/fix-rpath | 133 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 133 insertions(+)
>  create mode 100755 support/scripts/fix-rpath

I've applied to master, after removing trailing slashes.

However, I noticed that when patchelf is not available in HOST_DIR,
this script does nothing, but doesn't show any error:

thomas at windsurf:~/projets/buildroot (master)$ rm output/host/usr/bin/patchelf 
thomas at windsurf:~/projets/buildroot (master)$ make
[...]
>>>   Sanitizing RPATH in target tree
/home/thomas/projets/buildroot/support/scripts/fix-rpath target
thomas at windsurf:~/projets/buildroot (master)$

This is a little bit disturbing, so I believe it should be fixed in a
follow-up patch.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Arnout Vandecappelle]

On 20-07-17 22:49, Thomas Petazzoni wrote:
> Hello,
> 
> On Thu, 20 Jul 2017 16:35:14 +0200, Wolfgang Grandegger wrote:
>> From: Samuel Martin <s.martin49@gmail.com>
>>
>> This commit introduces the script "fix-rpath" able to scan a tree,
>> detect ELF files, check their RPATH and fix it in a proper way.
>> The RPATH fixup is done by the patchelf utility using the option
>> "--make-rpath-relative <root-directory>".
>>
>> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
>> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>> ---
>>  support/scripts/fix-rpath | 133 ++++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 133 insertions(+)
>>  create mode 100755 support/scripts/fix-rpath
> 
> I've applied to master, after removing trailing slashes.

 Which trailing slashes?


> However, I noticed that when patchelf is not available in HOST_DIR,
> this script does nothing, but doesn't show any error:
> 
> thomas at windsurf:~/projets/buildroot (master)$ rm output/host/usr/bin/patchelf 
> thomas at windsurf:~/projets/buildroot (master)$ make
> [...]
>>>>   Sanitizing RPATH in target tree
> /home/thomas/projets/buildroot/support/scripts/fix-rpath target
> thomas at windsurf:~/projets/buildroot (master)$
> 
> This is a little bit disturbing, so I believe it should be fixed in a
> follow-up patch.

 Right! That's because the "if ${PATCHELF} ..." just fails and the error is
redirected to /dev/null.

 Better check if ${PATCHELF} is executable somewhere in the beginning.

 However, there is never a guarantee. If e.g. it would be missing a shared
library, it would still fail in the same way. To be really sure, you'd have do
e.g. call patchelf on itself to verify it is actually working.

 Regards,
 Arnout



-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 23:05:59 +0200, Arnout Vandecappelle wrote:

> >>  support/scripts/fix-rpath | 133 ++++++++++++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 133 insertions(+)
> >>  create mode 100755 support/scripts/fix-rpath  
> > 
> > I've applied to master, after removing trailing slashes.  
> 
>  Which trailing slashes?

Gaah, I wanted to say "trailing spaces".

> > This is a little bit disturbing, so I believe it should be fixed in a
> > follow-up patch.  
> 
>  Right! That's because the "if ${PATCHELF} ..." just fails and the error is
> redirected to /dev/null.
> 
>  Better check if ${PATCHELF} is executable somewhere in the beginning.
> 
>  However, there is never a guarantee. If e.g. it would be missing a shared
> library, it would still fail in the same way. To be really sure, you'd have do
> e.g. call patchelf on itself to verify it is actually working.

Perhaps patchelf --help is sufficient?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Wolfgang Grandegger]

Am 20.07.2017 um 23:05 schrieb Arnout Vandecappelle:
> 
> 
> On 20-07-17 22:49, Thomas Petazzoni wrote:
>> Hello,
>>
>> On Thu, 20 Jul 2017 16:35:14 +0200, Wolfgang Grandegger wrote:
>>> From: Samuel Martin <s.martin49@gmail.com>
>>>
>>> This commit introduces the script "fix-rpath" able to scan a tree,
>>> detect ELF files, check their RPATH and fix it in a proper way.
>>> The RPATH fixup is done by the patchelf utility using the option
>>> "--make-rpath-relative <root-directory>".
>>>
>>> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
>>> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
>>> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>>> ---
>>>   support/scripts/fix-rpath | 133 ++++++++++++++++++++++++++++++++++++++++++++++
>>>   1 file changed, 133 insertions(+)
>>>   create mode 100755 support/scripts/fix-rpath
>>
>> I've applied to master, after removing trailing slashes.
> 
>   Which trailing slashes?
> 
> 
>> However, I noticed that when patchelf is not available in HOST_DIR,
>> this script does nothing, but doesn't show any error:
>>
>> thomas at windsurf:~/projets/buildroot (master)$ rm output/host/usr/bin/patchelf
>> thomas at windsurf:~/projets/buildroot (master)$ make
>> [...]
>>>>>    Sanitizing RPATH in target tree
>> /home/thomas/projets/buildroot/support/scripts/fix-rpath target
>> thomas at windsurf:~/projets/buildroot (master)$
>>
>> This is a little bit disturbing, so I believe it should be fixed in a
>> follow-up patch.
> 
>   Right! That's because the "if ${PATCHELF} ..." just fails and the error is
> redirected to /dev/null.
> 
>   Better check if ${PATCHELF} is executable somewhere in the beginning.
> 
>   However, there is never a guarantee. If e.g. it would be missing a shared
> library, it would still fail in the same way. To be really sure, you'd have do
> e.g. call patchelf on itself to verify it is actually working.


Or you can set "PATCHELF_DEBUG=1" to check what patchelf is doing.

Wolfgang.

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Thomas Petazzoni]

Hello,

On Fri, 21 Jul 2017 09:35:14 +0200, Wolfgang Grandegger wrote:

> >>>>>    Sanitizing RPATH in target tree  
> >> /home/thomas/projets/buildroot/support/scripts/fix-rpath target
> >> thomas at windsurf:~/projets/buildroot (master)$
> >>
> >> This is a little bit disturbing, so I believe it should be fixed in a
> >> follow-up patch.  
> > 
> >   Right! That's because the "if ${PATCHELF} ..." just fails and the error is
> > redirected to /dev/null.
> > 
> >   Better check if ${PATCHELF} is executable somewhere in the beginning.
> > 
> >   However, there is never a guarantee. If e.g. it would be missing a shared
> > library, it would still fail in the same way. To be really sure, you'd have do
> > e.g. call patchelf on itself to verify it is actually working.  
> 
> Or you can set "PATCHELF_DEBUG=1" to check what patchelf is doing.

We're not interested in knowing what patchelf is doing. My initial
report is that if the patchelf binary doesn't exist, the script runs
without reporting any error:

 - Nothing on stdout/stderr
 - The script has an exit code of 0

This is not good: the script should at the very least return a non-zero
error code if patchelf isn't available, and should probably display an
error as well.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Wolfgang Grandegger]

Am 21.07.2017 um 09:40 schrieb Thomas Petazzoni:
> Hello,
> 
> On Fri, 21 Jul 2017 09:35:14 +0200, Wolfgang Grandegger wrote:
> 
>>>>>>>     Sanitizing RPATH in target tree
>>>> /home/thomas/projets/buildroot/support/scripts/fix-rpath target
>>>> thomas at windsurf:~/projets/buildroot (master)$
>>>>
>>>> This is a little bit disturbing, so I believe it should be fixed in a
>>>> follow-up patch.
>>>
>>>    Right! That's because the "if ${PATCHELF} ..." just fails and the error is
>>> redirected to /dev/null.
>>>
>>>    Better check if ${PATCHELF} is executable somewhere in the beginning.
>>>
>>>    However, there is never a guarantee. If e.g. it would be missing a shared
>>> library, it would still fail in the same way. To be really sure, you'd have do
>>> e.g. call patchelf on itself to verify it is actually working.
>>
>> Or you can set "PATCHELF_DEBUG=1" to check what patchelf is doing.
> 
> We're not interested in knowing what patchelf is doing. My initial
> report is that if the patchelf binary doesn't exist, the script runs
> without reporting any error:
> 
>   - Nothing on stdout/stderr
>   - The script has an exit code of 0
> 
> This is not good: the script should at the very least return a non-zero
> error code if patchelf isn't available, and should probably display an
> error as well.

Yes, I already got that point.

Wolfgang.

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Arnout Vandecappelle]

On 20-07-17 16:35, Wolfgang Grandegger wrote:
[snip]

 You forgot:
> +    This script scans a tree and sanitize ELF files' RPATH found in there.
                                            ^s
> +
> +    Sanitization behaves the same whatever the kind of the processed tree,
> +    but the resulting RPATH differs. The rpath sanitization is done using
> +    "patchelf --make-rpath-relazive".
                                  ^t

[snip]
> +            for excludepath in ${HOST_EXCLUDEPATHS}; do
> +                find_args+=( "-path" "${HOST_DIR}""${excludepath}" "-prune" "-o" );

 Unneeded ;

 Regards,
 Arnout

[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v8 03/10] support/scripts: add fix-rpath script to sanitize the rpath

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 23:03:34 +0200, Arnout Vandecappelle wrote:
> On 20-07-17 16:35, Wolfgang Grandegger wrote:
> [snip]
> 
>  You forgot:
> > +    This script scans a tree and sanitize ELF files' RPATH found in there.  
>                                             ^s
> > +
> > +    Sanitization behaves the same whatever the kind of the processed tree,
> > +    but the resulting RPATH differs. The rpath sanitization is done using
> > +    "patchelf --make-rpath-relazive".  
>                                   ^t
> 
> [snip]
> > +            for excludepath in ${HOST_EXCLUDEPATHS}; do
> > +                find_args+=( "-path" "${HOST_DIR}""${excludepath}" "-prune" "-o" );  
> 
>  Unneeded ;

Please send follow-up patches. Thanks!

I've applied 1 to 5 and 10 tonight. I won't apply 6 to 9 today, so
there's more time to review/adjust them if needed.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 04/10] patchelf: always build this package required for RPATH sanitization

[Wolfgang Grandegger]

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 package/patchelf/Config.in.host | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/patchelf/Config.in.host b/package/patchelf/Config.in.host
index d1c8375..7b19e6a 100644
--- a/package/patchelf/Config.in.host
+++ b/package/patchelf/Config.in.host
@@ -1,5 +1,6 @@
 config BR2_PACKAGE_HOST_PATCHELF
-	bool "host patchelf"
+	bool
+	default "y"
 	help
 	  PatchELF is a small utility to modify the dynamic linker
 	  and RPATH of ELF executables.
-- 
2.7.4

[Buildroot] [PATCH v8 04/10] patchelf: always build this package required for RPATH sanitization

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 16:35:15 +0200, Wolfgang Grandegger wrote:
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> ---
>  package/patchelf/Config.in.host | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/package/patchelf/Config.in.host b/package/patchelf/Config.in.host
> index d1c8375..7b19e6a 100644
> --- a/package/patchelf/Config.in.host
> +++ b/package/patchelf/Config.in.host
> @@ -1,5 +1,6 @@
>  config BR2_PACKAGE_HOST_PATCHELF
> -	bool "host patchelf"
> +	bool
> +	default "y"

Double quotes are not needed, so I dropped them and applied.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 05/10] core: sanitize RPATH in target tree before copying the overlay

[Wolfgang Grandegger]

We sanitize the RPATH of ELF files in the target tree to deal
with stupid packages that don't correctly use --prefix/DESTDIR
and that end up putting the full absolute build-time directory
in the RPATH.

We do it before copying the overlay and calling the post-build
script. The user is completely responsible for what happens
in the last two steps, and it should never be touched by us.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Makefile b/Makefile
index 188ce9a..d4faa02 100644
--- a/Makefile
+++ b/Makefile
@@ -709,6 +709,9 @@ endif
 		echo "PRETTY_NAME=\"Buildroot $(BR2_VERSION)\"" \
 	) >  $(TARGET_DIR)/etc/os-release
 
+	@$(call MESSAGE,"Sanitizing RPATH in target tree")
+	$(TOPDIR)/support/scripts/fix-rpath target
+
 	@$(foreach d, $(call qstrip,$(BR2_ROOTFS_OVERLAY)), \
 		$(call MESSAGE,"Copying overlay $(d)"); \
 		rsync -a --ignore-times --keep-dirlinks $(RSYNC_VCS_EXCLUSIONS) \
-- 
2.7.4

[Buildroot] [PATCH v8 05/10] core: sanitize RPATH in target tree before copying the overlay

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 16:35:16 +0200, Wolfgang Grandegger wrote:
> We sanitize the RPATH of ELF files in the target tree to deal
> with stupid packages that don't correctly use --prefix/DESTDIR
> and that end up putting the full absolute build-time directory
> in the RPATH.
> 
> We do it before copying the overlay and calling the post-build
> script. The user is completely responsible for what happens
> in the last two steps, and it should never be touched by us.
> 
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> ---
>  Makefile | 3 +++
>  1 file changed, 3 insertions(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 06/10] core: introduce "sdk" target to make a relocatable SDK

[Wolfgang Grandegger]

We use a separate make target to build a relocatable SDK.
We first sanitize the RPATH in host tree depending on
the toolchain.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Makefile b/Makefile
index d4faa02..3019081 100644
--- a/Makefile
+++ b/Makefile
@@ -552,6 +552,11 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 .PHONY: world
 world: target-post-image
 
+.PHONY: sdk
+sdk: toolchain
+	@$(call MESSAGE,"Rendering the SDK relocatable")
+	$(TOPDIR)/support/scripts/fix-rpath host
+
 # Compatibility symlink in case a post-build script still uses $(HOST_DIR)/usr
 $(HOST_DIR)/usr: $(HOST_DIR)
 	@ln -snf . $@
@@ -1001,6 +1006,7 @@ help:
 	@echo 'Build:'
 	@echo '  all                    - make world'
 	@echo '  toolchain              - build toolchain'
+	@echo '  sdk                    - build relocatable SDK'
 	@echo
 	@echo 'Configuration:'
 	@echo '  menuconfig             - interactive curses-based configurator'
-- 
2.7.4

[Buildroot] [PATCH v8 06/10] core: introduce "sdk" target to make a relocatable SDK

[Arnout Vandecappelle]

On 20-07-17 16:35, Wolfgang Grandegger wrote:
> We use a separate make target to build a relocatable SDK.
> We first sanitize the RPATH in host tree depending on
> the toolchain.
> 
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> ---
>  Makefile | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index d4faa02..3019081 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -552,6 +552,11 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
>  .PHONY: world
>  world: target-post-image
>  
> +.PHONY: sdk
> +sdk: toolchain

 As mentioned by Thomas, this should depend on "world", not just toolchain.
Indeed, plenty of host packages are built *after* the toolchain. You could add
${PACKAGES} as a dependency, but that still misses the dependencies of the
rootfs targets that don't have a Kconfig option. So depending on "world" is the
easiest.

 With that (and updating the commit message accordingly):
 Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout


> +	@$(call MESSAGE,"Rendering the SDK relocatable")
> +	$(TOPDIR)/support/scripts/fix-rpath host
> +
>  # Compatibility symlink in case a post-build script still uses $(HOST_DIR)/usr
>  $(HOST_DIR)/usr: $(HOST_DIR)
>  	@ln -snf . $@
> @@ -1001,6 +1006,7 @@ help:
>  	@echo 'Build:'
>  	@echo '  all                    - make world'
>  	@echo '  toolchain              - build toolchain'
> +	@echo '  sdk                    - build relocatable SDK'
>  	@echo
>  	@echo 'Configuration:'
>  	@echo '  menuconfig             - interactive curses-based configurator'
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v8 06/10] core: introduce "sdk" target to make a relocatable SDK

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 23:11:34 +0200, Arnout Vandecappelle wrote:

>  As mentioned by Thomas, this should depend on "world", not just toolchain.
> Indeed, plenty of host packages are built *after* the toolchain. You could add
> ${PACKAGES} as a dependency, but that still misses the dependencies of the
> rootfs targets that don't have a Kconfig option. So depending on "world" is the
> easiest.
> 
>  With that (and updating the commit message accordingly):
>  Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Wolfgang added the dependency on $(PACKAGES) in PATCH 07/10. But
actually, I think PATCH 06/10 and 07/10 should probably be squashed
together, and indeed have sdk depend on world.

In fact even PATCH 09/10 should be squashed with 06/10 and 07/10. Just
one patch adding the "sdk" target is enough, it's just a few lines
overall.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 07/10] core: sanitize RPATH in staging tree to make a relocatable SDK

[Wolfgang Grandegger]

When calling "make sdk" we also sanitize the staging tree. Therefore
"sdk" must depend on all packages.

Sanitizing staging is not really needed, in the sense that any rpath
in there is simply not going to be used. We want to sanitize staging
for the following reasons:

- To avoid leaking references to the original output directory. This
  way, we can validate that the SDK is relocatable by running a simple
  "grep -r ${BASE_DIR} ${HOST_DIR}". Obviously RPATH sanitization is
  not sufficient (e.g. also the references to source files have to be
  stripped), but it's a step in the right direction. This reason is
  obviously only relevant for the SDK.

- To make sure that when an executable is copied to target that it
  actually executes correctly. Since within Buildroot we never copy
  stuff from staging to target, this is clearly only relevant for
  the SDK.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 3019081..c54c420 100644
--- a/Makefile
+++ b/Makefile
@@ -553,9 +553,11 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: target-post-image
 
 .PHONY: sdk
-sdk: toolchain
+sdk: toolchain $(PACKAGES)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
+	@$(call MESSAGE,"Sanitizing RPATH in staging tree")
+	$(TOPDIR)/support/scripts/fix-rpath staging
 
 # Compatibility symlink in case a post-build script still uses $(HOST_DIR)/usr
 $(HOST_DIR)/usr: $(HOST_DIR)
-- 
2.7.4

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Wolfgang Grandegger]

The location of the buildroot SDK is stored in the file "sdk-location"
in "share/buildroot". If it's content does not match the current
SDK location, ask the user to run the script "relocate-sdk.sh" in the
top directory once. The external toolchain may be a pre-installed one
in a directory that is not writeable by us. Therefore, we can't run
the script directly.

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 toolchain/helpers.mk                                   | 16 ++++++++++++++++
 toolchain/toolchain-external/pkg-toolchain-external.mk |  1 +
 2 files changed, 17 insertions(+)

diff --git a/toolchain/helpers.mk b/toolchain/helpers.mk
index e9e36d2..96fc4c8 100644
--- a/toolchain/helpers.mk
+++ b/toolchain/helpers.mk
@@ -484,3 +484,19 @@ define simplify_symlink
 	ln -sf "$${DOTS}$${REL_DEST}" "$${FULL_SRC}" ; \
 )
 endef
+
+#
+# Check if it's a buildroot toolchain and if it's already relocatable by
+# reading and testing the toolchain location file
+#
+# $1: toolchain installation directory
+#
+define check_buildroot_sdk_relocated
+	if [ -r $(1)/share/buildroot/sdk-location ]; then \
+		sdkroot="$(1)"; \
+		if [ "`cat $(1)/share/buildroot/sdk-location`" != "$${sdkroot}" ]; then \
+			echo "Please relocate the buildroot SDK by executing \"$${sdkroot}/relocate-sdk.sh\" once!" ; \
+			exit 1 ; \
+		fi \
+	fi
+endef
diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index 3263199..9762e81 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -529,6 +529,7 @@ endif
 # matches the configuration provided in Buildroot: ABI, C++ support,
 # kernel headers version, type of C library and all C library features.
 define $(2)_CONFIGURE_CMDS
+	$$(Q)$$(call check_buildroot_sdk_relocated,$$(TOOLCHAIN_EXTERNAL_INSTALL_DIR))
 	$$(Q)$$(call check_cross_compiler_exists,$$(TOOLCHAIN_EXTERNAL_CC))
 	$$(Q)$$(call check_unusable_toolchain,$$(TOOLCHAIN_EXTERNAL_CC))
 	$$(Q)SYSROOT_DIR="$$(call toolchain_find_sysroot,$$(TOOLCHAIN_EXTERNAL_CC))" ; \
-- 
2.7.4

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 16:35:19 +0200, Wolfgang Grandegger wrote:
> The location of the buildroot SDK is stored in the file "sdk-location"
> in "share/buildroot". If it's content does not match the current
> SDK location, ask the user to run the script "relocate-sdk.sh" in the
> top directory once. The external toolchain may be a pre-installed one
> in a directory that is not writeable by us. Therefore, we can't run
> the script directly.
> 
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> ---
>  toolchain/helpers.mk                                   | 16 ++++++++++++++++
>  toolchain/toolchain-external/pkg-toolchain-external.mk |  1 +
>  2 files changed, 17 insertions(+)

Now that I think of it, this is going to be a bit annoying. This means
that whenever you want to use a Buildroot-generated toolchain as an
external toolchain in another Buildroot configuration, you have to run
this relocate script.

This will break a lot of things, like the autobuilders, the runtime
test infrastructure, etc. that all rely on pre-built Buildroot
toolchains, but don't run the relocate-sdk script. In addition, we
simply can't do it, because it's directly Buildroot itself that
downloads the toolchain tarball and extracts it, so we don't even get
the opportunity to run the relocate-sdk script.

Therefore, I cannot apply this as-is, a better solution needs to be
found.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Arnout Vandecappelle]

On 20-07-17 23:21, Thomas Petazzoni wrote:
> Hello,
> 
> On Thu, 20 Jul 2017 16:35:19 +0200, Wolfgang Grandegger wrote:
>> The location of the buildroot SDK is stored in the file "sdk-location"
>> in "share/buildroot". If it's content does not match the current
>> SDK location, ask the user to run the script "relocate-sdk.sh" in the
>> top directory once. The external toolchain may be a pre-installed one
>> in a directory that is not writeable by us. Therefore, we can't run
>> the script directly.
>>
>> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
>> ---
>>  toolchain/helpers.mk                                   | 16 ++++++++++++++++
>>  toolchain/toolchain-external/pkg-toolchain-external.mk |  1 +
>>  2 files changed, 17 insertions(+)
> 
> Now that I think of it, this is going to be a bit annoying. This means
> that whenever you want to use a Buildroot-generated toolchain as an
> external toolchain in another Buildroot configuration, you have to run
> this relocate script.
> 
> This will break a lot of things, like the autobuilders, the runtime
> test infrastructure, etc. that all rely on pre-built Buildroot
> toolchains, but don't run the relocate-sdk script. In addition, we
> simply can't do it, because it's directly Buildroot itself that
> downloads the toolchain tarball and extracts it, so we don't even get
> the opportunity to run the relocate-sdk script.

 Well, we could run the script directly for downloaded toolchains, and do the
check for pre-installed toolchains. Does that sound acceptable?

 I'm probably forgetting something though :-)

 Regards,
 Arnout

> 
> Therefore, I cannot apply this as-is, a better solution needs to be
> found.
> 
> Best regards,
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Thomas Petazzoni]

Hello,

On Fri, 21 Jul 2017 00:30:52 +0200, Arnout Vandecappelle wrote:

> > Now that I think of it, this is going to be a bit annoying. This means
> > that whenever you want to use a Buildroot-generated toolchain as an
> > external toolchain in another Buildroot configuration, you have to run
> > this relocate script.
> > 
> > This will break a lot of things, like the autobuilders, the runtime
> > test infrastructure, etc. that all rely on pre-built Buildroot
> > toolchains, but don't run the relocate-sdk script. In addition, we
> > simply can't do it, because it's directly Buildroot itself that
> > downloads the toolchain tarball and extracts it, so we don't even get
> > the opportunity to run the relocate-sdk script.  
> 
>  Well, we could run the script directly for downloaded toolchains, and do the
> check for pre-installed toolchains. Does that sound acceptable?
> 
>  I'm probably forgetting something though :-)

In fact, I wonder why this check is necessary at all, for the use case
of an external toolchain. Indeed, an external toolchain is really a
very minimal SDK, with just gcc/binutils/gdb/libc. And there is nothing
that relocate-sdk will relocate in such a simple SDK: SDKs that are
just toolchains are relocatable by design, with no fixup needed.

The relocate-sdk is useful when you start having more stuff in the SDK,
with libtool files and other weird things that hardcore paths. But
that's not the case for the SDK-is-just-a-toolchain case.

Therefore, I would suggest to simply drop this check.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Wolfgang Grandegger]

Am 21.07.2017 um 08:36 schrieb Thomas Petazzoni:
> Hello,
> 
> On Fri, 21 Jul 2017 00:30:52 +0200, Arnout Vandecappelle wrote:
> 
>>> Now that I think of it, this is going to be a bit annoying. This means
>>> that whenever you want to use a Buildroot-generated toolchain as an
>>> external toolchain in another Buildroot configuration, you have to run
>>> this relocate script.
>>>
>>> This will break a lot of things, like the autobuilders, the runtime
>>> test infrastructure, etc. that all rely on pre-built Buildroot
>>> toolchains, but don't run the relocate-sdk script. In addition, we
>>> simply can't do it, because it's directly Buildroot itself that
>>> downloads the toolchain tarball and extracts it, so we don't even get
>>> the opportunity to run the relocate-sdk script.
>>
>>   Well, we could run the script directly for downloaded toolchains, and do the
>> check for pre-installed toolchains. Does that sound acceptable?
>>
>>   I'm probably forgetting something though :-)
> 
> In fact, I wonder why this check is necessary at all, for the use case
> of an external toolchain. Indeed, an external toolchain is really a
> very minimal SDK, with just gcc/binutils/gdb/libc. And there is nothing
> that relocate-sdk will relocate in such a simple SDK: SDKs that are
> just toolchains are relocatable by design, with no fixup needed.
> 
> The relocate-sdk is useful when you start having more stuff in the SDK,
> with libtool files and other weird things that hardcore paths. But
> that's not the case for the SDK-is-just-a-toolchain case.

Yes.

> Therefore, I would suggest to simply drop this check.

If you make a toolchain with "make toolchain" the check will not harm. 
If it's created with "make sdk" it will do the check because
"share/buildroot/sdk-location" does exist. "make sdk" will never create 
just the toolchain.

Wolfgang.

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Thomas Petazzoni]

Hello,

On Fri, 21 Jul 2017 08:52:56 +0200, Wolfgang Grandegger wrote:

> > Therefore, I would suggest to simply drop this check.  
> 
> If you make a toolchain with "make toolchain" the check will not harm. 
> If it's created with "make sdk" it will do the check because
> "share/buildroot/sdk-location" does exist. "make sdk" will never create 
> just the toolchain.

"make sdk" will definitely create just the toolchain if what you have
in your configuration is... just the toolchain, which is exactly how I
build Buildroot toolchains, such as the ones used by the autobuilders
or on toolchains.free-electrons.com.

Doing "make toolchain" doesn't work for this use case, because you
definitely want the RPATH sanitization of host binaries done in "make
sdk".

Again, I think this additional check in the external toolchain code is
useless. For a SDK to be usable as an external toolchain, it has to be
a SDK that contains *just* a toolchain and *nothing* else. If its
sysroot is polluted with additional libraries than the C library, it
cannot be used as an external toolchain. And therefore, the
relocate-sdk procedure is useless, because there's nothing to relocate
in an SDK that is just a toolchain.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Wolfgang Grandegger]

Am 21.07.2017 um 09:24 schrieb Thomas Petazzoni:
> Hello,
> 
> On Fri, 21 Jul 2017 08:52:56 +0200, Wolfgang Grandegger wrote:
> 
>>> Therefore, I would suggest to simply drop this check.
>>
>> If you make a toolchain with "make toolchain" the check will not harm.
>> If it's created with "make sdk" it will do the check because
>> "share/buildroot/sdk-location" does exist. "make sdk" will never create
>> just the toolchain.
> 
> "make sdk" will definitely create just the toolchain if what you have
> in your configuration is... just the toolchain, which is exactly how I
> build Buildroot toolchains, such as the ones used by the autobuilders
> or on toolchains.free-electrons.com.
> 
> Doing "make toolchain" doesn't work for this use case, because you
> definitely want the RPATH sanitization of host binaries done in "make
> sdk".
> 
> Again, I think this additional check in the external toolchain code is
> useless. For a SDK to be usable as an external toolchain, it has to be
> a SDK that contains *just* a toolchain and *nothing* else. If its
> sysroot is polluted with additional libraries than the C library, it
> cannot be used as an external toolchain. And therefore, the
> relocate-sdk procedure is useless, because there's nothing to relocate
> in an SDK that is just a toolchain.

OK. I see. Then I will drop this patch.

Wolfgang.

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Thomas Petazzoni]

Hello,

On Fri, 21 Jul 2017 09:44:00 +0200, Wolfgang Grandegger wrote:

> > "make sdk" will definitely create just the toolchain if what you have
> > in your configuration is... just the toolchain, which is exactly how I
> > build Buildroot toolchains, such as the ones used by the autobuilders
> > or on toolchains.free-electrons.com.
> > 
> > Doing "make toolchain" doesn't work for this use case, because you
> > definitely want the RPATH sanitization of host binaries done in "make
> > sdk".
> > 
> > Again, I think this additional check in the external toolchain code is
> > useless. For a SDK to be usable as an external toolchain, it has to be
> > a SDK that contains *just* a toolchain and *nothing* else. If its
> > sysroot is polluted with additional libraries than the C library, it
> > cannot be used as an external toolchain. And therefore, the
> > relocate-sdk procedure is useless, because there's nothing to relocate
> > in an SDK that is just a toolchain.  
> 
> OK. I see. Then I will drop this patch.

Let's see if Arnout agrees with my reasoning. Quite often he points me
at details I have missed :-)

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Arnout Vandecappelle]

On 21-07-17 10:05, Thomas Petazzoni wrote:
[snip]
>>> For a SDK to be usable as an external toolchain, it has to be
>>> a SDK that contains *just* a toolchain and *nothing* else. If its
>>> sysroot is polluted with additional libraries than the C library, it
>>> cannot be used as an external toolchain.

 This is not true. I already regularly use a Buildroot SDK in projects by
setting BR2_HOST_DIR=/opt/... and packaging that. This SDK is then used to
create a rootfs+kernel with Buildroot, and separately to build applications.
Since the applications need some additional libraries, these libraries are
included in the SDK. They are not used by Buildroot, so it's double work (they
are included in the SDK, and built again for the target rootfs).

 Admittedly it's not perfect, but it works. At least, as long as you don't go
and update Buildroot (thus putting new versions of the libraries on the target)
without updating the SDK as well.

 However...

>>> And therefore, the
>>> relocate-sdk procedure is useless, because there's nothing to relocate
>>> in an SDK that is just a toolchain.  
>>
>> OK. I see. Then I will drop this patch.
> 
> Let's see if Arnout agrees with my reasoning. Quite often he points me
> at details I have missed :-)

 relocate-sdk is still useless when using it as an external toolchain. You could
say it makes sense for a pre-installed toolchain, but then only if you're using
that SDK outside of Buildroot. Buildroot is not going to use any of the
additional stuff.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Thomas Petazzoni]

Hello,

On Fri, 21 Jul 2017 22:31:48 +0200, Arnout Vandecappelle wrote:

>  This is not true. I already regularly use a Buildroot SDK in projects by
> setting BR2_HOST_DIR=/opt/... and packaging that. This SDK is then used to
> create a rootfs+kernel with Buildroot, and separately to build applications.
> Since the applications need some additional libraries, these libraries are
> included in the SDK. They are not used by Buildroot, so it's double work (they
> are included in the SDK, and built again for the target rootfs).

This is really horrible, because it means that some libraries that
Buildroot doesn't know about may be present in the toolchain sysroot,
be detected by packages, etc.

>  Admittedly it's not perfect, but it works. At least, as long as you don't go
> and update Buildroot (thus putting new versions of the libraries on the target)
> without updating the SDK as well.
> 
>  However...
> 
> >>> And therefore, the
> >>> relocate-sdk procedure is useless, because there's nothing to relocate
> >>> in an SDK that is just a toolchain.    
> >>
> >> OK. I see. Then I will drop this patch.  
> > 
> > Let's see if Arnout agrees with my reasoning. Quite often he points me
> > at details I have missed :-)  
> 
>  relocate-sdk is still useless when using it as an external toolchain. You could
> say it makes sense for a pre-installed toolchain, but then only if you're using
> that SDK outside of Buildroot. Buildroot is not going to use any of the
> additional stuff.

Sorry, but I did not understand this paragraph :/

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v8 08/10] external-toolchain: check if a buildroot SDK has already been relocated

[Arnout Vandecappelle]

On 21-07-17 22:40, Thomas Petazzoni wrote:
> Hello,
> 
> On Fri, 21 Jul 2017 22:31:48 +0200, Arnout Vandecappelle wrote:
> 
>>  This is not true. I already regularly use a Buildroot SDK in projects by
>> setting BR2_HOST_DIR=/opt/... and packaging that. This SDK is then used to
>> create a rootfs+kernel with Buildroot, and separately to build applications.
>> Since the applications need some additional libraries, these libraries are
>> included in the SDK. They are not used by Buildroot, so it's double work (they
>> are included in the SDK, and built again for the target rootfs).
> 
> This is really horrible, because it means that some libraries that
> Buildroot doesn't know about may be present in the toolchain sysroot,
> be detected by packages, etc.

 No it doesn't. Buildroot doesn't use the external toolchain as-is; the relevant
files are copied to STAGING_DIR. Only the gcc related includes are still used
from HOST_DIR/opt/ext-toolchain. Just do a gcc -v link and you'll see.

[snip]
>>  relocate-sdk is still useless when using it as an external toolchain. You could
>> say it makes sense for a pre-installed toolchain, but then only if you're using
>> that SDK outside of Buildroot. Buildroot is not going to use any of the
>> additional stuff.
> 
> Sorry, but I did not understand this paragraph :/

 Since Buildroot doesn't use anything from HOSTDIR/opt/ext-toolchain,
"relocating" it isn't relevant.

 Bottom line: indeed this patch should be dropped.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v8 09/10] core: install relocation script and location to make a relocatable SDK

[Wolfgang Grandegger]

The script "relocate-sdk.sh" is installed into the top directory of
the SDK (HOST_DIR) and the SDK location path is stored in the file
"HOST_DIR/share/buildroot/sdk-location"-

Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/Makefile b/Makefile
index c54c420..ee0ff13 100644
--- a/Makefile
+++ b/Makefile
@@ -558,6 +558,8 @@ sdk: toolchain $(PACKAGES)
 	$(TOPDIR)/support/scripts/fix-rpath host
 	@$(call MESSAGE,"Sanitizing RPATH in staging tree")
 	$(TOPDIR)/support/scripts/fix-rpath staging
+	install $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)
+	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
 
 # Compatibility symlink in case a post-build script still uses $(HOST_DIR)/usr
 $(HOST_DIR)/usr: $(HOST_DIR)
-- 
2.7.4

[Buildroot] [PATCH v8 10/10] package/qt5base: provide "qt.conf" to make "qmake" relocatable

[Wolfgang Grandegger]

The file "qt.conf" can be used to override the hard-coded paths that
are compiled into the Qt library. This is required to make "qmake"
relocatable. Actually, we need to specify all variables to overwrite
the compiled in values.

CC: Julien Corjon <corjon.j@ecagroup.com>
CC: Peter Seiderer <ps.report@gmx.net>
Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
---
 package/qt5/qt5base/qt.conf.in | 19 +++++++++++++++++++
 package/qt5/qt5base/qt5base.mk |  8 ++++++++
 2 files changed, 27 insertions(+)
 create mode 100644 package/qt5/qt5base/qt.conf.in

diff --git a/package/qt5/qt5base/qt.conf.in b/package/qt5/qt5base/qt.conf.in
new file mode 100644
index 0000000..9a0b9cd
--- /dev/null
+++ b/package/qt5/qt5base/qt.conf.in
@@ -0,0 +1,19 @@
+[Paths]
+Prefix=@@HOST_DIR@@
+Sysroot=@@STAGING_DIR@@
+Headers=/usr/include/qt5
+Libraries=/usr/lib
+LibraryExecutables=/usr/libexec
+Binaries=/usr/bin
+Plugins=/usr/lib/qt/plugins
+Examples=/usr/lib/qt/examples
+Qml2Imports=/usr/qml
+Imports=/usr/imports
+Translations=/usr/translations
+Examples=/usr/lib/qt/examples
+Demos=/usr/lib/qt/examples
+Tests=/usr/tests
+Settings=/usr
+Documentation=/usr/doc
+ArchData=/usr
+Data=/usr
diff --git a/package/qt5/qt5base/qt5base.mk b/package/qt5/qt5base/qt5base.mk
index 92633af..bbbdd41 100644
--- a/package/qt5/qt5base/qt5base.mk
+++ b/package/qt5/qt5base/qt5base.mk
@@ -264,9 +264,17 @@ define QT5BASE_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
+# The file "qt.conf" can be used to override the hard-coded paths that are
+# compiled into the Qt library. We need it to make "qmake" relocatable.
+define QT5BASE_INSTALL_QT_CONF
+	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
+		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
+endef
+
 define QT5BASE_INSTALL_STAGING_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) install
 	$(QT5_LA_PRL_FILES_FIXUP)
+	$(QT5BASE_INSTALL_QT_CONF)
 endef
 
 define QT5BASE_INSTALL_TARGET_LIBS
-- 
2.7.4

[Buildroot] [PATCH v8 10/10] package/qt5base: provide "qt.conf" to make "qmake" relocatable

[Thomas Petazzoni]

Hello,

On Thu, 20 Jul 2017 16:35:21 +0200, Wolfgang Grandegger wrote:
> The file "qt.conf" can be used to override the hard-coded paths that
> are compiled into the Qt library. This is required to make "qmake"
> relocatable. Actually, we need to specify all variables to overwrite
> the compiled in values.
> 
> CC: Julien Corjon <corjon.j@ecagroup.com>
> CC: Peter Seiderer <ps.report@gmx.net>
> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com>
> ---
>  package/qt5/qt5base/qt.conf.in | 19 +++++++++++++++++++
>  package/qt5/qt5base/qt5base.mk |  8 ++++++++
>  2 files changed, 27 insertions(+)
>  create mode 100644 package/qt5/qt5base/qt.conf.in

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com