[Buildroot] [RFCv3 00/15] Per-package host/target directory support

Buildroot Archive on lore.kernel.org
 help / color / mirror / Atom feed

* [Buildroot] [RFCv3 00/15] Per-package host/target directory support
@ 2017-12-01 20:53 Thomas Petazzoni
  2017-12-01 20:53 ` [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path Thomas Petazzoni
                   ` (15 more replies)
  0 siblings, 16 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

Hello,

Here is a third iteration of the per-package SDK and target directory
implementation.

If you're interested in testing it, you can find the patch series at:

  http://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v3

Note: in the following discussion, we use the terminology "SDK" to
refer to the combination of HOST_DIR and STAGING_DIR, i.e the
cross-compiler and its sysroot with target libraries/headers, and a
bunch of other native programs. Since STAGING_DIR is included inside
HOST_DIR, the SDK directory is in essence just HOST_DIR.

Changes since v2
================

 - Solved the problem of all DEPENDENCIES_HOST_PREREQ targets:
   host-ccache, host-tar, host-lzip, host-xz, host-fakedate.

   To achieve this, introduced <pkg>_EXTRACT_DEPENDENCIES and used
   that to handle the dependencies on host-tar, host-lzip and host-xz.

   Used regular dependencies for host-ccache and host-fakedate.

   See below for more details.

Changes since v1
================

 - Rebased on the latest Buildroot next branch

 - The pkg-config changes are reworked according to Arnout's comments:
   only @STAGING_SUBDIR@ is replaced in pkg-config.in to generate
   pkg-config, the rest of the logic is internal to the script.

 - New patch to move the "host skeleton" logic into a proper
   host-skeleton package.

 - New patch to have the CMake related files generated as part of the
   toolchain package.

 - New patch to add a new "install" step that depends on the different
   install steps (target, staging, images, host). This is needed to
   later add some logic that is executed once all install steps of a
   package have been done.

 - Fix the approach to solve the RPATH logic: instead of using
   LD_LIBRARY_PATH, we actually fix with patchelf the RPATH of host
   binaries right after their installation.

 - Misc other improvements in the per-package SDK implementation.

What is per-package SDK and target?
===================================

Currently, output/host (SDK) and output/target are global directories:
all packages use files installed by other packages in those global
directories, and in turn into those global directories.

The idea of per-package SDK and target is that instead of having such
global directories, we would have one SDK directory and one target
directory per-package, containing just what this package needs
(dependencies), and in which the package installs its files.

Why do we want per-package SDK and target?
==========================================

There are two main motivations for per-package SDK and target
directories, which are related:

 1. Since the SDK directory is global, a package can currently see
    libraries/headers that it has not explicitly expressed a
    dependency on. So package A may not have a dependency on package
    B, but if package B happens to have been installed before, and
    package A "configure" script automaticatically detects a library
    installed by package B, it will use it. We have added tons of
    conditional dependencies in Buildroot to make such situations less
    problematic, but it is an endless fight, as upstream developers
    constantly add more optional dependencies to their software.

    Using per-package SDK, a given package uses as its SDK a directory
    that only contains the libraries/headers from packages explicitly
    listed in its dependencies. So it cannot mistakenly see a
    library/header installed by another package.

 2. Top-level parallel build (building multiple packages in parallel)
    is currently not enabled because we have global SDK and target
    directories.

    Let's imagine package A configure script auto-detects a library
    provided by package B, but Buildroot does not encode this
    dependency in package A's .mk file. Package A and package B are
    therefore totally independent from a build dependency point of
    view.

    Without top-level parallel build (current situation), you have a
    guarantee on the build order: either A is always built before B,
    or B is always built before A. So at least, every build gives the
    same result: A is built with B's functionality or without it, but
    it is consistent accross rebuilds.

    If you enable top-level parallel build, because A and B are
    totally independent, you can get A built before B or B built
    before A depending on the build scheduling. This can change at
    every build! This means that for a given configuration, A will
    sometimes be built with B's functionality, sometimes not. Totally
    unacceptable.

    And of course, this extends to case where package B gets installed
    while package A is being configured. So package A configure script
    will see some parts of B being installed, other parts not
    installed, causing all sort of weird and difficult to debug race
    conditions.

    By having per-package SDK directories, we ensure that package A
    will only see the dependencies its has explicitly expressed, and
    that no other package gets installed in parallel in the same SDK
    directory.

How is it implemented?
======================

The basic idea is pretty simple:

 - For each package, we have:

   output/per-package/<package>/host, which is the specific SDK
   directory for <package>.

   output/per-package/<package>/target, which is the specific target
   directory for <package>.

 - Before <package> is configured, we copy into its specific SDK and
   target directories the SDK and target directories from its direct
   dependencies. This basically populates its SDK with the
   cross-compiler and all libraries that <package> needs. This copy is
   done using rsync with hard links, so it is very fast and cheap from
   a storage consumption point of view.

 - While <package> is being built, HOST_DIR, STAGING_DIR and
   TARGET_DIR are tweaked to point to the current <package> specific
   SDK and target directories. I.e, HOST_DIR no longer points to
   output/host, but to output/per-package/<package>/host/.

And this is basically enough to get things working!

There are of course a few additional things to do:

 - At the end of the build, we still want a global target directory
   (to generate the filesystem images) and a global SDK. Therefore, in
   target-finalize, we assemble such global directories by copying the
   per-package SDK and target directories from all packages listed in
   the $(PACKAGES) variable.

 - We need to fix our pkg-config wrapper script so that it uses
   relative path instead of absolute paths. This allows the wrapper
   script to work regardless of which per-package SDK directory it is
   installed in.

 - We need to move away from using absolute RPATH for host binaries,
   and to achieve that we fix the RPATH to $ORIGIN/../lib in host
   binaries right after the installation of each package.

 - We need to get away from the DEPENDENCIES_HOST_PREREQ mechanism,
   used for host-ccache, host-tar, host-lzip, host-xz and
   host-fakedate. The problem with this mechanism is that because
   those dependencies are not accounted in each package
   <pkg>_DEPENDENCIES variable, the per-package host/target folders
   produced by those packages are not rsync'ed when building other
   packages. Due to this, ccache/tar/lzip/xz/date cannot be found in
   the host directory of the package being built.

   To solve this, we basically move all those packages to be proper
   dependencies of all other packages. This involves adding a few
   exceptions, but more importantly, it requires adding the concept of
   "extract dependency", i.e a dependency that is ready before the
   extract step of a package. This is used for host-tar, host-lzip and
   host-xz. host-ccache and host-fakedate are regular dependencies.

What remains to be done?
========================

 - Completely get rid of the global HOST_DIR, TARGET_DIR and
   STAGING_DIR definitions, so that nothing uses them, and they really
   become per-package variables.

 - While the toolchain sysroot, pkg-config paths and host RPATHs are
   properly handled, there are quite certainly a lot of other places
   that have absolute paths that won't work well with per-package SDK,
   and that need to be adjusted.

 - Perhaps use a temporary per-package SDK and target directory when
   building a package, and rename it to the final name once the
   package has finished building. This would allow to detect
   situations where the per-package SDK directory of package A
   contains absolute paths referring to the per-package SDK directory
   of package B. Such absolute paths are wrong, but we currently don't
   see them because the per-package SDK directory for package B still
   exists.

 - Think about what needs to be done about the output/images/
   folder. Should it also be per-package like we do for output/target?

 - Many other issues that I'm sure people will report.

Comparison with Gustavo's patch series
======================================

Gustavo Zacarias did a previous implementation of this, available in
http://repo.or.cz/buildroot-gz.git/shortlog/refs/heads/pps3-next. Compared
to Gustavo's patch series, this patch series:

 - Implement per-package SDK and target directories, while Gustavo was
   only doing per-package staging.

 - Gustavo had several patches to pass BR_TOOLCHAIN_SYSROOT to the
   toolchain wrapper to pass the location of the current sysroot.

   This is no longer needed, because 1/ we're doing a full per-package
   SDK directory rather than per-package staging, which means the
   sysroot is always at the same relative location compared to the
   cross-compiler binary and 2/ the toolchain wrapper derives the
   sysroot location relatively to the wrapper location. Thanks to
   this, the following patches from Gustavo are (I believe) not
   needed:

   http://repo.or.cz/buildroot-gz.git/commitdiff/1ef6e798064649726d396bcb581ddbe4573c2460
   http://repo.or.cz/buildroot-gz.git/commitdiff/d15f6c6917b555bbbbbf97c292fad4db9d4007d4
   http://repo.or.cz/buildroot-gz.git/commitdiff/95900b65c29a010d85a769c9c6374cf3835f2169
   http://repo.or.cz/buildroot-gz.git/commitdiff/7ed53a5437ab09b85bf6d1953f6ff6049dfcc114

 - Gustavo had a patch to make our pkg-config wrapper script look at
   BR_TOOLCHAIN_SYSROOT, in order to find the
   /usr/{lib,share}/pkgconfig folders of the current sysroot. Just
   like for the toolchain-wrapper, such a change is no longer needed,
   and a simple change to our pkg-config script to use relative paths
   is sufficient.

 - I haven't integrated Gustavo patches that move from $(shell ...) to
   using backticks to delay the evaluation of
   STAGING_DIR/HOST_DIR/TARGET_DIR:

   http://repo.or.cz/buildroot-gz.git/commitdiff/0c11c60865f1445830643bb404f92c20d563260c
   http://repo.or.cz/buildroot-gz.git/commitdiff/207d2248ec8542293b6201b5c86f971021a3a591
   http://repo.or.cz/buildroot-gz.git/commitdiff/7f6a69819fbb176e29a1c3a53b4a76efe87dad15
   http://repo.or.cz/buildroot-gz.git/commitdiff/3328a9745eee555f5e5ef7df835afc26612ecd7b
   http://repo.or.cz/buildroot-gz.git/commitdiff/45a9d8c2aa6f3c0d2d8ed989d843890025faa3e8

 - I haven't looked at Qt specific issues, such as the ones fixed by
   Gustavo in:

   http://repo.or.cz/buildroot-gz.git/commitdiff/643e982e635f4386ccefcda9da4b84536af84d04

 - I am not doing all the .la files, *-config, *.prl, etc. tweaks that
   Gustavo is doing in:

   http://repo.or.cz/buildroot-gz.git/commitdiff/3f7b227b4c8e4514df6e5d54f88fcfb3a3a4bae0

   It is part of the "remaining absolute paths that need to be fixed"
   item that I mentionned above.

Best regards,

Thomas

Thomas Petazzoni (15):
  pkgconf: use relative path to STAGING_DIR instead of absolute path
  toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN
  Makefile, skeleton: move the host skeleton logic to host-skeleton
    package
  pkg-cmake: install CMake files as part of a package
  pkg-generic: add .stamp_installed step
  package/pkg-generic: add the concept of extract dependency
  package/pkg-generic: handle host-tar as an extract dependency
  package/pkg-generic: handle host-xz as an extract dependency
  package/pkg-generic: handle host-lzip as an extract dependency
  package/pkg-generic: handle host-ccache as a regular dependency
  package/pkg-generic: handle host-fakedate as a regular dependency
  core: kill DEPENDENCIES_HOST_PREREQ
  core: change host RPATH handling
  Makefile: evaluate CCACHE and HOST{CC,CXX} at time of use
  core: implement per-package SDK and target

 Makefile                                           |  43 ++++----
 docs/manual/adding-packages-generic.txt            |   7 ++
 package/Makefile.in                                |   2 +-
 package/ccache/ccache.mk                           |   5 +
 package/lzip/lzip.mk                               |   2 +-
 package/pkg-cmake.mk                               |  12 ++-
 package/pkg-generic.mk                             | 113 ++++++++++++++++-----
 package/pkgconf/pkg-config.in                      |   5 +-
 package/pkgconf/pkgconf.mk                         |   3 +-
 package/skeleton/skeleton.mk                       |  12 +++
 package/tar/tar.mk                                 |   6 ++
 package/xz/xz.mk                                   |   5 +
 support/dependencies/check-host-lzip.mk            |   2 +-
 support/dependencies/check-host-tar.mk             |   2 +-
 support/dependencies/check-host-xzcat.mk           |   2 +-
 support/dependencies/dependencies.mk               |  14 +--
 support/scripts/fix-rpath                          |  31 +++---
 .../toolchain-external/pkg-toolchain-external.mk   |   2 +-
 toolchain/toolchain/toolchain.mk                   |   3 -
 19 files changed, 180 insertions(+), 91 deletions(-)

-- 
2.13.6

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 15:22   ` Yann E. MORIN
  2017-12-31 17:24   ` Thomas Petazzoni
  2017-12-01 20:53 ` [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN Thomas Petazzoni
                   ` (14 subsequent siblings)
  15 siblings, 2 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

The pkg-config wrapper script is currently generated with absolute
paths to $(STAGING_DIR). However, this will not work properly with
per-package SDK, and each package will be built with a different
STAGING_DIR value.

In order to fix this, we adjust how the pkg-config wrapper script is
generated, so that it uses a relative path to itself: the sysroot (i.e
STAGING_DIR) is always located in $(path of
pkg-config)/../$(STAGING_SUBDIR).

This change is independent from the per-package SDK work, and could be
applied independently from it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - None
Changes since v1:
 - As suggested by Arnout, simplify the generation of the pkg-config
   script by doing only a replacement on @STAGING_SUBDIR@, the rest
   being encoded inside the pkg-config script.
---
 package/pkgconf/pkg-config.in | 5 ++++-
 package/pkgconf/pkgconf.mk    | 3 +--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/package/pkgconf/pkg-config.in b/package/pkgconf/pkg-config.in
index 4dec48789a..b9ce0935cc 100644
--- a/package/pkgconf/pkg-config.in
+++ b/package/pkgconf/pkg-config.in
@@ -1,2 +1,5 @@
 #!/bin/sh
-PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:- at PKG_CONFIG_LIBDIR@} PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:- at STAGING_DIR@} $(dirname $0)/pkgconf @STATIC@ $@
+PKGCONFDIR=$(dirname $0)
+DEFAULT_PKG_CONFIG_LIBDIR=${PKGCONFDIR}/../@STAGING_SUBDIR@/usr/lib/pkgconfig:${PKGCONFDIR}/../@STAGING_SUBDIR@/usr/share/pkgconfig
+DEFAULT_PKG_CONFIG_SYSROOT_DIR=${PKGCONFDIR}/../@STAGING_SUBDIR@
+PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:-${DEFAULT_PKG_CONFIG_LIBDIR}} PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:-${DEFAULT_PKG_CONFIG_SYSROOT_DIR}} ${PKGCONFDIR}/pkgconf @STATIC@ $@
diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index cc190d26da..00b2d017ee 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -19,8 +19,7 @@ endef
 define HOST_PKGCONF_INSTALL_WRAPPER
 	$(INSTALL) -m 0755 -D package/pkgconf/pkg-config.in \
 		$(HOST_DIR)/bin/pkg-config
-	$(SED) 's, at PKG_CONFIG_LIBDIR@,$(STAGING_DIR)/usr/lib/pkgconfig:$(STAGING_DIR)/usr/share/pkgconfig,' \
-		-e 's, at STAGING_DIR@,$(STAGING_DIR),' \
+	$(SED) 's, at STAGING_SUBDIR@,$(STAGING_SUBDIR),g' \
 		$(HOST_DIR)/bin/pkg-config
 endef
 
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
  2017-12-01 20:53 ` [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 15:22   ` Yann E. MORIN
  2017-12-31 17:24   ` Thomas Petazzoni
  2017-12-01 20:53 ` [Buildroot] [RFCv3 03/15] Makefile, skeleton: move the host skeleton logic to host-skeleton package Thomas Petazzoni
                   ` (13 subsequent siblings)
  15 siblings, 2 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

The upcoming per-package SDK functionality is heavily based on the
fact that HOST_DIR, STAGING_DIR and TARGET_DIR are evaluated during
the configure/build/install steps of the packages. Therefore, any
evaluation-during-assignment using := is going to cause problems, and
need to be turned into evaluation-during-use using =.

This patch fix up one such instance in the external toolchain code.

This change is independent from the per-package SDK functionality, and
could be applied separately.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
Changes since v2:
 - None
Changes since v1:
 - Added Arnout Reviewed-by.
---
 toolchain/toolchain-external/pkg-toolchain-external.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
index dc0588c536..b9ad1720a1 100644
--- a/toolchain/toolchain-external/pkg-toolchain-external.mk
+++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
@@ -77,7 +77,7 @@ ifneq ($(TOOLCHAIN_EXTERNAL_PREFIX),)
 TOOLCHAIN_EXTERNAL_BIN := $(dir $(shell which $(TOOLCHAIN_EXTERNAL_PREFIX)-gcc))
 endif
 else
-TOOLCHAIN_EXTERNAL_BIN := $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/bin
+TOOLCHAIN_EXTERNAL_BIN = $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/bin
 endif
 
 # If this is a buildroot toolchain, it already has a wrapper which we want to
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 03/15] Makefile, skeleton: move the host skeleton logic to host-skeleton package
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
  2017-12-01 20:53 ` [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path Thomas Petazzoni
  2017-12-01 20:53 ` [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-03 22:22   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package Thomas Petazzoni
                   ` (12 subsequent siblings)
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

As part of the per-package SDK work, we want to avoid having logic
that installs files to the global HOST_DIR, and instead do it inside
packages. One thing that gets installed to the global HOST_DIR is the
minimal "skeleton" that we create in host:

 - the "usr" symbolic link for backward compatibility

 - the "lib" directory, and its lib64 or lib32 symbolic links

This commit moves this logic to a new host-skeleton package, and makes
all packages (except itself) depend on it. We also make sure that this
host-skeleton package doesn't depend on host-patchelf, because
host-patchelf depends on host-skeleton.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - None
Changes since v1:
 - New patch
---
 Makefile                     | 13 +------------
 package/pkg-generic.mk       |  4 ++++
 package/skeleton/skeleton.mk | 12 ++++++++++++
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/Makefile b/Makefile
index d52852153f..56bf083098 100644
--- a/Makefile
+++ b/Makefile
@@ -544,7 +544,7 @@ endif
 
 .PHONY: dirs
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
-	$(HOST_DIR) $(HOST_DIR)/usr $(HOST_DIR)/lib $(BINARIES_DIR)
+	$(HOST_DIR) $(BINARIES_DIR)
 
 $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 	$(MAKE1) $(EXTRAMAKEARGS) HOSTCC="$(HOSTCC_NOCCACHE)" HOSTCXX="$(HOSTCXX_NOCCACHE)" silentoldconfig
@@ -563,17 +563,6 @@ sdk: world
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
 
-# Compatibility symlink in case a post-build script still uses $(HOST_DIR)/usr
-$(HOST_DIR)/usr: $(HOST_DIR)
-	@ln -snf . $@
-
-$(HOST_DIR)/lib: $(HOST_DIR)
-	@mkdir -p $@
-	@case $(HOSTARCH) in \
-		(*64) ln -snf lib $(@D)/lib64;; \
-		(*)   ln -snf lib $(@D)/lib32;; \
-	esac
-
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):
 	@mkdir -p $(STAGING_DIR)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index fb5a1ca725..0019814276 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -577,6 +577,10 @@ $(2)_DEPENDENCIES += toolchain
 endif
 endif
 
+ifneq ($(1),host-skeleton)
+$(2)_DEPENDENCIES += host-skeleton
+endif
+
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index d380f41649..a32bacd0b3 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -11,4 +11,16 @@
 SKELETON_ADD_TOOLCHAIN_DEPENDENCY = NO
 SKELETON_ADD_SKELETON_DEPENDENCY = NO
 
+# We create a compatibility symlink in case a post-build script still
+# uses $(HOST_DIR)/usr
+define HOST_SKELETON_INSTALL_CMDS
+	@ln -snf . $(HOST_DIR)/usr
+	@mkdir -p $(HOST_DIR)/lib
+	@case $(HOSTARCH) in \
+		(*64) ln -snf lib $(HOST_DIR)/lib64;; \
+		(*)   ln -snf lib $(HOST_DIR)/lib32;; \
+	esac
+endef
+
 $(eval $(virtual-package))
+$(eval $(host-generic-package))
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (2 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 03/15] Makefile, skeleton: move the host skeleton logic to host-skeleton package Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-03 22:34   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step Thomas Petazzoni
                   ` (11 subsequent siblings)
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

Currently, the toolchainfile.cmake and Buildroot.cmake files are
installed outside of any package, just triggered by the toolchain
target.

As part of the per-package SDK effort, we are trying to avoid anything
that installs to the global $(HOST_DIR), and this is one of the
remaining files installed in $(HOST_DIR) outside of any package. We
fix this by installing such files as part of the toolchain package
post-install staging hooks.

Yes, a post-install staging hook to install things to $(HOST_DIR) is a
bit weird, but the toolchain infrastructure is made of target packages
only, and they all install a lot of stuff to $(HOST_DIR) already.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - None
Changes since v1:
 - New patch
---
 package/pkg-cmake.mk             | 12 +++++++-----
 toolchain/toolchain/toolchain.mk |  3 ---
 2 files changed, 7 insertions(+), 8 deletions(-)

diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
index 6739704e3c..6d4f948d07 100644
--- a/package/pkg-cmake.mk
+++ b/package/pkg-cmake.mk
@@ -244,8 +244,8 @@ endif
 # based on the toolchainfile.cmake file's location: $(HOST_DIR)/share/buildroot
 # In all the other variables, HOST_DIR will be replaced by RELOCATED_HOST_DIR,
 # so we have to strip "$(HOST_DIR)/" from the paths that contain it.
-$(HOST_DIR)/share/buildroot/toolchainfile.cmake:
-	@mkdir -p $(@D)
+define TOOLCHAIN_CMAKE_INSTALL_FILES
+	@mkdir -p $(HOST_DIR)/share/buildroot
 	sed \
 		-e 's#@@STAGING_SUBDIR@@#$(call qstrip,$(STAGING_SUBDIR))#' \
 		-e 's#@@TARGET_CFLAGS@@#$(call qstrip,$(TARGET_CFLAGS))#' \
@@ -259,7 +259,9 @@ $(HOST_DIR)/share/buildroot/toolchainfile.cmake:
 		-e 's#@@TOOLCHAIN_HAS_FORTRAN@@#$(if $(BR2_TOOLCHAIN_HAS_FORTRAN),1,0)#' \
 		-e 's#@@CMAKE_BUILD_TYPE@@#$(if $(BR2_ENABLE_DEBUG),Debug,Release)#' \
 		$(TOPDIR)/support/misc/toolchainfile.cmake.in \
-		> $@
+		> $(HOST_DIR)/share/buildroot/toolchainfile.cmake
+	$(Q)$(INSTALL) -D -m 0644 support/misc/Buildroot.cmake \
+		$(HOST_DIR)/share/buildroot/Platform/Buildroot.cmake
+endef
 
-$(HOST_DIR)/share/buildroot/Platform/Buildroot.cmake:
-	$(Q)$(INSTALL) -D -m 0644 support/misc/Buildroot.cmake $(@)
+TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += TOOLCHAIN_CMAKE_INSTALL_FILES
diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index b55b0c712c..283e0a74ee 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -38,6 +38,3 @@ TOOLCHAIN_INSTALL_STAGING = YES
 endif
 
 $(eval $(virtual-package))
-
-toolchain: $(HOST_DIR)/share/buildroot/toolchainfile.cmake
-toolchain: $(HOST_DIR)/share/buildroot/Platform/Buildroot.cmake
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (3 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 15:30   ` Yann E. MORIN
  2018-02-22 21:48   ` Matthew Weber
  2017-12-01 20:53 ` [Buildroot] [RFCv3 06/15] package/pkg-generic: add the concept of extract dependency Thomas Petazzoni
                   ` (10 subsequent siblings)
  15 siblings, 2 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

The installation to target, staging and images as well as the host
installation are all done in 4 different make targets. We don't have
any place currently to run something at the end of a package
installation, i.e once all of target/staging/images (for target) or
host (for host) installation have completed.

We will need such a step to fix the RPATH of binaries in the host
directory. Indeed, while normally only host packages should install
things in $(HOST_DIR), in practice a number of target packages (ex:
qt4, qt5base) do install things in $(HOST_DIR).

Therefore, we want to run our RPATH fixing logic at the end of all
installation steps. To achieve this, this commit introduces a new make
target, .stamp_installed, which depends on
.stamp_{target,staging,images}_installed for target packages and
.stamp_host_installed for host packages.

This new step currently doesn't do anything, but the actual logic is
added in a follow-up commit.

This change in fact makes the overall step sequencing more logical:
for all steps the dependencies were done on the stamp file targets,
except for the install step where we were using phony targets to make
$(1)-install depends on $(1)-install-{target,staging,images,host}. So
now, the phony targets $(1)-install-{target,staging,images,host} are
only human-readable targets that are shortcuts to the corresponding
stamp file targets, and not usded anymore for the step
sequencing. This makes them more consistent with the other phony
$(1)-{build,configure,patch,...} phony targets.

It also makes $(1)-install-host more consistent with other phony
targets: $(1)-install-target only existed if the package had
<pkg>_INSTALL_TARGET = YES, but $(1)-install-host existed
unconditionally. Now it only exists if it's a host package.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - None
Changes since v1:
 - New patch.
---
 package/pkg-generic.mk | 44 +++++++++++++++++++++++++++-----------------
 1 file changed, 27 insertions(+), 17 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 0019814276..2b821e9bdf 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -340,6 +340,12 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 	@$(call step_end,install-target)
 	$(Q)touch $@
 
+# Finalize installation
+$(BUILD_DIR)/%/.stamp_installed:
+	@$(call step_start,install)
+	@$(call step_end,install)
+	$(Q)touch $@
+
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
 	rm -Rf $(@D)
@@ -595,6 +601,7 @@ $(2)_TARGET_INSTALL_TARGET =	$$($(2)_DIR)/.stamp_target_installed
 $(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
 $(2)_TARGET_INSTALL_IMAGES =	$$($(2)_DIR)/.stamp_images_installed
 $(2)_TARGET_INSTALL_HOST =      $$($(2)_DIR)/.stamp_host_installed
+$(2)_TARGET_INSTALL =      	$$($(2)_DIR)/.stamp_installed
 $(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
 $(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
 $(2)_TARGET_RSYNC =	        $$($(2)_DIR)/.stamp_rsynced
@@ -642,38 +649,40 @@ $(2)_ROOTFS_POST_CMD_HOOKS      ?=
 # human-friendly targets and target sequencing
 $(1):			$(1)-install
 
+$(1)-install:			$$($(2)_TARGET_INSTALL)
+
 ifeq ($$($(2)_TYPE),host)
-$(1)-install:	        $(1)-install-host
-else
-$(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
-endif
+
+$(1)-install-host:		$$($(2)_TARGET_INSTALL_HOST)
+$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_HOST)
+
+else # host vs. target
 
 ifeq ($$($(2)_INSTALL_TARGET),YES)
 $(1)-install-target:		$$($(2)_TARGET_INSTALL_TARGET)
-$$($(2)_TARGET_INSTALL_TARGET):	$$($(2)_TARGET_BUILD)
-else
-$(1)-install-target:
+$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_TARGET)
 endif
 
 ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:			$$($(2)_TARGET_INSTALL_STAGING)
-$$($(2)_TARGET_INSTALL_STAGING):	$$($(2)_TARGET_BUILD)
-# Some packages use install-staging stuff for install-target
-$$($(2)_TARGET_INSTALL_TARGET):		$$($(2)_TARGET_INSTALL_STAGING)
-else
-$(1)-install-staging:
+$(1)-install-staging:		$$($(2)_TARGET_INSTALL_STAGING)
+$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_STAGING)
 endif
 
 ifeq ($$($(2)_INSTALL_IMAGES),YES)
 $(1)-install-images:		$$($(2)_TARGET_INSTALL_IMAGES)
-$$($(2)_TARGET_INSTALL_IMAGES):	$$($(2)_TARGET_BUILD)
-else
-$(1)-install-images:
+$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_IMAGES)
 endif
 
-$(1)-install-host:		$$($(2)_TARGET_INSTALL_HOST)
+endif # host vs.target
+
+$$($(2)_TARGET_INSTALL_TARGET):	$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_STAGING):$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_IMAGES):	$$($(2)_TARGET_BUILD)
 $$($(2)_TARGET_INSTALL_HOST):	$$($(2)_TARGET_BUILD)
 
+# Some packages use install-staging stuff for install-target
+$$($(2)_TARGET_INSTALL_TARGET):		$$($(2)_TARGET_INSTALL_STAGING)
+
 $(1)-build:		$$($(2)_TARGET_BUILD)
 $$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE)
 
@@ -813,6 +822,7 @@ $(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
 
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
+$$($(2)_TARGET_INSTALL):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
 $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 06/15] package/pkg-generic: add the concept of extract dependency
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (4 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 15:33   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an " Thomas Petazzoni
                   ` (9 subsequent siblings)
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

Extract dependencies are dependencies that must be ready before the
extract step of a package, i.e for tools that are needed to extract
packages themselves. Current examples of such tools are host-tar,
host-lzip and host-xz.

They are currently handled through DEPENDENCIES_HOST_PREREQ. However,
this mechanism has a number of drawbacks:

 - First and foremost, because host-tar/host-lzip/host-xz are not
   listed in the dependencies of packages, the package infrastructure
   does not know it should rsync them in the context of per-package
   SDK.

 - Second, there is no dependency handling *between* them. I.e, we
   have no mechanism that says host-tar should be built before
   host-lzip, while it is in fact the case: if you need to build
   host-lzip, you need to extract a tarball, so you may need host-tar
   if your system tarball is not capable enough.

For those reasons, it makes sense to add explicit support for "extract
dependencies" in the package infrastructure, through the
<pkg>_EXTRACT_DEPENDENCIES variable. It is unlikely this variable will
ever be used by a package .mk file, but it will be used internally by
the package infrastructure.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 docs/manual/adding-packages-generic.txt | 7 +++++++
 package/pkg-generic.mk                  | 8 +++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 63ea51bf89..7f42cb09e9 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -347,6 +347,13 @@ information is (assuming the package name is +libfoo+) :
   a similar way, +HOST_LIBFOO_DEPENDENCIES+ lists the dependencies for
   the current host package.
 
+* +LIBFOO_EXTRACT_DEPENDENCIES+ lists the dependencies (in terms of
+  package name) that are required for the current target package to be
+  extracted. These depnedencies are guaranteed to be compiled and
+  installed before the extract step of the current package
+  starts. This is only use internally by the package infrastructure,
+  and should typically not be used directly by packages.
+
 * +LIBFOO_PATCH_DEPENDENCIES+ lists the dependencies (in terms of
   package name) that are required for the current package to be
   patched. These dependencies are guaranteed to be extracted and
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 2b821e9bdf..824c34bec9 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -589,8 +589,13 @@ endif
 
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
+$(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
 $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
-$(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))
+$(2)_FINAL_ALL_DEPENDENCIES = \
+	$$(sort \
+		$$($(2)_FINAL_DEPENDENCIES) \
+		$$($(2)_FINAL_EXTRACT_DEPENDENCIES) \
+		$$($(2)_FINAL_PATCH_DEPENDENCIES))
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
@@ -716,6 +721,7 @@ $$($(2)_TARGET_PATCH):  | $$(patsubst %,%-patch,$$($(2)_FINAL_PATCH_DEPENDENCIES
 
 $(1)-extract:			$$($(2)_TARGET_EXTRACT)
 $$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
+$$($(2)_TARGET_EXTRACT): | $$($(2)_FINAL_EXTRACT_DEPENDENCIES)
 
 $(1)-depends:		$$($(2)_FINAL_DEPENDENCIES)
 
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an extract dependency
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (5 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 06/15] package/pkg-generic: add the concept of extract dependency Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-02 15:06   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz " Thomas Petazzoni
                   ` (8 subsequent siblings)
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

This moves the host-tar dependency handling from
DEPENDENCY_HOST_PREREQ to an extract dependency.

To achieve that, check-host-tar.mk fills in the
BR2_TAR_HOST_DEPENDENCY with host-tar if building a host-tar is
needed. The name BR2_TAR_HOST_DEPENDENCY has been chosen because it
matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
check-host-cmake.mk.

The BR2_TAR_HOST_DEPENDENCY is added to all packages, except host-tar
itself (obviously) and host-skeleton, because we depend on
host-skeleton to install host-tar properly in HOST_DIR.

In addition, we modify tar.mk to explicitly build host-tar without
ccache: since ccache source code is available as a tarball, ccache
will obviously depend on host-tar if the system tar is insufficient.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 package/pkg-generic.mk                 | 4 ++++
 package/tar/tar.mk                     | 6 ++++++
 support/dependencies/check-host-tar.mk | 2 +-
 3 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 824c34bec9..86dd2561d4 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -587,6 +587,10 @@ ifneq ($(1),host-skeleton)
 $(2)_DEPENDENCIES += host-skeleton
 endif
 
+ifeq ($(filter host-tar host-skeleton,$(1)),)
+$(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
+endif
+
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
diff --git a/package/tar/tar.mk b/package/tar/tar.mk
index cb2ddc0eca..813eabed14 100644
--- a/package/tar/tar.mk
+++ b/package/tar/tar.mk
@@ -44,4 +44,10 @@ define HOST_TAR_EXTRACT_CMDS
 	mv $(@D)/tar-$(TAR_VERSION)/* $(@D)
 	rmdir $(@D)/tar-$(TAR_VERSION)
 endef
+
+# we are built before ccache
+HOST_TAR_CONF_ENV = \
+	CC="$(HOSTCC_NOCCACHE)" \
+	CXX="$(HOSTCXX_NOCCACHE)"
+
 $(eval $(host-autotools-package))
diff --git a/support/dependencies/check-host-tar.mk b/support/dependencies/check-host-tar.mk
index ad0b32e277..d07f727c4c 100644
--- a/support/dependencies/check-host-tar.mk
+++ b/support/dependencies/check-host-tar.mk
@@ -1,6 +1,6 @@
 TAR ?= tar
 
 ifeq (,$(call suitable-host-package,tar,$(TAR)))
-DEPENDENCIES_HOST_PREREQ += host-tar
 TAR = $(HOST_DIR)/bin/tar
+BR2_TAR_HOST_DEPENDENCY += host-tar
 endif
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz as an extract dependency
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (6 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an " Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-02 15:08   ` Yann E. MORIN
  2017-12-29 15:36   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip " Thomas Petazzoni
                   ` (7 subsequent siblings)
  15 siblings, 2 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

This moves the host-xz dependency handling from
DEPENDENCY_HOST_PREREQ to an extract dependency.

To achieve that, check-host-xz.mk fills in the
BR2_XZ_HOST_DEPENDENCY with host-tar if building a host-tar is
needed. The name BR2_XZ_HOST_DEPENDENCY has been chosen because it
matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
check-host-cmake.mk.

The BR2_XZ_HOST_DEPENDENCY is added to all packages, except:

 - host-xz, because we would otherwise depend on ourself.

 - host-tar, because xz itself is delivered as a tarball, so we need
   to have host-xz depend on host-tar, and not host-tar depend on
   host-xz

 - host-skeleton, because we need to have host-xz depend on
   host-skeleton, and not the opposite.

In addition, we modify xz.mk to explicitly build host-xz without
ccache. We generally took the approach of building host-ccache *after*
all the extractors have been built.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 package/pkg-generic.mk                   | 4 ++++
 package/xz/xz.mk                         | 5 +++++
 support/dependencies/check-host-xzcat.mk | 2 +-
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 86dd2561d4..dfd4719d29 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -591,6 +591,10 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
 endif
 
+ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
+$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
+endif
+
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
diff --git a/package/xz/xz.mk b/package/xz/xz.mk
index e8116f24ae..bcdac13ee3 100644
--- a/package/xz/xz.mk
+++ b/package/xz/xz.mk
@@ -18,5 +18,10 @@ else
 XZ_CONF_OPTS = --disable-threads
 endif
 
+# we are built before ccache
+HOST_XZ_CONF_ENV = \
+	CC="$(HOSTCC_NOCCACHE)" \
+	CXX="$(HOSTCXX_NOCCACHE)"
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
index 9be75c7311..e5d72f95dd 100644
--- a/support/dependencies/check-host-xzcat.mk
+++ b/support/dependencies/check-host-xzcat.mk
@@ -2,7 +2,7 @@
 # If it is not present, build our own host-xzcat
 
 ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
-DEPENDENCIES_HOST_PREREQ += host-xz
+BR2_XZCAT_HOST_DEPENDENCY = host-xz
 EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
 XZCAT = $(HOST_DIR)/bin/xzcat
 endif
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip as an extract dependency
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (7 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz " Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-02 15:12   ` Yann E. MORIN
  2017-12-29 15:36   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 10/15] package/pkg-generic: handle host-ccache as a regular dependency Thomas Petazzoni
                   ` (6 subsequent siblings)
  15 siblings, 2 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

This moves the host-xz dependency handling from
DEPENDENCY_HOST_PREREQ to an extract dependency.

To achieve that, check-host-xz.mk fills in the
BR2_LZIP_HOST_DEPENDENCY with host-tar if building a host-tar is
needed. The name BR2_LZIP_HOST_DEPENDENCY has been chosen because it
matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
check-host-cmake.mk.

The BR2_LZIP_HOST_DEPENDENCY is added to all packages, except:

 - host-lzip, because we would otherwise depend on ourself.

 - host-tar, because lzip itself is delivered as a tarball, so we need
   to have host-lzip depend on host-tar, and not host-tar depend on
   host-lzip

 - host-skeleton, because we need to have host-lzip depend on
   host-skeleton, and not the opposite.

We also mutually exclude host-lzip and host-xz from dependending on
each other, to avoid a circular dependency.

In addition, we modify lzip.mk to explicitly build host-lzip without
ccache. We generally took the approach of building host-ccache *after*
all the extractors have been built.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 package/lzip/lzip.mk                    | 2 +-
 package/pkg-generic.mk                  | 6 +++++-
 support/dependencies/check-host-lzip.mk | 2 +-
 3 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/package/lzip/lzip.mk b/package/lzip/lzip.mk
index b7ba5dd21d..a16a22214c 100644
--- a/package/lzip/lzip.mk
+++ b/package/lzip/lzip.mk
@@ -16,7 +16,7 @@ endef
 
 define HOST_LZIP_CONFIGURE_CMDS
 	(cd $(@D); $(HOST_MAKE_ENV) ./configure --prefix=$(HOST_DIR) \
-		$(HOST_CONFIGURE_OPTS) )
+		$(HOST_CONFIGURE_OPTS) CC="$(HOSTCC_NOCCACHE)" CXX="$(HOSTCXX_NOCCACHE)")
 endef
 
 define LZIP_BUILD_CMDS
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index dfd4719d29..e623afe7bc 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -591,10 +591,14 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
 endif
 
-ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
+ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
 endif
 
+ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
+$(2)_EXTRACT_DEPENDENCIES += $(BR2_LZIP_HOST_DEPENDENCY)
+endif
+
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
index 00cdd0a236..cdd784058c 100644
--- a/support/dependencies/check-host-lzip.mk
+++ b/support/dependencies/check-host-lzip.mk
@@ -1,5 +1,5 @@
 ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
-DEPENDENCIES_HOST_PREREQ += host-lzip
+BR2_LZIP_HOST_DEPENDENCY = host-lzip
 EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
 LZCAT = $(HOST_DIR)/bin/lzip -d -c
 endif
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 10/15] package/pkg-generic: handle host-ccache as a regular dependency
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (8 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip " Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 15:42   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 11/15] package/pkg-generic: handle host-fakedate " Thomas Petazzoni
                   ` (5 subsequent siblings)
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

This moves the host-ccache dependency handling from
DEPENDENCIES_HOST_PREREQ to a proper package dependency. When
BR2_CCACHE=y, we add host-ccache as a regular dependency of all
packages except:

 - The extractor packages host-tar, host-xz and host-lzip

 - host-ccache itself

 - host-skeleton, because all packages depend on it

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 package/ccache/ccache.mk             | 5 +++++
 package/pkg-generic.mk               | 6 ++++++
 support/dependencies/dependencies.mk | 4 ----
 3 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index afbec44fac..d6ef455e8b 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -21,6 +21,11 @@ CCACHE_LICENSE_FILES = LICENSE.txt GPL-3.0.txt
 # has zero dependency besides the C library.
 HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
 
+# We are ccache, so we can't use ccache
+HOST_CCACHE_CONF_ENV = \
+	CC="$(HOSTCC_NOCCACHE)" \
+	CXX="$(HOSTCXX_NOCCACHE)"
+
 # Patch host-ccache as follows:
 #  - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
 #    is already used by autotargets for the ccache package.
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index e623afe7bc..cb5889c9ef 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -599,6 +599,12 @@ ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_LZIP_HOST_DEPENDENCY)
 endif
 
+ifeq ($(BR2_CCACHE),y)
+ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
+$(2)_DEPENDENCIES += host-ccache
+endif
+endif
+
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
diff --git a/support/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
index ef2ae9b7e1..1a4b5df9f2 100644
--- a/support/dependencies/dependencies.mk
+++ b/support/dependencies/dependencies.mk
@@ -14,10 +14,6 @@ $(shell support/dependencies/check-host-$(1).sh $(2))
 endef
 -include $(sort $(wildcard support/dependencies/check-host-*.mk))
 
-ifeq ($(BR2_CCACHE),y)
-DEPENDENCIES_HOST_PREREQ += host-ccache
-endif
-
 core-dependencies:
 	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
 		DL_TOOLS="$(sort $(DL_TOOLS_DEPENDENCIES))" \
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 11/15] package/pkg-generic: handle host-fakedate as a regular dependency
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (9 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 10/15] package/pkg-generic: handle host-ccache as a regular dependency Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 15:50   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 12/15] core: kill DEPENDENCIES_HOST_PREREQ Thomas Petazzoni
                   ` (4 subsequent siblings)
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

This commit moves the host-fakedate dependency handling from
DEPENDENCIES_HOST_PREREQ to a proper regular dependency handled by the
package infrastructure.

host-fakedate is added as dependency to all packages, except
host-skeleton, because we depend on it.

In addition, we make sure that host-fakedate does not grow a
dependency on host-{tar,xz,lzip,ccache} to avoid circular
dependencies. host-fakedate does not need any extraction tool and does
not need to build C/C++ code (the source code is just a shell script
available in Buildroot).

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 Makefile               |  1 -
 package/pkg-generic.mk | 14 ++++++++++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/Makefile b/Makefile
index 56bf083098..f31834682c 100644
--- a/Makefile
+++ b/Makefile
@@ -255,7 +255,6 @@ export LC_ALL = C
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
-DEPENDENCIES_HOST_PREREQ += host-fakedate
 endif
 
 # To put more focus on warnings, be less verbose as default
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index cb5889c9ef..7c5d951af9 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -587,24 +587,30 @@ ifneq ($(1),host-skeleton)
 $(2)_DEPENDENCIES += host-skeleton
 endif
 
-ifeq ($(filter host-tar host-skeleton,$(1)),)
+ifeq ($(filter host-tar host-skeleton host-fakedate,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
 endif
 
-ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
+ifeq ($(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
 endif
 
-ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
+ifeq ($(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_LZIP_HOST_DEPENDENCY)
 endif
 
 ifeq ($(BR2_CCACHE),y)
-ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
+ifeq ($(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
 $(2)_DEPENDENCIES += host-ccache
 endif
 endif
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+ifeq ($(filter host-skeleton,$(1)),)
+$(2)_DEPENDENCIES += host-fakedate
+endif
+endif
+
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 12/15] core: kill DEPENDENCIES_HOST_PREREQ
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (10 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 11/15] package/pkg-generic: handle host-fakedate " Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 15:53   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 13/15] core: change host RPATH handling Thomas Petazzoni
                   ` (3 subsequent siblings)
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

Now that DEPENDENCIES_HOST_PREREQ is no longer used anywhere, we can
kill it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 Makefile                             |  2 --
 package/pkg-generic.mk               |  2 --
 support/dependencies/dependencies.mk | 10 +++-------
 3 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/Makefile b/Makefile
index f31834682c..090b3ba191 100644
--- a/Makefile
+++ b/Makefile
@@ -485,8 +485,6 @@ include package/Makefile.in
 -include $(wildcard arch/arch.mk.*)
 include support/dependencies/dependencies.mk
 
-PACKAGES += $(DEPENDENCIES_HOST_PREREQ)
-
 include toolchain/*.mk
 include toolchain/*/*.mk
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 7c5d951af9..8c6e34cc9f 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -725,9 +725,7 @@ $(1)-configure:			$$($(2)_TARGET_CONFIGURE)
 $$($(2)_TARGET_CONFIGURE):	| $$($(2)_FINAL_DEPENDENCIES)
 
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
-ifeq ($$(filter $(1),$$(DEPENDENCIES_HOST_PREREQ)),)
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
-endif
 
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
diff --git a/support/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
index 1a4b5df9f2..d8ec137b12 100644
--- a/support/dependencies/dependencies.mk
+++ b/support/dependencies/dependencies.mk
@@ -14,18 +14,14 @@ $(shell support/dependencies/check-host-$(1).sh $(2))
 endef
 -include $(sort $(wildcard support/dependencies/check-host-*.mk))
 
-core-dependencies:
-	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
+dependencies:
+	@HOSTCC="$(firstword $(HOSTCC_NOCCACHE))" MAKE="$(MAKE)" \
 		DL_TOOLS="$(sort $(DL_TOOLS_DEPENDENCIES))" \
 		$(TOPDIR)/support/dependencies/dependencies.sh
 
-core-dependencies $(DEPENDENCIES_HOST_PREREQ): HOSTCC=$(HOSTCC_NOCCACHE)
-core-dependencies $(DEPENDENCIES_HOST_PREREQ): HOSTCXX=$(HOSTCXX_NOCCACHE)
-dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
-
 ################################################################################
 #
 # Toplevel Makefile options
 #
 ################################################################################
-.PHONY: dependencies core-dependencies
+.PHONY: dependencies
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (11 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 12/15] core: kill DEPENDENCIES_HOST_PREREQ Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 17:31   ` Yann E. MORIN
  2018-02-06 22:04   ` Matthew Weber
  2017-12-01 20:53 ` [Buildroot] [RFCv3 14/15] Makefile: evaluate CCACHE and HOST{CC, CXX} at time of use Thomas Petazzoni
                   ` (2 subsequent siblings)
  15 siblings, 2 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

Currently, our strategy for the RPATH of host binaries is as follows:

 - During the build, we force an absolute RPATH to be encoded into
   binaries, by passing -Wl,-rpath,$(HOST_DIR)/lib. This ensures that
   host binaries will find their libraries.

 - In the "make sdk" target, when preparing the SDK to be relocatable,
   we use patchelf to replace those absolute RPATHs by relative RPATHs
   that use $ORIGIN.

Unfortunately, the use of absolute RPATH during the build is not
compatible with the move to per-package SDK, where a different host
directory is used for the build of each package. Therefore, we need to
move to a different strategy for RPATH handling.

The new strategy is as follows:

 - We no longer pass -Wl,-rpath,$(HOST_DIR)/lib when building
   packages, so the host binaries no longer have any rpath encoded.

 - At the end of the installation of every package, we just the
   fix-rpath logic on the host binaries, so that all host binaries
   that don't already have $ORIGIN/../lib as their RPATH are updated
   to have such a RPATH.

In order to achieve this, the following changes are made:

 - HOST_LDFLAGS no longer contains -Wl,-rpath,$(HOST_DIR)

 - The hook that ran at the end of every package installation
   (check_host_rpath) to verify that our hardcoded RPATH is indeed
   present in all binaries is removed, as it is no longer needed: we
   will force $ORIGIN/../LIB as an RPATH in all binaries.

 - host-patchelf is added as a dependency of all packages, except
   itself. This is necessary as all packages now need to run patchelf
   at the end of their installation process.

   TODO: things like host-tar, host-lzip and host-ccache will have to
   be handled properly.

 - "fix-rpath host" is called at the end of the package installation,
   in the recently introduced .stamp_installed, which guarantees to be
   executed after all of target, staging, images and host
   installations have completed. Indeed, while in theory only host
   packages install files into $(HOST_DIR), in practice a number of
   target packages also install host binaries. So to be reliable, we
   do the "fix-rpath host" logic at the end of the installation of
   every package.

 - The fix-rpath script is modified:

   * The "host" target now simply hardcodes setting $ORIGIN/../lib as
     an RPATH, instead of using --relative-to-file

   * We simply exclude the patchelf program instead of making a copy
     of it: since patchelf depends only on the C/C++ libraries, it
     doesn't need to have a RPATH set to $ORIGIN/../lib

   * Also, in order to avoid re-writing host binaries over and over
     again, we only set the RPATH if it is not already set.

This change is independent from the per-package SDK functionality, and
could be applied separately.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
NOTE: an alternative to running "fix-rpath host" at the end of every
package installation would be to run it only for host packages
(unconditionally) and have a special flag that target packages can use
to indicate that they have installed things to $(HOST_DIR). Not sure
if this is worth the complexity, though.

NOTE2: is it OK to force $ORIGIN/../lib as the RPATH of all host
binaries? This is OK for binaries in host/{bin,sbin} of course, but do
we have host binaries in other places, for which this $ORIGIN/../lib
RPATH wouldn't be appropriate?

Changes since v2:
 - No changes

Changes since v1:
 - Following the feedback from Yann and Arnout, the approach in v2 was
   completely changed: instead of using LD_LIBRARY_PATH, we now run
   patchelf at the end of each package installation to fix the RPATH.
---
 Makefile                  |  1 -
 package/Makefile.in       |  2 +-
 package/pkg-generic.mk    | 14 ++++++--------
 support/scripts/fix-rpath | 31 +++++++++++++++++++------------
 4 files changed, 26 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index 090b3ba191..d8fa91120b 100644
--- a/Makefile
+++ b/Makefile
@@ -555,7 +555,6 @@ world: target-post-image
 .PHONY: sdk
 sdk: world
 	@$(call MESSAGE,"Rendering the SDK relocatable")
-	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
diff --git a/package/Makefile.in b/package/Makefile.in
index a1a5316051..e94a75c230 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -220,7 +220,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
 HOST_CFLAGS   ?= -O2
 HOST_CFLAGS   += $(HOST_CPPFLAGS)
 HOST_CXXFLAGS += $(HOST_CFLAGS)
-HOST_LDFLAGS  += -L$(HOST_DIR)/lib -Wl,-rpath,$(HOST_DIR)/lib
+HOST_LDFLAGS  += -L$(HOST_DIR)/lib
 
 # The macros below are taken from linux 4.11 and adapted slightly.
 # Copy more when needed.
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 8c6e34cc9f..f4a943e3ce 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -118,14 +118,6 @@ endef
 
 GLOBAL_INSTRUMENTATION_HOOKS += check_bin_arch
 
-# This hook checks that host packages that need libraries that we build
-# have a proper DT_RPATH or DT_RUNPATH tag
-define check_host_rpath
-	$(if $(filter install-host,$(2)),\
-		$(if $(filter end,$(1)),support/scripts/check-host-rpath $(3) $(HOST_DIR)))
-endef
-GLOBAL_INSTRUMENTATION_HOOKS += check_host_rpath
-
 define step_check_build_dir_one
 	if [ -d $(2) ]; then \
 		printf "%s: installs files in %s\n" $(1) $(2) >&2; \
@@ -343,6 +335,8 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 # Finalize installation
 $(BUILD_DIR)/%/.stamp_installed:
 	@$(call step_start,install)
+	$(if $(filter host-patchelf,$($(PKG)_FINAL_DEPENDENCIES)), \
+		$(TOPDIR)/support/scripts/fix-rpath host)
 	@$(call step_end,install)
 	$(Q)touch $@
 
@@ -587,6 +581,10 @@ ifneq ($(1),host-skeleton)
 $(2)_DEPENDENCIES += host-skeleton
 endif
 
+ifeq ($(filter host-patchelf host-skeleton host-tar host-xz host-lzip host-ccache host-fakedate,$(1)),)
+$(2)_DEPENDENCIES += host-patchelf
+endif
+
 ifeq ($(filter host-tar host-skeleton host-fakedate,$(1)),)
 $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
 endif
diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
index 15705a3b0d..5b2f24ed32 100755
--- a/support/scripts/fix-rpath
+++ b/support/scripts/fix-rpath
@@ -61,7 +61,7 @@ main() {
     local rootdir
     local tree="${1}"
     local find_args=( )
-    local sanitize_extra_args=( )
+    local sanitize_args=( )
 
     if ! "${PATCHELF}" --version > /dev/null 2>&1; then
 	echo "Error: can't execute patchelf utility '${PATCHELF}'"
@@ -84,12 +84,14 @@ main() {
                 find_args+=( "-path" "${HOST_DIR}""${excludepath}" "-prune" "-o" )
             done
 
-            # do not process the patchelf binary but a copy to work-around "file in use"
+            # do not process the patchelf binary, as it is ourselves
+            # (and it doesn't need a rpath as it doesn't use libraries
+            # from HOST_DIR)
             find_args+=( "-path" "${PATCHELF}" "-prune" "-o" )
-            cp "${PATCHELF}" "${PATCHELF}.__to_be_patched"
 
             # we always want $ORIGIN-based rpaths to make it relocatable.
-            sanitize_extra_args+=( "--relative-to-file" )
+            sanitize_args+=( "--set-rpath" )
+            sanitize_args+=( "\$ORIGIN/../lib" )
             ;;
 
         staging)
@@ -101,14 +103,18 @@ main() {
             done
 
             # should be like for the target tree below
-            sanitize_extra_args+=( "--no-standard-lib-dirs" )
+            sanitize_args+=( "--no-standard-lib-dirs" )
+            sanitize_args+=( "--make-rpath-relative" )
+            sanitize_args+=( "${rootdir}" )
             ;;
 
         target)
             rootdir="${TARGET_DIR}"
             # we don't want $ORIGIN-based rpaths but absolute paths without rootdir.
             # we also want to remove rpaths pointing to /lib or /usr/lib.
-            sanitize_extra_args+=( "--no-standard-lib-dirs" )
+            sanitize_args+=( "--no-standard-lib-dirs" )
+            sanitize_args+=( "--make-rpath-relative" )
+            sanitize_args+=( "${rootdir}" )
             ;;
 
         *)
@@ -120,20 +126,21 @@ main() {
     find_args+=( "-type" "f" "-print" )
 
     while read file ; do
-        # check if it's an ELF file
-        if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
+        rpath=$(${PATCHELF} --print-rpath "${file}" 2>/dev/null)
+        if test $? -eq 0; then
+            # For host binaries, if the rpath is already correct, skip
+            if test "${tree}" = "host" -a "${rpath}" = "\$ORIGIN/../lib" ; then
+                continue
+            fi
             # make files writable if necessary
             changed=$(chmod -c u+w "${file}")
             # call patchelf to sanitize the rpath
-            ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}"
+            ${PATCHELF} ${sanitize_args[@]} "${file}"
             # restore the original permission
             test "${changed}" != "" && chmod u-w "${file}"
         fi
     done < <(find "${rootdir}" ${find_args[@]})
 
-    # Restore patched patchelf utility
-    test "${tree}" = "host" && mv "${PATCHELF}.__to_be_patched" "${PATCHELF}"
-
     # ignore errors
     return 0
 }
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 14/15] Makefile: evaluate CCACHE and HOST{CC, CXX} at time of use
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (12 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 13/15] core: change host RPATH handling Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2017-12-29 17:32   ` Yann E. MORIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target Thomas Petazzoni
  2018-01-08 22:41 ` [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

As we are going to move to per-package SDK, the location of CCACHE and
therefore the definitions of HOSTCC and HOSTCXX need to be evaluated
at the time of use and not at the time of assignment. Indeed, the
value of HOST_DIR changes from one package to the other.

Therefore, we need to change from := to =.

In addition, while doing A := $(something) $(A) is possible, doing A =
$(something) $(A) is not legal. So, instead of defining HOSTCC in
terms of the current HOSTCC variable, we re-use HOSTCC_NOCCACHE
instead.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - New patch
---
 Makefile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index d8fa91120b..e05a1ec09a 100644
--- a/Makefile
+++ b/Makefile
@@ -446,11 +446,11 @@ BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(PATH)"
 TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
 
 ifeq ($(BR2_CCACHE),y)
-CCACHE := $(HOST_DIR)/bin/ccache
+CCACHE = $(HOST_DIR)/bin/ccache
 BR_CACHE_DIR ?= $(call qstrip,$(BR2_CCACHE_DIR))
 export BR_CACHE_DIR
-HOSTCC := $(CCACHE) $(HOSTCC)
-HOSTCXX := $(CCACHE) $(HOSTCXX)
+HOSTCC = $(CCACHE) $(HOSTCC_NOCCACHE)
+HOSTCXX = $(CCACHE) $(HOSTCXX_NOCCACHE)
 else
 export BR_NO_CCACHE
 endif
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (13 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 14/15] Makefile: evaluate CCACHE and HOST{CC, CXX} at time of use Thomas Petazzoni
@ 2017-12-01 20:53 ` Thomas Petazzoni
  2018-02-06 14:45   ` Matthew Weber
  2018-01-08 22:41 ` [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
  15 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-01 20:53 UTC (permalink / raw)
  To: buildroot

This commit implemnts the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target folders, that only contain their
explicit dependencies.

There are two main benefits:

 - Packages will no longer discover dependencies that they do not
   explicitly indicate in their <pkg>_DEPENDENCIES variable.

 - We can support top-level parallel build properly, because a package
   only "sees" its own host directory and target directory, isolated
   from the build of other packages that can happen in parallel.

It works as follows:

 - A new output/per-package/ folder is created, which will contain one
   sub-folder per package, and inside it, a "host" folder and a
   "target" folder:

   output/per-package/busybox/target
   output/per-package/busybox/host
   output/per-package/host-fakeroot/target
   output/per-package/host-fakeroot/host

   This output/per-package/ folder is PER_PACKAGE_DIR.

 - The global TARGET_DIR and HOST_DIR variable now automatically point
   to the per-package directory when PKG is defined. So whenever a
   package references $(HOST_DIR) or $(TARGET_DIR) in its build
   process, it effectively references the per-package host/target
   directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
   is handled as well.

 - Of course, packages have dependencies, so those dependencies must
   be installed in the per-package host and target folders. To do so,
   we simply rsync (using hard links to save space and time) the host
   and target folders of the direct dependencies of the package to the
   current package host and target folders.

   We only need to take care of direct dependencies (and not
   recursively all dependencies), because we accumulate into those
   per-package host and target folders the files installed by the
   dependencies. Note that this only works because we make the
   assumption that one package does *not* overwrite files installed by
   another package.

   This is done for "extract dependencies" at the beginning of the
   extract step, and for "normal dependencies" at the beginning of the
   configure step.

This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:

 - The filesystem image creation code cannot work.

 - We don't have a SDK to build code outside of Buildroot.

In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
Changes since v2:
 - Account for <pkg>_EXTRACT_DEPENDENCIES in the extract step of the
   package, by rsync'ing their host and target directories to the
   current package host/target directory.

Changes since v1:
 - Remove the LD_LIBRARY_PATH change since we're now longer relying on
   LD_LIBRARY_PATH to allow host programs to find their libraries.
 - Improve commit log according to Arnout suggestions
 - Remove -u option from rsync calls in the main Makefile, suggested
   by Arnout
 - Drop entirely the definitions of <pkg>_TARGET_DIR,
   <pkg>_STAGING_DIR and <pkg>_HOST_DIR, and instead make the global
   TARGET_DIR, HOST_DIR variables point to the per-package directories
   when PKG is defined. Suggested by Arnout.
---
 Makefile               | 20 ++++++++++++++++----
 package/pkg-generic.mk | 17 +++++++++++++++++
 2 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index e05a1ec09a..978177fc83 100644
--- a/Makefile
+++ b/Makefile
@@ -215,7 +215,9 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf)
 
 BUILD_DIR := $(BASE_DIR)/build
 BINARIES_DIR := $(BASE_DIR)/images
-TARGET_DIR := $(BASE_DIR)/target
+PER_PACKAGE_DIR := $(BASE_DIR)/per-package
+TARGET_DIR = $(if $(PKG),$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/target,$(BASE_DIR)/target)
+
 # initial definition so that 'make clean' works for most users, even without
 # .config. HOST_DIR will be overwritten later when .config is included.
 HOST_DIR := $(BASE_DIR)/host
@@ -237,7 +239,7 @@ LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
 # dependencies anywhere else
 #
 ################################################################################
-$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
+$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST) $(PER_PACKAGE_DIR):
 	@mkdir -p $@
 
 BR2_CONFIG = $(CONFIG_DIR)/.config
@@ -436,7 +438,7 @@ LZCAT := $(call qstrip,$(BR2_LZCAT))
 TAR_OPTIONS = $(call qstrip,$(BR2_TAR_OPTIONS)) -xf
 
 # packages compiled for the host go here
-HOST_DIR := $(call qstrip,$(BR2_HOST_DIR))
+HOST_DIR = $(if $(PKG),$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/host,$(call qstrip,$(BR2_HOST_DIR)))
 
 # Quotes are needed for spaces and all in the original PATH content.
 BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(PATH)"
@@ -663,6 +665,16 @@ $(TARGETS_ROOTFS): target-finalize
 
 .PHONY: target-finalize
 target-finalize: $(PACKAGES)
+	@$(call MESSAGE,"Creating global target directory")
+	$(foreach pkg,$(PACKAGES),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(TARGET_DIR)$(sep))
+	@$(call MESSAGE,"Creating global host directory")
+	$(foreach pkg,$(PACKAGES),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(HOST_DIR)$(sep))
 	@$(call MESSAGE,"Finalizing target directory")
 	# Check files that are touched by more than one package
 	./support/scripts/check-uniq-files -t target $(BUILD_DIR)/packages-file-list.txt
@@ -959,7 +971,7 @@ printvars:
 clean:
 	rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(BUILD_DIR) $(BASE_DIR)/staging \
-		$(LEGAL_INFO_DIR) $(GRAPHS_DIR)
+		$(LEGAL_INFO_DIR) $(GRAPHS_DIR) $(PER_PACKAGE_DIR)
 
 .PHONY: distclean
 distclean: clean
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index f4a943e3ce..f659248416 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -171,6 +171,15 @@ $(BUILD_DIR)/%/.stamp_actual_downloaded:
 $(BUILD_DIR)/%/.stamp_extracted:
 	@$(call step_start,extract)
 	@$(call MESSAGE,"Extracting")
+	@mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR)
+	$(foreach pkg,$($(PKG)_FINAL_EXTRACT_DEPENDENCIES),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(HOST_DIR)$(sep))
+	$(foreach pkg,$($(PKG)_FINAL_EXTRACT_DEPENDENCIES),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(TARGET_DIR)$(sep))
 	$(foreach hook,$($(PKG)_PRE_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -228,6 +237,14 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\
 $(BUILD_DIR)/%/.stamp_configured:
 	@$(call step_start,configure)
 	@$(call MESSAGE,"Configuring")
+	$(foreach pkg,$($(PKG)_FINAL_DEPENDENCIES),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/host/ \
+		$(HOST_DIR)$(sep))
+	$(foreach pkg,$($(PKG)_FINAL_DEPENDENCIES),\
+		rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(PER_PACKAGE_DIR)/$(pkg)/target/ \
+		$(TARGET_DIR)$(sep))
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
-- 
2.13.6

^ permalink raw reply related	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an extract dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an " Thomas Petazzoni
@ 2017-12-02 15:06   ` Yann E. MORIN
  2017-12-02 20:19     ` Thomas Petazzoni
  0 siblings, 1 reply; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-02 15:06 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-tar dependency handling from
> DEPENDENCY_HOST_PREREQ to an extract dependency.
> 
> To achieve that, check-host-tar.mk fills in the

s/$/ variable/

> BR2_TAR_HOST_DEPENDENCY with host-tar if building a host-tar is
> needed. The name BR2_TAR_HOST_DEPENDENCY has been chosen because it
> matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> check-host-cmake.mk.
> 
> The BR2_TAR_HOST_DEPENDENCY is added to all packages, except host-tar
> itself (obviously) and host-skeleton, because we depend on
> host-skeleton to install host-tar properly in HOST_DIR.

As we discussed on IRC, there is also the case for the tar filesystem,
which may require host-tar.

Granted, it is very very unlikely that we end up with no package that
require host-tar, but still...

> In addition, we modify tar.mk to explicitly build host-tar without
> ccache: since ccache source code is available as a tarball, ccache
> will obviously depend on host-tar if the system tar is insufficient.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> Changes since v2:
>  - New patch
> ---
>  package/pkg-generic.mk                 | 4 ++++
>  package/tar/tar.mk                     | 6 ++++++
>  support/dependencies/check-host-tar.mk | 2 +-
>  3 files changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 824c34bec9..86dd2561d4 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -587,6 +587,10 @@ ifneq ($(1),host-skeleton)
>  $(2)_DEPENDENCIES += host-skeleton
>  endif
>  
> +ifeq ($(filter host-tar host-skeleton,$(1)),)
> +$(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
> +endif

So, even packages that are available as a zip file will get the
dependency on host-tar?

>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/package/tar/tar.mk b/package/tar/tar.mk
> index cb2ddc0eca..813eabed14 100644
> --- a/package/tar/tar.mk
> +++ b/package/tar/tar.mk
> @@ -44,4 +44,10 @@ define HOST_TAR_EXTRACT_CMDS
>  	mv $(@D)/tar-$(TAR_VERSION)/* $(@D)
>  	rmdir $(@D)/tar-$(TAR_VERSION)
>  endef
> +
> +# we are built before ccache
> +HOST_TAR_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  $(eval $(host-autotools-package))
> diff --git a/support/dependencies/check-host-tar.mk b/support/dependencies/check-host-tar.mk
> index ad0b32e277..d07f727c4c 100644
> --- a/support/dependencies/check-host-tar.mk
> +++ b/support/dependencies/check-host-tar.mk
> @@ -1,6 +1,6 @@
>  TAR ?= tar
>  
>  ifeq (,$(call suitable-host-package,tar,$(TAR)))
> -DEPENDENCIES_HOST_PREREQ += host-tar
>  TAR = $(HOST_DIR)/bin/tar
> +BR2_TAR_HOST_DEPENDENCY += host-tar

Why assign-append here, instead of simply assign?

Regards,
Yann E. MORIN.

>  endif
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz as an extract dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz " Thomas Petazzoni
@ 2017-12-02 15:08   ` Yann E. MORIN
  2017-12-02 20:16     ` Thomas Petazzoni
  2017-12-29 15:36   ` Yann E. MORIN
  1 sibling, 1 reply; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-02 15:08 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-xz dependency handling from
> DEPENDENCY_HOST_PREREQ to an extract dependency.
> 
> To achieve that, check-host-xz.mk fills in the
> BR2_XZ_HOST_DEPENDENCY with host-tar if building a host-tar is
> needed. The name BR2_XZ_HOST_DEPENDENCY has been chosen because it
> matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> check-host-cmake.mk.
> 
> The BR2_XZ_HOST_DEPENDENCY is added to all packages, except:
> 
>  - host-xz, because we would otherwise depend on ourself.
> 
>  - host-tar, because xz itself is delivered as a tarball, so we need
>    to have host-xz depend on host-tar, and not host-tar depend on
>    host-xz
> 
>  - host-skeleton, because we need to have host-xz depend on
>    host-skeleton, and not the opposite.
> 
> In addition, we modify xz.mk to explicitly build host-xz without
> ccache. We generally took the approach of building host-ccache *after*
> all the extractors have been built.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> Changes since v2:
>  - New patch
> ---
>  package/pkg-generic.mk                   | 4 ++++
>  package/xz/xz.mk                         | 5 +++++
>  support/dependencies/check-host-xzcat.mk | 2 +-
>  3 files changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 86dd2561d4..dfd4719d29 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -591,6 +591,10 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
> +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> +endif

So, all packages will now depend on this, even those that are not
compressed with xz?

Currently, filesystems that want to compress with xz all depend on
host-xz unconditioanlly. Do we want to change that as well, so they only
depend on it if needed?

Regards,
Yann E. MORIN.

>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/package/xz/xz.mk b/package/xz/xz.mk
> index e8116f24ae..bcdac13ee3 100644
> --- a/package/xz/xz.mk
> +++ b/package/xz/xz.mk
> @@ -18,5 +18,10 @@ else
>  XZ_CONF_OPTS = --disable-threads
>  endif
>  
> +# we are built before ccache
> +HOST_XZ_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> index 9be75c7311..e5d72f95dd 100644
> --- a/support/dependencies/check-host-xzcat.mk
> +++ b/support/dependencies/check-host-xzcat.mk
> @@ -2,7 +2,7 @@
>  # If it is not present, build our own host-xzcat
>  
>  ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
> -DEPENDENCIES_HOST_PREREQ += host-xz
> +BR2_XZCAT_HOST_DEPENDENCY = host-xz
>  EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
>  XZCAT = $(HOST_DIR)/bin/xzcat
>  endif
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip as an extract dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip " Thomas Petazzoni
@ 2017-12-02 15:12   ` Yann E. MORIN
  2017-12-02 20:13     ` Thomas Petazzoni
  2017-12-29 15:36   ` Yann E. MORIN
  1 sibling, 1 reply; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-02 15:12 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-xz dependency handling from
> DEPENDENCY_HOST_PREREQ to an extract dependency.
> 
> To achieve that, check-host-xz.mk fills in the
> BR2_LZIP_HOST_DEPENDENCY with host-tar if building a host-tar is
> needed. The name BR2_LZIP_HOST_DEPENDENCY has been chosen because it
> matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> check-host-cmake.mk.
> 
> The BR2_LZIP_HOST_DEPENDENCY is added to all packages, except:

So even when no package compresed with lzip is enabled?

That was a gripe of yours, that host-lzip was always built even when not
needed...

Regards,
Yann E. MORIN.

>  - host-lzip, because we would otherwise depend on ourself.
> 
>  - host-tar, because lzip itself is delivered as a tarball, so we need
>    to have host-lzip depend on host-tar, and not host-tar depend on
>    host-lzip
> 
>  - host-skeleton, because we need to have host-lzip depend on
>    host-skeleton, and not the opposite.
> 
> We also mutually exclude host-lzip and host-xz from dependending on
> each other, to avoid a circular dependency.
> 
> In addition, we modify lzip.mk to explicitly build host-lzip without
> ccache. We generally took the approach of building host-ccache *after*
> all the extractors have been built.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> Changes since v2:
>  - New patch
> ---
>  package/lzip/lzip.mk                    | 2 +-
>  package/pkg-generic.mk                  | 6 +++++-
>  support/dependencies/check-host-lzip.mk | 2 +-
>  3 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/package/lzip/lzip.mk b/package/lzip/lzip.mk
> index b7ba5dd21d..a16a22214c 100644
> --- a/package/lzip/lzip.mk
> +++ b/package/lzip/lzip.mk
> @@ -16,7 +16,7 @@ endef
>  
>  define HOST_LZIP_CONFIGURE_CMDS
>  	(cd $(@D); $(HOST_MAKE_ENV) ./configure --prefix=$(HOST_DIR) \
> -		$(HOST_CONFIGURE_OPTS) )
> +		$(HOST_CONFIGURE_OPTS) CC="$(HOSTCC_NOCCACHE)" CXX="$(HOSTCXX_NOCCACHE)")
>  endef
>  
>  define LZIP_BUILD_CMDS
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index dfd4719d29..e623afe7bc 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -591,10 +591,14 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
> -ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
>  endif
>  
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
> +$(2)_EXTRACT_DEPENDENCIES += $(BR2_LZIP_HOST_DEPENDENCY)
> +endif
> +
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
> index 00cdd0a236..cdd784058c 100644
> --- a/support/dependencies/check-host-lzip.mk
> +++ b/support/dependencies/check-host-lzip.mk
> @@ -1,5 +1,5 @@
>  ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
> -DEPENDENCIES_HOST_PREREQ += host-lzip
> +BR2_LZIP_HOST_DEPENDENCY = host-lzip
>  EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
>  LZCAT = $(HOST_DIR)/bin/lzip -d -c
>  endif
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip as an extract dependency
  2017-12-02 15:12   ` Yann E. MORIN
@ 2017-12-02 20:13     ` Thomas Petazzoni
  0 siblings, 0 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-02 20:13 UTC (permalink / raw)
  To: buildroot

Hello,

On Sat, 2 Dec 2017 16:12:24 +0100, Yann E. MORIN wrote:

> > To achieve that, check-host-xz.mk fills in the
> > BR2_LZIP_HOST_DEPENDENCY with host-tar if building a host-tar is
> > needed. The name BR2_LZIP_HOST_DEPENDENCY has been chosen because it
> > matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> > check-host-cmake.mk.
> > 
> > The BR2_LZIP_HOST_DEPENDENCY is added to all packages, except:  
> 
> So even when no package compresed with lzip is enabled?
> 
> That was a gripe of yours, that host-lzip was always built even when not
> needed...

This problem already exists today, and my series does not aim at fixing
it. Fixing it is a separate problem, that I didn't want to solve in the
same series (which as you can see is already growing significantly in
terms of number of patches).

So, it is intentional that I do not solve this specific problem as part
of this series.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz as an extract dependency
  2017-12-02 15:08   ` Yann E. MORIN
@ 2017-12-02 20:16     ` Thomas Petazzoni
  2017-12-03  8:32       ` Yann E. MORIN
  0 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-02 20:16 UTC (permalink / raw)
  To: buildroot

Hello,

On Sat, 2 Dec 2017 16:08:43 +0100, Yann E. MORIN wrote:

> > +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> > +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> > +endif  
> 
> So, all packages will now depend on this, even those that are not
> compressed with xz?

Yes, if there is no xzcat available on the system. This is exactly what
the current behavior is with DEPENDENCIES_HOST_PREREQ. My series does
not intend to solve this problem, which is a separate one.

I do believe however that by moving those dependencies (host-xz and
host-lzip) as proper package dependencies, it will be easier to have
follow-up patches that add the dependency only if lzip or xz are
needed. But again, my series does not intend to fix that, it's a
separate problem.

> Currently, filesystems that want to compress with xz all depend on
> host-xz unconditioanlly. Do we want to change that as well, so they only
> depend on it if needed?

I don't understand this part, because filesystems do not depend on
host-xz unconditionally. They only depend on it if xz compression of
the filesystem image is requested. From fs/common.mk:

ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
ROOTFS_$(2)_DEPENDENCIES += host-xz
ROOTFS_$(2)_COMPRESS_EXT = .xz
ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
endif

Best regards,

Thomas Petazzoni
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an extract dependency
  2017-12-02 15:06   ` Yann E. MORIN
@ 2017-12-02 20:19     ` Thomas Petazzoni
  2017-12-29 15:34       ` Yann E. MORIN
  0 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-02 20:19 UTC (permalink / raw)
  To: buildroot

Hello,

On Sat, 2 Dec 2017 16:06:06 +0100, Yann E. MORIN wrote:

> > To achieve that, check-host-tar.mk fills in the  
> 
> s/$/ variable/

ACK, will fix.

> > The BR2_TAR_HOST_DEPENDENCY is added to all packages, except host-tar
> > itself (obviously) and host-skeleton, because we depend on
> > host-skeleton to install host-tar properly in HOST_DIR.  
> 
> As we discussed on IRC, there is also the case for the tar filesystem,
> which may require host-tar.
> 
> Granted, it is very very unlikely that we end up with no package that
> require host-tar, but still...

Yes, I agree, the tar filesystem should depend on
$(BR2_TAR_HOST_DEPENDENCY) as well. I'll fix that.

> > +ifeq ($(filter host-tar host-skeleton,$(1)),)
> > +$(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
> > +endif  
> 
> So, even packages that are available as a zip file will get the
> dependency on host-tar?

Yes. Like host-xz/host-lzip, my series does not attempt to switch to
more fine-grained dependencies, it only intends to move to proper
package dependencies. Using more fine-grained dependencies is left for
future work.

> >  ifeq (,$(call suitable-host-package,tar,$(TAR)))
> > -DEPENDENCIES_HOST_PREREQ += host-tar
> >  TAR = $(HOST_DIR)/bin/tar
> > +BR2_TAR_HOST_DEPENDENCY += host-tar  
> 
> Why assign-append here, instead of simply assign?

Good point, will fix.

Thanks for the review!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz as an extract dependency
  2017-12-02 20:16     ` Thomas Petazzoni
@ 2017-12-03  8:32       ` Yann E. MORIN
  2017-12-03  9:29         ` Thomas Petazzoni
  0 siblings, 1 reply; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-03  8:32 UTC (permalink / raw)
  To: buildroot

On 2017-12-02 21:16 +0100, Thomas Petazzoni spake thusly:
> Hello,
> 
> On Sat, 2 Dec 2017 16:08:43 +0100, Yann E. MORIN wrote:
> 
> > > +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> > > +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> > > +endif  
> > 
> > So, all packages will now depend on this, even those that are not
> > compressed with xz?
> 
> Yes, if there is no xzcat available on the system. This is exactly what
> the current behavior is with DEPENDENCIES_HOST_PREREQ. My series does
> not intend to solve this problem, which is a separate one.

I somehow interpreted your patches in my head as "only build them when
needed". ;-)

> I do believe however that by moving those dependencies (host-xz and
> host-lzip) as proper package dependencies, it will be easier to have
> follow-up patches that add the dependency only if lzip or xz are
> needed. But again, my series does not intend to fix that, it's a
> separate problem.

OK, agreed.

> > Currently, filesystems that want to compress with xz all depend on
> > host-xz unconditioanlly. Do we want to change that as well, so they only
> > depend on it if needed?
> 
> I don't understand this part, because filesystems do not depend on
> host-xz unconditionally. They only depend on it if xz compression of
> the filesystem image is requested. From fs/common.mk:
> 
> ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> ROOTFS_$(2)_DEPENDENCIES += host-xz
> ROOTFS_$(2)_COMPRESS_EXT = .xz
> ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
> endif

Well, first, let's assume that when the host has a suitable xzcat, it
also has the xz command as well; I think it's not too far-fetched an
assumption.

Then, we have BR2_XZCAT_HOST_DEPENDENCY empty, because we do have a
suitable xzcat, and thus a suitable xz. Yet, when the user requires
their filesystem to be xz-compressed, we still forcefully depend on
host-xz, even though we do have a suitable xz command.

What I mean is that, probably, we will want to make it:

    ROOTFS_$(2)_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)

Regards,
Yann E. MORIN.

> Best regards,
> 
> Thomas Petazzoni
> -- 
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz as an extract dependency
  2017-12-03  8:32       ` Yann E. MORIN
@ 2017-12-03  9:29         ` Thomas Petazzoni
  2017-12-03  9:34           ` Yann E. MORIN
  0 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-03  9:29 UTC (permalink / raw)
  To: buildroot

Hello,

On Sun, 3 Dec 2017 09:32:56 +0100, Yann E. MORIN wrote:

> > Yes, if there is no xzcat available on the system. This is exactly what
> > the current behavior is with DEPENDENCIES_HOST_PREREQ. My series does
> > not intend to solve this problem, which is a separate one.  
> 
> I somehow interpreted your patches in my head as "only build them when
> needed". ;-)

Nope, I'm not fixing that (separate) problem. I still want to fix this
problem as well, but not as part of the per-package SDK stuff.

> > > Currently, filesystems that want to compress with xz all depend on
> > > host-xz unconditioanlly. Do we want to change that as well, so they only
> > > depend on it if needed?  
> > 
> > I don't understand this part, because filesystems do not depend on
> > host-xz unconditionally. They only depend on it if xz compression of
> > the filesystem image is requested. From fs/common.mk:
> > 
> > ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> > ROOTFS_$(2)_DEPENDENCIES += host-xz
> > ROOTFS_$(2)_COMPRESS_EXT = .xz
> > ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
> > endif  
> 
> Well, first, let's assume that when the host has a suitable xzcat, it
> also has the xz command as well; I think it's not too far-fetched an
> assumption.
> 
> Then, we have BR2_XZCAT_HOST_DEPENDENCY empty, because we do have a
> suitable xzcat, and thus a suitable xz. Yet, when the user requires
> their filesystem to be xz-compressed, we still forcefully depend on
> host-xz, even though we do have a suitable xz command.
> 
> What I mean is that, probably, we will want to make it:
> 
>     ROOTFS_$(2)_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)

Yes, I thought about this too. But again, it's a separate issue: the
filesystem code already depends on host-xz, regardless of whether xz is
already available on the system.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz as an extract dependency
  2017-12-03  9:29         ` Thomas Petazzoni
@ 2017-12-03  9:34           ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-03  9:34 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-03 10:29 +0100, Thomas Petazzoni spake thusly:
[--SNIP--]
> > > > Currently, filesystems that want to compress with xz all depend on
> > > > host-xz unconditioanlly. Do we want to change that as well, so they only
> > > > depend on it if needed?  
> > > 
> > > I don't understand this part, because filesystems do not depend on
> > > host-xz unconditionally. They only depend on it if xz compression of
> > > the filesystem image is requested. From fs/common.mk:
> > > 
> > > ifeq ($$(BR2_TARGET_ROOTFS_$(2)_XZ),y)
> > > ROOTFS_$(2)_DEPENDENCIES += host-xz
> > > ROOTFS_$(2)_COMPRESS_EXT = .xz
> > > ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
> > > endif  
[--SNIP--]
> > What I mean is that, probably, we will want to make it:
> > 
> >     ROOTFS_$(2)_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)
> 
> Yes, I thought about this too. But again, it's a separate issue: the
> filesystem code already depends on host-xz, regardless of whether xz is
> already available on the system.

Yes, sure. I was just merely saying that when you eventually make those
dependencies really when-needed, you should also have a look at the
filesystem stuff. ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 03/15] Makefile, skeleton: move the host skeleton logic to host-skeleton package
  2017-12-01 20:53 ` [Buildroot] [RFCv3 03/15] Makefile, skeleton: move the host skeleton logic to host-skeleton package Thomas Petazzoni
@ 2017-12-03 22:22   ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-03 22:22 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> As part of the per-package SDK work, we want to avoid having logic
> that installs files to the global HOST_DIR, and instead do it inside
> packages. One thing that gets installed to the global HOST_DIR is the
> minimal "skeleton" that we create in host:
> 
>  - the "usr" symbolic link for backward compatibility
> 
>  - the "lib" directory, and its lib64 or lib32 symbolic links
> 
> This commit moves this logic to a new host-skeleton package, and makes
> all packages (except itself) depend on it. We also make sure that this
> host-skeleton package doesn't depend on host-patchelf, because
> host-patchelf depends on host-skeleton.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

With for a small nit, below...

[--SNIP--]
> diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
> index d380f41649..a32bacd0b3 100644
> --- a/package/skeleton/skeleton.mk
> +++ b/package/skeleton/skeleton.mk
> @@ -11,4 +11,16 @@
>  SKELETON_ADD_TOOLCHAIN_DEPENDENCY = NO
>  SKELETON_ADD_SKELETON_DEPENDENCY = NO
>  
> +# We create a compatibility symlink in case a post-build script still
> +# uses $(HOST_DIR)/usr
> +define HOST_SKELETON_INSTALL_CMDS
> +	@ln -snf . $(HOST_DIR)/usr
> +	@mkdir -p $(HOST_DIR)/lib
> +	@case $(HOSTARCH) in \
> +		(*64) ln -snf lib $(HOST_DIR)/lib64;; \
> +		(*)   ln -snf lib $(HOST_DIR)/lib32;; \
> +	esac

Maybe take the opportunity to replace '@' with '$(Q)'?

That aside, my Reviewed-by tag stands.

> +endef
> +
>  $(eval $(virtual-package))
> +$(eval $(host-generic-package))

Aham that's certainly bizarre that the target variant is a virtual
package but the host a real one. :-)

But nothing wrong with that.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package
  2017-12-01 20:53 ` [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package Thomas Petazzoni
@ 2017-12-03 22:34   ` Yann E. MORIN
  2017-12-03 22:55     ` Thomas Petazzoni
  0 siblings, 1 reply; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-03 22:34 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> Currently, the toolchainfile.cmake and Buildroot.cmake files are
> installed outside of any package, just triggered by the toolchain
> target.
> 
> As part of the per-package SDK effort, we are trying to avoid anything
> that installs to the global $(HOST_DIR), and this is one of the
> remaining files installed in $(HOST_DIR) outside of any package. We
> fix this by installing such files as part of the toolchain package
> post-install staging hooks.
> 
> Yes, a post-install staging hook to install things to $(HOST_DIR) is a
> bit weird, but the toolchain infrastructure is made of target packages
> only, and they all install a lot of stuff to $(HOST_DIR) already.

What I am more concerned about, is that the cmake package is registering
a hook for another package, toolchain.

Have you tried to run check-package on that?

Regards,
Yann E. MORIN.

> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> Changes since v2:
>  - None
> Changes since v1:
>  - New patch
> ---
>  package/pkg-cmake.mk             | 12 +++++++-----
>  toolchain/toolchain/toolchain.mk |  3 ---
>  2 files changed, 7 insertions(+), 8 deletions(-)
> 
> diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
> index 6739704e3c..6d4f948d07 100644
> --- a/package/pkg-cmake.mk
> +++ b/package/pkg-cmake.mk
> @@ -244,8 +244,8 @@ endif
>  # based on the toolchainfile.cmake file's location: $(HOST_DIR)/share/buildroot
>  # In all the other variables, HOST_DIR will be replaced by RELOCATED_HOST_DIR,
>  # so we have to strip "$(HOST_DIR)/" from the paths that contain it.
> -$(HOST_DIR)/share/buildroot/toolchainfile.cmake:
> -	@mkdir -p $(@D)
> +define TOOLCHAIN_CMAKE_INSTALL_FILES
> +	@mkdir -p $(HOST_DIR)/share/buildroot
>  	sed \
>  		-e 's#@@STAGING_SUBDIR@@#$(call qstrip,$(STAGING_SUBDIR))#' \
>  		-e 's#@@TARGET_CFLAGS@@#$(call qstrip,$(TARGET_CFLAGS))#' \
> @@ -259,7 +259,9 @@ $(HOST_DIR)/share/buildroot/toolchainfile.cmake:
>  		-e 's#@@TOOLCHAIN_HAS_FORTRAN@@#$(if $(BR2_TOOLCHAIN_HAS_FORTRAN),1,0)#' \
>  		-e 's#@@CMAKE_BUILD_TYPE@@#$(if $(BR2_ENABLE_DEBUG),Debug,Release)#' \
>  		$(TOPDIR)/support/misc/toolchainfile.cmake.in \
> -		> $@
> +		> $(HOST_DIR)/share/buildroot/toolchainfile.cmake
> +	$(Q)$(INSTALL) -D -m 0644 support/misc/Buildroot.cmake \
> +		$(HOST_DIR)/share/buildroot/Platform/Buildroot.cmake
> +endef
>  
> -$(HOST_DIR)/share/buildroot/Platform/Buildroot.cmake:
> -	$(Q)$(INSTALL) -D -m 0644 support/misc/Buildroot.cmake $(@)
> +TOOLCHAIN_POST_INSTALL_STAGING_HOOKS += TOOLCHAIN_CMAKE_INSTALL_FILES
> diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
> index b55b0c712c..283e0a74ee 100644
> --- a/toolchain/toolchain/toolchain.mk
> +++ b/toolchain/toolchain/toolchain.mk
> @@ -38,6 +38,3 @@ TOOLCHAIN_INSTALL_STAGING = YES
>  endif
>  
>  $(eval $(virtual-package))
> -
> -toolchain: $(HOST_DIR)/share/buildroot/toolchainfile.cmake
> -toolchain: $(HOST_DIR)/share/buildroot/Platform/Buildroot.cmake
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package
  2017-12-03 22:34   ` Yann E. MORIN
@ 2017-12-03 22:55     ` Thomas Petazzoni
  2017-12-29 15:27       ` Yann E. MORIN
  0 siblings, 1 reply; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-03 22:55 UTC (permalink / raw)
  To: buildroot

Hello,

On Sun, 3 Dec 2017 23:34:54 +0100, Yann E. MORIN wrote:

> On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> > Currently, the toolchainfile.cmake and Buildroot.cmake files are
> > installed outside of any package, just triggered by the toolchain
> > target.
> > 
> > As part of the per-package SDK effort, we are trying to avoid anything
> > that installs to the global $(HOST_DIR), and this is one of the
> > remaining files installed in $(HOST_DIR) outside of any package. We
> > fix this by installing such files as part of the toolchain package
> > post-install staging hooks.
> > 
> > Yes, a post-install staging hook to install things to $(HOST_DIR) is a
> > bit weird, but the toolchain infrastructure is made of target packages
> > only, and they all install a lot of stuff to $(HOST_DIR) already.  
> 
> What I am more concerned about, is that the cmake package is registering
> a hook for another package, toolchain.

Do we want to introduce a toolchain-cmake package, that all CMake
packages would depend on, just for the sake of installing those files ?

If we would do this, then a normal toolchain build (i.e without
building any CMake package) would no longer generate the
toolchainfile.cmake file, which is quite useful for SDKs. People could
indeed enable that package explicitly when they are building a SDK.

But all in all, I'm not sure it's really worth the hassle. But I'm open
to comments on that.

Or perhaps you had other suggestions in mind?

> Have you tried to run check-package on that?

Nope, I did not, but it does not complain:

$ ./utils/check-package package/pkg-cmake.mk 
0 lines processed
0 warnings generated

Thanks for the review!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path
  2017-12-01 20:53 ` [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path Thomas Petazzoni
@ 2017-12-29 15:22   ` Yann E. MORIN
  2017-12-31 17:24   ` Thomas Petazzoni
  1 sibling, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:22 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> The pkg-config wrapper script is currently generated with absolute
> paths to $(STAGING_DIR). However, this will not work properly with
> per-package SDK, and each package will be built with a different
> STAGING_DIR value.
> 
> In order to fix this, we adjust how the pkg-config wrapper script is
> generated, so that it uses a relative path to itself: the sysroot (i.e
> STAGING_DIR) is always located in $(path of
> pkg-config)/../$(STAGING_SUBDIR).
> 
> This change is independent from the per-package SDK work, and could be
> applied independently from it.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - None
> Changes since v1:
>  - As suggested by Arnout, simplify the generation of the pkg-config
>    script by doing only a replacement on @STAGING_SUBDIR@, the rest
>    being encoded inside the pkg-config script.
> ---
>  package/pkgconf/pkg-config.in | 5 ++++-
>  package/pkgconf/pkgconf.mk    | 3 +--
>  2 files changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/package/pkgconf/pkg-config.in b/package/pkgconf/pkg-config.in
> index 4dec48789a..b9ce0935cc 100644
> --- a/package/pkgconf/pkg-config.in
> +++ b/package/pkgconf/pkg-config.in
> @@ -1,2 +1,5 @@
>  #!/bin/sh
> -PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:- at PKG_CONFIG_LIBDIR@} PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:- at STAGING_DIR@} $(dirname $0)/pkgconf @STATIC@ $@
> +PKGCONFDIR=$(dirname $0)
> +DEFAULT_PKG_CONFIG_LIBDIR=${PKGCONFDIR}/../@STAGING_SUBDIR@/usr/lib/pkgconfig:${PKGCONFDIR}/../@STAGING_SUBDIR@/usr/share/pkgconfig
> +DEFAULT_PKG_CONFIG_SYSROOT_DIR=${PKGCONFDIR}/../@STAGING_SUBDIR@
> +PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:-${DEFAULT_PKG_CONFIG_LIBDIR}} PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:-${DEFAULT_PKG_CONFIG_SYSROOT_DIR}} ${PKGCONFDIR}/pkgconf @STATIC@ $@
> diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
> index cc190d26da..00b2d017ee 100644
> --- a/package/pkgconf/pkgconf.mk
> +++ b/package/pkgconf/pkgconf.mk
> @@ -19,8 +19,7 @@ endef
>  define HOST_PKGCONF_INSTALL_WRAPPER
>  	$(INSTALL) -m 0755 -D package/pkgconf/pkg-config.in \
>  		$(HOST_DIR)/bin/pkg-config
> -	$(SED) 's, at PKG_CONFIG_LIBDIR@,$(STAGING_DIR)/usr/lib/pkgconfig:$(STAGING_DIR)/usr/share/pkgconfig,' \
> -		-e 's, at STAGING_DIR@,$(STAGING_DIR),' \
> +	$(SED) 's, at STAGING_SUBDIR@,$(STAGING_SUBDIR),g' \
>  		$(HOST_DIR)/bin/pkg-config
>  endef
>  
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN Thomas Petazzoni
@ 2017-12-29 15:22   ` Yann E. MORIN
  2017-12-31 17:24   ` Thomas Petazzoni
  1 sibling, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:22 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> The upcoming per-package SDK functionality is heavily based on the
> fact that HOST_DIR, STAGING_DIR and TARGET_DIR are evaluated during
> the configure/build/install steps of the packages. Therefore, any
> evaluation-during-assignment using := is going to cause problems, and
> need to be turned into evaluation-during-use using =.
> 
> This patch fix up one such instance in the external toolchain code.
> 
> This change is independent from the per-package SDK functionality, and
> could be applied separately.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - None
> Changes since v1:
>  - Added Arnout Reviewed-by.
> ---
>  toolchain/toolchain-external/pkg-toolchain-external.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk
> index dc0588c536..b9ad1720a1 100644
> --- a/toolchain/toolchain-external/pkg-toolchain-external.mk
> +++ b/toolchain/toolchain-external/pkg-toolchain-external.mk
> @@ -77,7 +77,7 @@ ifneq ($(TOOLCHAIN_EXTERNAL_PREFIX),)
>  TOOLCHAIN_EXTERNAL_BIN := $(dir $(shell which $(TOOLCHAIN_EXTERNAL_PREFIX)-gcc))
>  endif
>  else
> -TOOLCHAIN_EXTERNAL_BIN := $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/bin
> +TOOLCHAIN_EXTERNAL_BIN = $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/bin
>  endif
>  
>  # If this is a buildroot toolchain, it already has a wrapper which we want to
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package
  2017-12-03 22:55     ` Thomas Petazzoni
@ 2017-12-29 15:27       ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:27 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-03 23:55 +0100, Thomas Petazzoni spake thusly:
> On Sun, 3 Dec 2017 23:34:54 +0100, Yann E. MORIN wrote:
> > On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> > > Currently, the toolchainfile.cmake and Buildroot.cmake files are
> > > installed outside of any package, just triggered by the toolchain
> > > target.
> > > 
> > > As part of the per-package SDK effort, we are trying to avoid anything
> > > that installs to the global $(HOST_DIR), and this is one of the
> > > remaining files installed in $(HOST_DIR) outside of any package. We
> > > fix this by installing such files as part of the toolchain package
> > > post-install staging hooks.
> > > 
> > > Yes, a post-install staging hook to install things to $(HOST_DIR) is a
> > > bit weird, but the toolchain infrastructure is made of target packages
> > > only, and they all install a lot of stuff to $(HOST_DIR) already.  
> > 
> > What I am more concerned about, is that the cmake package is registering
> > a hook for another package, toolchain.
> 
> Do we want to introduce a toolchain-cmake package, that all CMake
> packages would depend on, just for the sake of installing those files ?
> 
> If we would do this, then a normal toolchain build (i.e without
> building any CMake package) would no longer generate the
> toolchainfile.cmake file, which is quite useful for SDKs. People could
> indeed enable that package explicitly when they are building a SDK.
> 
> But all in all, I'm not sure it's really worth the hassle. But I'm open
> to comments on that.
> 
> Or perhaps you had other suggestions in mind?

No, I don't have a better solution. And in fact, it is not a package
that is registering a hook for another one; it is the cmake infra that
registers a hook for the toolchain package.

And I'm pretty fine with this, in fact.

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

> > Have you tried to run check-package on that?
> 
> Nope, I did not, but it does not complain:
> 
> $ ./utils/check-package package/pkg-cmake.mk 
> 0 lines processed
> 0 warnings generated

Right, because this is not a package, but a package infra.

Thanks! :-)

Regards,
Yann E. MORIN.

> Thanks for the review!
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step
  2017-12-01 20:53 ` [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step Thomas Petazzoni
@ 2017-12-29 15:30   ` Yann E. MORIN
  2018-02-22 21:48   ` Matthew Weber
  1 sibling, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:30 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> The installation to target, staging and images as well as the host
> installation are all done in 4 different make targets. We don't have
> any place currently to run something at the end of a package
> installation, i.e once all of target/staging/images (for target) or
> host (for host) installation have completed.
> 
> We will need such a step to fix the RPATH of binaries in the host
> directory. Indeed, while normally only host packages should install
> things in $(HOST_DIR), in practice a number of target packages (ex:
> qt4, qt5base) do install things in $(HOST_DIR).
> 
> Therefore, we want to run our RPATH fixing logic at the end of all
> installation steps. To achieve this, this commit introduces a new make
> target, .stamp_installed, which depends on
> .stamp_{target,staging,images}_installed for target packages and
> .stamp_host_installed for host packages.
> 
> This new step currently doesn't do anything, but the actual logic is
> added in a follow-up commit.
> 
> This change in fact makes the overall step sequencing more logical:
> for all steps the dependencies were done on the stamp file targets,
> except for the install step where we were using phony targets to make
> $(1)-install depends on $(1)-install-{target,staging,images,host}. So
> now, the phony targets $(1)-install-{target,staging,images,host} are
> only human-readable targets that are shortcuts to the corresponding
> stamp file targets, and not usded anymore for the step
> sequencing. This makes them more consistent with the other phony
> $(1)-{build,configure,patch,...} phony targets.
> 
> It also makes $(1)-install-host more consistent with other phony
> targets: $(1)-install-target only existed if the package had
> <pkg>_INSTALL_TARGET = YES, but $(1)-install-host existed
> unconditionally. Now it only exists if it's a host package.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - None
> Changes since v1:
>  - New patch.
> ---
>  package/pkg-generic.mk | 44 +++++++++++++++++++++++++++-----------------
>  1 file changed, 27 insertions(+), 17 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 0019814276..2b821e9bdf 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -340,6 +340,12 @@ $(BUILD_DIR)/%/.stamp_target_installed:
>  	@$(call step_end,install-target)
>  	$(Q)touch $@
>  
> +# Finalize installation
> +$(BUILD_DIR)/%/.stamp_installed:
> +	@$(call step_start,install)
> +	@$(call step_end,install)
> +	$(Q)touch $@
> +
>  # Remove package sources
>  $(BUILD_DIR)/%/.stamp_dircleaned:
>  	rm -Rf $(@D)
> @@ -595,6 +601,7 @@ $(2)_TARGET_INSTALL_TARGET =	$$($(2)_DIR)/.stamp_target_installed
>  $(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
>  $(2)_TARGET_INSTALL_IMAGES =	$$($(2)_DIR)/.stamp_images_installed
>  $(2)_TARGET_INSTALL_HOST =      $$($(2)_DIR)/.stamp_host_installed
> +$(2)_TARGET_INSTALL =      	$$($(2)_DIR)/.stamp_installed
>  $(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
>  $(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
>  $(2)_TARGET_RSYNC =	        $$($(2)_DIR)/.stamp_rsynced
> @@ -642,38 +649,40 @@ $(2)_ROOTFS_POST_CMD_HOOKS      ?=
>  # human-friendly targets and target sequencing
>  $(1):			$(1)-install
>  
> +$(1)-install:			$$($(2)_TARGET_INSTALL)
> +
>  ifeq ($$($(2)_TYPE),host)
> -$(1)-install:	        $(1)-install-host
> -else
> -$(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
> -endif
> +
> +$(1)-install-host:		$$($(2)_TARGET_INSTALL_HOST)
> +$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_HOST)
> +
> +else # host vs. target
>  
>  ifeq ($$($(2)_INSTALL_TARGET),YES)
>  $(1)-install-target:		$$($(2)_TARGET_INSTALL_TARGET)
> -$$($(2)_TARGET_INSTALL_TARGET):	$$($(2)_TARGET_BUILD)
> -else
> -$(1)-install-target:
> +$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_TARGET)
>  endif
>  
>  ifeq ($$($(2)_INSTALL_STAGING),YES)
> -$(1)-install-staging:			$$($(2)_TARGET_INSTALL_STAGING)
> -$$($(2)_TARGET_INSTALL_STAGING):	$$($(2)_TARGET_BUILD)
> -# Some packages use install-staging stuff for install-target
> -$$($(2)_TARGET_INSTALL_TARGET):		$$($(2)_TARGET_INSTALL_STAGING)
> -else
> -$(1)-install-staging:
> +$(1)-install-staging:		$$($(2)_TARGET_INSTALL_STAGING)
> +$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_STAGING)
>  endif
>  
>  ifeq ($$($(2)_INSTALL_IMAGES),YES)
>  $(1)-install-images:		$$($(2)_TARGET_INSTALL_IMAGES)
> -$$($(2)_TARGET_INSTALL_IMAGES):	$$($(2)_TARGET_BUILD)
> -else
> -$(1)-install-images:
> +$$($(2)_TARGET_INSTALL):	$$($(2)_TARGET_INSTALL_IMAGES)
>  endif
>  
> -$(1)-install-host:		$$($(2)_TARGET_INSTALL_HOST)
> +endif # host vs.target
> +
> +$$($(2)_TARGET_INSTALL_TARGET):	$$($(2)_TARGET_BUILD)
> +$$($(2)_TARGET_INSTALL_STAGING):$$($(2)_TARGET_BUILD)
> +$$($(2)_TARGET_INSTALL_IMAGES):	$$($(2)_TARGET_BUILD)
>  $$($(2)_TARGET_INSTALL_HOST):	$$($(2)_TARGET_BUILD)
>  
> +# Some packages use install-staging stuff for install-target
> +$$($(2)_TARGET_INSTALL_TARGET):		$$($(2)_TARGET_INSTALL_STAGING)
> +
>  $(1)-build:		$$($(2)_TARGET_BUILD)
>  $$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE)
>  
> @@ -813,6 +822,7 @@ $(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
>  
>  # define the PKG variable for all targets, containing the
>  # uppercase package variable prefix
> +$$($(2)_TARGET_INSTALL):		PKG=$(2)
>  $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
>  $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
>  $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 06/15] package/pkg-generic: add the concept of extract dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 06/15] package/pkg-generic: add the concept of extract dependency Thomas Petazzoni
@ 2017-12-29 15:33   ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:33 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> Extract dependencies are dependencies that must be ready before the
> extract step of a package, i.e for tools that are needed to extract
> packages themselves. Current examples of such tools are host-tar,
> host-lzip and host-xz.
> 
> They are currently handled through DEPENDENCIES_HOST_PREREQ. However,
> this mechanism has a number of drawbacks:
> 
>  - First and foremost, because host-tar/host-lzip/host-xz are not
>    listed in the dependencies of packages, the package infrastructure
>    does not know it should rsync them in the context of per-package
>    SDK.
> 
>  - Second, there is no dependency handling *between* them. I.e, we
>    have no mechanism that says host-tar should be built before
>    host-lzip, while it is in fact the case: if you need to build
>    host-lzip, you need to extract a tarball, so you may need host-tar
>    if your system tarball is not capable enough.
> 
> For those reasons, it makes sense to add explicit support for "extract
> dependencies" in the package infrastructure, through the
> <pkg>_EXTRACT_DEPENDENCIES variable. It is unlikely this variable will
> ever be used by a package .mk file, but it will be used internally by
> the package infrastructure.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Except for a typo (see below):

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

> ---
> Changes since v2:
>  - New patch
> ---
>  docs/manual/adding-packages-generic.txt | 7 +++++++
>  package/pkg-generic.mk                  | 8 +++++++-
>  2 files changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
> index 63ea51bf89..7f42cb09e9 100644
> --- a/docs/manual/adding-packages-generic.txt
> +++ b/docs/manual/adding-packages-generic.txt
> @@ -347,6 +347,13 @@ information is (assuming the package name is +libfoo+) :
>    a similar way, +HOST_LIBFOO_DEPENDENCIES+ lists the dependencies for
>    the current host package.
>  
> +* +LIBFOO_EXTRACT_DEPENDENCIES+ lists the dependencies (in terms of
> +  package name) that are required for the current target package to be
> +  extracted. These depnedencies are guaranteed to be compiled and
> +  installed before the extract step of the current package
> +  starts. This is only use internally by the package infrastructure,

*used

Regards,
Yann E. MORIN.

> +  and should typically not be used directly by packages.
> +
>  * +LIBFOO_PATCH_DEPENDENCIES+ lists the dependencies (in terms of
>    package name) that are required for the current package to be
>    patched. These dependencies are guaranteed to be extracted and
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 2b821e9bdf..824c34bec9 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -589,8 +589,13 @@ endif
>  
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
> +$(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
>  $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
> -$(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))
> +$(2)_FINAL_ALL_DEPENDENCIES = \
> +	$$(sort \
> +		$$($(2)_FINAL_DEPENDENCIES) \
> +		$$($(2)_FINAL_EXTRACT_DEPENDENCIES) \
> +		$$($(2)_FINAL_PATCH_DEPENDENCIES))
>  
>  $(2)_INSTALL_STAGING		?= NO
>  $(2)_INSTALL_IMAGES		?= NO
> @@ -716,6 +721,7 @@ $$($(2)_TARGET_PATCH):  | $$(patsubst %,%-patch,$$($(2)_FINAL_PATCH_DEPENDENCIES
>  
>  $(1)-extract:			$$($(2)_TARGET_EXTRACT)
>  $$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
> +$$($(2)_TARGET_EXTRACT): | $$($(2)_FINAL_EXTRACT_DEPENDENCIES)
>  
>  $(1)-depends:		$$($(2)_FINAL_DEPENDENCIES)
>  
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an extract dependency
  2017-12-02 20:19     ` Thomas Petazzoni
@ 2017-12-29 15:34       ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:34 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-02 21:19 +0100, Thomas Petazzoni spake thusly:
> On Sat, 2 Dec 2017 16:06:06 +0100, Yann E. MORIN wrote:
> > > To achieve that, check-host-tar.mk fills in the  
> > s/$/ variable/
> ACK, will fix.
> 
> > > The BR2_TAR_HOST_DEPENDENCY is added to all packages, except host-tar
> > > itself (obviously) and host-skeleton, because we depend on
> > > host-skeleton to install host-tar properly in HOST_DIR.  
> > 
> > As we discussed on IRC, there is also the case for the tar filesystem,
> > which may require host-tar.
> > 
> > Granted, it is very very unlikely that we end up with no package that
> > require host-tar, but still...
> 
> Yes, I agree, the tar filesystem should depend on
> $(BR2_TAR_HOST_DEPENDENCY) as well. I'll fix that.
> 
> > > +ifeq ($(filter host-tar host-skeleton,$(1)),)
> > > +$(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
> > > +endif  
> > 
> > So, even packages that are available as a zip file will get the
> > dependency on host-tar?
> 
> Yes. Like host-xz/host-lzip, my series does not attempt to switch to
> more fine-grained dependencies, it only intends to move to proper
> package dependencies. Using more fine-grained dependencies is left for
> future work.
> 
> > >  ifeq (,$(call suitable-host-package,tar,$(TAR)))
> > > -DEPENDENCIES_HOST_PREREQ += host-tar
> > >  TAR = $(HOST_DIR)/bin/tar
> > > +BR2_TAR_HOST_DEPENDENCY += host-tar  
> > Why assign-append here, instead of simply assign?
> Good point, will fix.

Once the two minor issues are fixed:

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz as an extract dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz " Thomas Petazzoni
  2017-12-02 15:08   ` Yann E. MORIN
@ 2017-12-29 15:36   ` Yann E. MORIN
  1 sibling, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:36 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-xz dependency handling from
> DEPENDENCY_HOST_PREREQ to an extract dependency.
> 
> To achieve that, check-host-xz.mk fills in the
> BR2_XZ_HOST_DEPENDENCY with host-tar if building a host-tar is
> needed. The name BR2_XZ_HOST_DEPENDENCY has been chosen because it
> matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> check-host-cmake.mk.
> 
> The BR2_XZ_HOST_DEPENDENCY is added to all packages, except:
> 
>  - host-xz, because we would otherwise depend on ourself.
> 
>  - host-tar, because xz itself is delivered as a tarball, so we need
>    to have host-xz depend on host-tar, and not host-tar depend on
>    host-xz
> 
>  - host-skeleton, because we need to have host-xz depend on
>    host-skeleton, and not the opposite.
> 
> In addition, we modify xz.mk to explicitly build host-xz without
> ccache. We generally took the approach of building host-ccache *after*
> all the extractors have been built.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

My prior questions have been answered, so:

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - New patch
> ---
>  package/pkg-generic.mk                   | 4 ++++
>  package/xz/xz.mk                         | 5 +++++
>  support/dependencies/check-host-xzcat.mk | 2 +-
>  3 files changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 86dd2561d4..dfd4719d29 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -591,6 +591,10 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
> +ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> +$(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
> +endif
> +
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/package/xz/xz.mk b/package/xz/xz.mk
> index e8116f24ae..bcdac13ee3 100644
> --- a/package/xz/xz.mk
> +++ b/package/xz/xz.mk
> @@ -18,5 +18,10 @@ else
>  XZ_CONF_OPTS = --disable-threads
>  endif
>  
> +# we are built before ccache
> +HOST_XZ_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> index 9be75c7311..e5d72f95dd 100644
> --- a/support/dependencies/check-host-xzcat.mk
> +++ b/support/dependencies/check-host-xzcat.mk
> @@ -2,7 +2,7 @@
>  # If it is not present, build our own host-xzcat
>  
>  ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
> -DEPENDENCIES_HOST_PREREQ += host-xz
> +BR2_XZCAT_HOST_DEPENDENCY = host-xz
>  EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
>  XZCAT = $(HOST_DIR)/bin/xzcat
>  endif
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip as an extract dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip " Thomas Petazzoni
  2017-12-02 15:12   ` Yann E. MORIN
@ 2017-12-29 15:36   ` Yann E. MORIN
  1 sibling, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:36 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-xz dependency handling from
> DEPENDENCY_HOST_PREREQ to an extract dependency.
> 
> To achieve that, check-host-xz.mk fills in the
> BR2_LZIP_HOST_DEPENDENCY with host-tar if building a host-tar is
> needed. The name BR2_LZIP_HOST_DEPENDENCY has been chosen because it
> matches the name BR2_CMAKE_HOST_DEPENDENCY already used in
> check-host-cmake.mk.
> 
> The BR2_LZIP_HOST_DEPENDENCY is added to all packages, except:
> 
>  - host-lzip, because we would otherwise depend on ourself.
> 
>  - host-tar, because lzip itself is delivered as a tarball, so we need
>    to have host-lzip depend on host-tar, and not host-tar depend on
>    host-lzip
> 
>  - host-skeleton, because we need to have host-lzip depend on
>    host-skeleton, and not the opposite.
> 
> We also mutually exclude host-lzip and host-xz from dependending on
> each other, to avoid a circular dependency.
> 
> In addition, we modify lzip.mk to explicitly build host-lzip without
> ccache. We generally took the approach of building host-ccache *after*
> all the extractors have been built.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Since my comments have been answered:

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - New patch
> ---
>  package/lzip/lzip.mk                    | 2 +-
>  package/pkg-generic.mk                  | 6 +++++-
>  support/dependencies/check-host-lzip.mk | 2 +-
>  3 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/package/lzip/lzip.mk b/package/lzip/lzip.mk
> index b7ba5dd21d..a16a22214c 100644
> --- a/package/lzip/lzip.mk
> +++ b/package/lzip/lzip.mk
> @@ -16,7 +16,7 @@ endef
>  
>  define HOST_LZIP_CONFIGURE_CMDS
>  	(cd $(@D); $(HOST_MAKE_ENV) ./configure --prefix=$(HOST_DIR) \
> -		$(HOST_CONFIGURE_OPTS) )
> +		$(HOST_CONFIGURE_OPTS) CC="$(HOSTCC_NOCCACHE)" CXX="$(HOSTCXX_NOCCACHE)")
>  endef
>  
>  define LZIP_BUILD_CMDS
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index dfd4719d29..e623afe7bc 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -591,10 +591,14 @@ ifeq ($(filter host-tar host-skeleton,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
> -ifeq ($(filter host-tar host-skeleton host-xz,$(1)),)
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
>  endif
>  
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
> +$(2)_EXTRACT_DEPENDENCIES += $(BR2_LZIP_HOST_DEPENDENCY)
> +endif
> +
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
> index 00cdd0a236..cdd784058c 100644
> --- a/support/dependencies/check-host-lzip.mk
> +++ b/support/dependencies/check-host-lzip.mk
> @@ -1,5 +1,5 @@
>  ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
> -DEPENDENCIES_HOST_PREREQ += host-lzip
> +BR2_LZIP_HOST_DEPENDENCY = host-lzip
>  EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
>  LZCAT = $(HOST_DIR)/bin/lzip -d -c
>  endif
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 10/15] package/pkg-generic: handle host-ccache as a regular dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 10/15] package/pkg-generic: handle host-ccache as a regular dependency Thomas Petazzoni
@ 2017-12-29 15:42   ` Yann E. MORIN
  2017-12-29 15:48     ` Thomas Petazzoni
  0 siblings, 1 reply; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:42 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This moves the host-ccache dependency handling from
> DEPENDENCIES_HOST_PREREQ to a proper package dependency. When
> BR2_CCACHE=y, we add host-ccache as a regular dependency of all
> packages except:
> 
>  - The extractor packages host-tar, host-xz and host-lzip
> 
>  - host-ccache itself
> 
>  - host-skeleton, because all packages depend on it
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> Changes since v2:
>  - New patch
> ---
>  package/ccache/ccache.mk             | 5 +++++
>  package/pkg-generic.mk               | 6 ++++++
>  support/dependencies/dependencies.mk | 4 ----
>  3 files changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index afbec44fac..d6ef455e8b 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -21,6 +21,11 @@ CCACHE_LICENSE_FILES = LICENSE.txt GPL-3.0.txt
>  # has zero dependency besides the C library.
>  HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
>  
> +# We are ccache, so we can't use ccache
> +HOST_CCACHE_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  # Patch host-ccache as follows:
>  #  - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
>  #    is already used by autotargets for the ccache package.
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index e623afe7bc..cb5889c9ef 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -599,6 +599,12 @@ ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_LZIP_HOST_DEPENDENCY)
>  endif
>  
> +ifeq ($(BR2_CCACHE),y)
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
> +$(2)_DEPENDENCIES += host-ccache

Sorry, I must be dumb, but I don't see where you exclude host-ccache
from being its own dependency...

But I surely agree on the principle of this patch! ;-)

Regards,
Yann E. MORIN.

> +endif
> +endif
> +
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> diff --git a/support/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
> index ef2ae9b7e1..1a4b5df9f2 100644
> --- a/support/dependencies/dependencies.mk
> +++ b/support/dependencies/dependencies.mk
> @@ -14,10 +14,6 @@ $(shell support/dependencies/check-host-$(1).sh $(2))
>  endef
>  -include $(sort $(wildcard support/dependencies/check-host-*.mk))
>  
> -ifeq ($(BR2_CCACHE),y)
> -DEPENDENCIES_HOST_PREREQ += host-ccache
> -endif
> -
>  core-dependencies:
>  	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
>  		DL_TOOLS="$(sort $(DL_TOOLS_DEPENDENCIES))" \
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 10/15] package/pkg-generic: handle host-ccache as a regular dependency
  2017-12-29 15:42   ` Yann E. MORIN
@ 2017-12-29 15:48     ` Thomas Petazzoni
  0 siblings, 0 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-29 15:48 UTC (permalink / raw)
  To: buildroot

Hello,

On Fri, 29 Dec 2017 16:42:28 +0100, Yann E. MORIN wrote:

> > +ifeq ($(BR2_CCACHE),y)
> > +ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
> > +$(2)_DEPENDENCIES += host-ccache  
> 
> Sorry, I must be dumb, but I don't see where you exclude host-ccache
> from being its own dependency...

Good point. It's make that does it for me:

make[1]: Circular /home/thomas/projets/buildroot/output/build/host-ccache-3.3.4/.stamp_configured <- host-ccache dependency dropped.

 :-)

I'll fix that in the next iteration. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 11/15] package/pkg-generic: handle host-fakedate as a regular dependency
  2017-12-01 20:53 ` [Buildroot] [RFCv3 11/15] package/pkg-generic: handle host-fakedate " Thomas Petazzoni
@ 2017-12-29 15:50   ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:50 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> This commit moves the host-fakedate dependency handling from
> DEPENDENCIES_HOST_PREREQ to a proper regular dependency handled by the
> package infrastructure.
> 
> host-fakedate is added as dependency to all packages, except
> host-skeleton, because we depend on it.
> 
> In addition, we make sure that host-fakedate does not grow a
> dependency on host-{tar,xz,lzip,ccache} to avoid circular
> dependencies. host-fakedate does not need any extraction tool and does
> not need to build C/C++ code (the source code is just a shell script
> available in Buildroot).
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

But see below...

> ---
> Changes since v2:
>  - New patch
> ---
>  Makefile               |  1 -
>  package/pkg-generic.mk | 14 ++++++++++----
>  2 files changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 56bf083098..f31834682c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -255,7 +255,6 @@ export LC_ALL = C
>  export GZIP = -n
>  BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
>  export SOURCE_DATE_EPOCH ?= $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
> -DEPENDENCIES_HOST_PREREQ += host-fakedate
>  endif
>  
>  # To put more focus on warnings, be less verbose as default
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index cb5889c9ef..7c5d951af9 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -587,24 +587,30 @@ ifneq ($(1),host-skeleton)
>  $(2)_DEPENDENCIES += host-skeleton
>  endif
>  
> -ifeq ($(filter host-tar host-skeleton,$(1)),)
> +ifeq ($(filter host-tar host-skeleton host-fakedate,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
>  
> -ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_XZCAT_HOST_DEPENDENCY)
>  endif
>  
> -ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_LZIP_HOST_DEPENDENCY)
>  endif
>  
>  ifeq ($(BR2_CCACHE),y)
> -ifeq ($(filter host-tar host-skeleton host-xz host-lzip,$(1)),)
> +ifeq ($(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)

Those last three conditions are using the exact same list, maybe we can
re-factorise the code to avoid duplicating that list?

Regards,
Yann E. MORIN.

>  $(2)_DEPENDENCIES += host-ccache
>  endif
>  endif
>  
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +ifeq ($(filter host-skeleton,$(1)),)
> +$(2)_DEPENDENCIES += host-fakedate
> +endif
> +endif
> +
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_EXTRACT_DEPENDENCIES = $$(sort $$($(2)_EXTRACT_DEPENDENCIES))
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 12/15] core: kill DEPENDENCIES_HOST_PREREQ
  2017-12-01 20:53 ` [Buildroot] [RFCv3 12/15] core: kill DEPENDENCIES_HOST_PREREQ Thomas Petazzoni
@ 2017-12-29 15:53   ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 15:53 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> Now that DEPENDENCIES_HOST_PREREQ is no longer used anywhere, we can
> kill it.

With fire! A lot of fire! ;-]

> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Defintiely:

Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - New patch
> ---
>  Makefile                             |  2 --
>  package/pkg-generic.mk               |  2 --
>  support/dependencies/dependencies.mk | 10 +++-------
>  3 files changed, 3 insertions(+), 11 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index f31834682c..090b3ba191 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -485,8 +485,6 @@ include package/Makefile.in
>  -include $(wildcard arch/arch.mk.*)
>  include support/dependencies/dependencies.mk
>  
> -PACKAGES += $(DEPENDENCIES_HOST_PREREQ)
> -
>  include toolchain/*.mk
>  include toolchain/*/*.mk
>  
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 7c5d951af9..8c6e34cc9f 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -725,9 +725,7 @@ $(1)-configure:			$$($(2)_TARGET_CONFIGURE)
>  $$($(2)_TARGET_CONFIGURE):	| $$($(2)_FINAL_DEPENDENCIES)
>  
>  $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
> -ifeq ($$(filter $(1),$$(DEPENDENCIES_HOST_PREREQ)),)
>  $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
> -endif
>  
>  ifeq ($$($(2)_OVERRIDE_SRCDIR),)
>  # In the normal case (no package override), the sequence of steps is
> diff --git a/support/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
> index 1a4b5df9f2..d8ec137b12 100644
> --- a/support/dependencies/dependencies.mk
> +++ b/support/dependencies/dependencies.mk
> @@ -14,18 +14,14 @@ $(shell support/dependencies/check-host-$(1).sh $(2))
>  endef
>  -include $(sort $(wildcard support/dependencies/check-host-*.mk))
>  
> -core-dependencies:
> -	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
> +dependencies:
> +	@HOSTCC="$(firstword $(HOSTCC_NOCCACHE))" MAKE="$(MAKE)" \
>  		DL_TOOLS="$(sort $(DL_TOOLS_DEPENDENCIES))" \
>  		$(TOPDIR)/support/dependencies/dependencies.sh
>  
> -core-dependencies $(DEPENDENCIES_HOST_PREREQ): HOSTCC=$(HOSTCC_NOCCACHE)
> -core-dependencies $(DEPENDENCIES_HOST_PREREQ): HOSTCXX=$(HOSTCXX_NOCCACHE)
> -dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
> -
>  ################################################################################
>  #
>  # Toplevel Makefile options
>  #
>  ################################################################################
> -.PHONY: dependencies core-dependencies
> +.PHONY: dependencies
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2017-12-01 20:53 ` [Buildroot] [RFCv3 13/15] core: change host RPATH handling Thomas Petazzoni
@ 2017-12-29 17:31   ` Yann E. MORIN
  2017-12-29 20:20     ` Thomas Petazzoni
  2018-02-06 22:04   ` Matthew Weber
  1 sibling, 1 reply; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 17:31 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> Currently, our strategy for the RPATH of host binaries is as follows:
> 
>  - During the build, we force an absolute RPATH to be encoded into
>    binaries, by passing -Wl,-rpath,$(HOST_DIR)/lib. This ensures that
>    host binaries will find their libraries.
> 
>  - In the "make sdk" target, when preparing the SDK to be relocatable,
>    we use patchelf to replace those absolute RPATHs by relative RPATHs
>    that use $ORIGIN.
> 
> Unfortunately, the use of absolute RPATH during the build is not
> compatible with the move to per-package SDK, where a different host
> directory is used for the build of each package. Therefore, we need to
> move to a different strategy for RPATH handling.
> 
> The new strategy is as follows:
> 
>  - We no longer pass -Wl,-rpath,$(HOST_DIR)/lib when building
>    packages, so the host binaries no longer have any rpath encoded.
> 
>  - At the end of the installation of every package, we just the
>    fix-rpath logic on the host binaries, so that all host binaries
>    that don't already have $ORIGIN/../lib as their RPATH are updated
>    to have such a RPATH.
> 
> In order to achieve this, the following changes are made:
> 
>  - HOST_LDFLAGS no longer contains -Wl,-rpath,$(HOST_DIR)
> 
>  - The hook that ran at the end of every package installation
>    (check_host_rpath) to verify that our hardcoded RPATH is indeed
>    present in all binaries is removed, as it is no longer needed: we
>    will force $ORIGIN/../LIB as an RPATH in all binaries.

Then you can just get rid of support/scripts/check-host-rpath that is
now no longer used.

>  - host-patchelf is added as a dependency of all packages, except
>    itself. This is necessary as all packages now need to run patchelf
>    at the end of their installation process.
> 
>    TODO: things like host-tar, host-lzip and host-ccache will have to
>    be handled properly.

Indeed, because the very moment that patchelf is available, the next
package will trigger patchelf on all executables, which means that
package will be responsible for touching files from previous packages,
and thus will trigger the no-two-packages-should-touch-the-same-file
check.

>  - "fix-rpath host" is called at the end of the package installation,
>    in the recently introduced .stamp_installed, which guarantees to be
>    executed after all of target, staging, images and host
>    installations have completed. Indeed, while in theory only host
>    packages install files into $(HOST_DIR), in practice a number of
>    target packages also install host binaries. So to be reliable, we
>    do the "fix-rpath host" logic at the end of the installation of
>    every package.
> 
>  - The fix-rpath script is modified:
> 
>    * The "host" target now simply hardcodes setting $ORIGIN/../lib as
>      an RPATH, instead of using --relative-to-file
> 
>    * We simply exclude the patchelf program instead of making a copy
>      of it: since patchelf depends only on the C/C++ libraries, it
>      doesn't need to have a RPATH set to $ORIGIN/../lib
> 
>    * Also, in order to avoid re-writing host binaries over and over
>      again, we only set the RPATH if it is not already set.
> 
> This change is independent from the per-package SDK functionality, and
> could be applied separately.

Except it requires the host-tar et al. stuff to be handled first.

Regards,
Yann E. MORIN.

> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> NOTE: an alternative to running "fix-rpath host" at the end of every
> package installation would be to run it only for host packages
> (unconditionally) and have a special flag that target packages can use
> to indicate that they have installed things to $(HOST_DIR). Not sure
> if this is worth the complexity, though.
> 
> NOTE2: is it OK to force $ORIGIN/../lib as the RPATH of all host
> binaries? This is OK for binaries in host/{bin,sbin} of course, but do
> we have host binaries in other places, for which this $ORIGIN/../lib
> RPATH wouldn't be appropriate?
> 
> Changes since v2:
>  - No changes
> 
> Changes since v1:
>  - Following the feedback from Yann and Arnout, the approach in v2 was
>    completely changed: instead of using LD_LIBRARY_PATH, we now run
>    patchelf at the end of each package installation to fix the RPATH.
> ---
>  Makefile                  |  1 -
>  package/Makefile.in       |  2 +-
>  package/pkg-generic.mk    | 14 ++++++--------
>  support/scripts/fix-rpath | 31 +++++++++++++++++++------------
>  4 files changed, 26 insertions(+), 22 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 090b3ba191..d8fa91120b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -555,7 +555,6 @@ world: target-post-image
>  .PHONY: sdk
>  sdk: world
>  	@$(call MESSAGE,"Rendering the SDK relocatable")
> -	$(TOPDIR)/support/scripts/fix-rpath host
>  	$(TOPDIR)/support/scripts/fix-rpath staging
>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> diff --git a/package/Makefile.in b/package/Makefile.in
> index a1a5316051..e94a75c230 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -220,7 +220,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
>  HOST_CFLAGS   ?= -O2
>  HOST_CFLAGS   += $(HOST_CPPFLAGS)
>  HOST_CXXFLAGS += $(HOST_CFLAGS)
> -HOST_LDFLAGS  += -L$(HOST_DIR)/lib -Wl,-rpath,$(HOST_DIR)/lib
> +HOST_LDFLAGS  += -L$(HOST_DIR)/lib
>  
>  # The macros below are taken from linux 4.11 and adapted slightly.
>  # Copy more when needed.
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 8c6e34cc9f..f4a943e3ce 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -118,14 +118,6 @@ endef
>  
>  GLOBAL_INSTRUMENTATION_HOOKS += check_bin_arch
>  
> -# This hook checks that host packages that need libraries that we build
> -# have a proper DT_RPATH or DT_RUNPATH tag
> -define check_host_rpath
> -	$(if $(filter install-host,$(2)),\
> -		$(if $(filter end,$(1)),support/scripts/check-host-rpath $(3) $(HOST_DIR)))
> -endef
> -GLOBAL_INSTRUMENTATION_HOOKS += check_host_rpath
> -
>  define step_check_build_dir_one
>  	if [ -d $(2) ]; then \
>  		printf "%s: installs files in %s\n" $(1) $(2) >&2; \
> @@ -343,6 +335,8 @@ $(BUILD_DIR)/%/.stamp_target_installed:
>  # Finalize installation
>  $(BUILD_DIR)/%/.stamp_installed:
>  	@$(call step_start,install)
> +	$(if $(filter host-patchelf,$($(PKG)_FINAL_DEPENDENCIES)), \
> +		$(TOPDIR)/support/scripts/fix-rpath host)
>  	@$(call step_end,install)
>  	$(Q)touch $@
>  
> @@ -587,6 +581,10 @@ ifneq ($(1),host-skeleton)
>  $(2)_DEPENDENCIES += host-skeleton
>  endif
>  
> +ifeq ($(filter host-patchelf host-skeleton host-tar host-xz host-lzip host-ccache host-fakedate,$(1)),)
> +$(2)_DEPENDENCIES += host-patchelf
> +endif
> +
>  ifeq ($(filter host-tar host-skeleton host-fakedate,$(1)),)
>  $(2)_EXTRACT_DEPENDENCIES += $(BR2_TAR_HOST_DEPENDENCY)
>  endif
> diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath
> index 15705a3b0d..5b2f24ed32 100755
> --- a/support/scripts/fix-rpath
> +++ b/support/scripts/fix-rpath
> @@ -61,7 +61,7 @@ main() {
>      local rootdir
>      local tree="${1}"
>      local find_args=( )
> -    local sanitize_extra_args=( )
> +    local sanitize_args=( )
>  
>      if ! "${PATCHELF}" --version > /dev/null 2>&1; then
>  	echo "Error: can't execute patchelf utility '${PATCHELF}'"
> @@ -84,12 +84,14 @@ main() {
>                  find_args+=( "-path" "${HOST_DIR}""${excludepath}" "-prune" "-o" )
>              done
>  
> -            # do not process the patchelf binary but a copy to work-around "file in use"
> +            # do not process the patchelf binary, as it is ourselves
> +            # (and it doesn't need a rpath as it doesn't use libraries
> +            # from HOST_DIR)
>              find_args+=( "-path" "${PATCHELF}" "-prune" "-o" )
> -            cp "${PATCHELF}" "${PATCHELF}.__to_be_patched"
>  
>              # we always want $ORIGIN-based rpaths to make it relocatable.
> -            sanitize_extra_args+=( "--relative-to-file" )
> +            sanitize_args+=( "--set-rpath" )
> +            sanitize_args+=( "\$ORIGIN/../lib" )
>              ;;
>  
>          staging)
> @@ -101,14 +103,18 @@ main() {
>              done
>  
>              # should be like for the target tree below
> -            sanitize_extra_args+=( "--no-standard-lib-dirs" )
> +            sanitize_args+=( "--no-standard-lib-dirs" )
> +            sanitize_args+=( "--make-rpath-relative" )
> +            sanitize_args+=( "${rootdir}" )
>              ;;
>  
>          target)
>              rootdir="${TARGET_DIR}"
>              # we don't want $ORIGIN-based rpaths but absolute paths without rootdir.
>              # we also want to remove rpaths pointing to /lib or /usr/lib.
> -            sanitize_extra_args+=( "--no-standard-lib-dirs" )
> +            sanitize_args+=( "--no-standard-lib-dirs" )
> +            sanitize_args+=( "--make-rpath-relative" )
> +            sanitize_args+=( "${rootdir}" )
>              ;;
>  
>          *)
> @@ -120,20 +126,21 @@ main() {
>      find_args+=( "-type" "f" "-print" )
>  
>      while read file ; do
> -        # check if it's an ELF file
> -        if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then
> +        rpath=$(${PATCHELF} --print-rpath "${file}" 2>/dev/null)
> +        if test $? -eq 0; then
> +            # For host binaries, if the rpath is already correct, skip
> +            if test "${tree}" = "host" -a "${rpath}" = "\$ORIGIN/../lib" ; then
> +                continue
> +            fi
>              # make files writable if necessary
>              changed=$(chmod -c u+w "${file}")
>              # call patchelf to sanitize the rpath
> -            ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}"
> +            ${PATCHELF} ${sanitize_args[@]} "${file}"
>              # restore the original permission
>              test "${changed}" != "" && chmod u-w "${file}"
>          fi
>      done < <(find "${rootdir}" ${find_args[@]})
>  
> -    # Restore patched patchelf utility
> -    test "${tree}" = "host" && mv "${PATCHELF}.__to_be_patched" "${PATCHELF}"
> -
>      # ignore errors
>      return 0
>  }
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 14/15] Makefile: evaluate CCACHE and HOST{CC, CXX} at time of use
  2017-12-01 20:53 ` [Buildroot] [RFCv3 14/15] Makefile: evaluate CCACHE and HOST{CC, CXX} at time of use Thomas Petazzoni
@ 2017-12-29 17:32   ` Yann E. MORIN
  0 siblings, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-29 17:32 UTC (permalink / raw)
  To: buildroot

Thomas, All,

On 2017-12-01 21:53 +0100, Thomas Petazzoni spake thusly:
> As we are going to move to per-package SDK, the location of CCACHE and
> therefore the definitions of HOSTCC and HOSTCXX need to be evaluated
> at the time of use and not at the time of assignment. Indeed, the
> value of HOST_DIR changes from one package to the other.
> 
> Therefore, we need to change from := to =.
> 
> In addition, while doing A := $(something) $(A) is possible, doing A =
> $(something) $(A) is not legal. So, instead of defining HOSTCC in
> terms of the current HOSTCC variable, we re-use HOSTCC_NOCCACHE
> instead.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
> Changes since v2:
>  - New patch
> ---
>  Makefile | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index d8fa91120b..e05a1ec09a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -446,11 +446,11 @@ BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(PATH)"
>  TARGET_DIR_WARNING_FILE = $(TARGET_DIR)/THIS_IS_NOT_YOUR_ROOT_FILESYSTEM
>  
>  ifeq ($(BR2_CCACHE),y)
> -CCACHE := $(HOST_DIR)/bin/ccache
> +CCACHE = $(HOST_DIR)/bin/ccache
>  BR_CACHE_DIR ?= $(call qstrip,$(BR2_CCACHE_DIR))
>  export BR_CACHE_DIR
> -HOSTCC := $(CCACHE) $(HOSTCC)
> -HOSTCXX := $(CCACHE) $(HOSTCXX)
> +HOSTCC = $(CCACHE) $(HOSTCC_NOCCACHE)
> +HOSTCXX = $(CCACHE) $(HOSTCXX_NOCCACHE)
>  else
>  export BR_NO_CCACHE
>  endif
> -- 
> 2.13.6
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2017-12-29 17:31   ` Yann E. MORIN
@ 2017-12-29 20:20     ` Thomas Petazzoni
  2017-12-30 13:56       ` Yann E. MORIN
  2018-02-06  0:00       ` Matthew Weber
  0 siblings, 2 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-29 20:20 UTC (permalink / raw)
  To: buildroot

Hello,

On Fri, 29 Dec 2017 18:31:18 +0100, Yann E. MORIN wrote:

> > In order to achieve this, the following changes are made:
> > 
> >  - HOST_LDFLAGS no longer contains -Wl,-rpath,$(HOST_DIR)
> > 
> >  - The hook that ran at the end of every package installation
> >    (check_host_rpath) to verify that our hardcoded RPATH is indeed
> >    present in all binaries is removed, as it is no longer needed: we
> >    will force $ORIGIN/../LIB as an RPATH in all binaries.  
> 
> Then you can just get rid of support/scripts/check-host-rpath that is
> now no longer used.

True.

> 
> >  - host-patchelf is added as a dependency of all packages, except
> >    itself. This is necessary as all packages now need to run patchelf
> >    at the end of their installation process.
> > 
> >    TODO: things like host-tar, host-lzip and host-ccache will have to
> >    be handled properly.  

In fact, this TODO was no longer accurate: I did take care of host-tar,
host-lzip and host-ccache as part of this RFCv3. However...

> Indeed, because the very moment that patchelf is available, the next
> package will trigger patchelf on all executables, which means that
> package will be responsible for touching files from previous packages,
> and thus will trigger the no-two-packages-should-touch-the-same-file
> check.

... I indeed didn't think of this problem. I see two possibilities to
address this:

 1. Have host-patchelf built earlier. I.e have only host-tar be a
    dependency of host-patchelf and nothing else (in my current patch
    series, host-patchelf is built after host-tar, host-xz, host-lzip,
    host-ccache). This requires using HOSTCC_NOCCACHE to build host-tar
    and host-patchelf, but I don't think it's a big deal. And then do a
    special case in fix-rpath to not fix the rpath of the tar program,
    since it should not have dependencies on libraries in HOST_DIR/lib

 2. Make the RPATH fixing logic smarter, and don't set a RPATH to
    $ORIGIN/../lib if the program doesn't use a library that is in
    HOST_DIR/lib. This is perhaps the most pretty solution.

> > This change is independent from the per-package SDK functionality, and
> > could be applied separately.  
> 
> Except it requires the host-tar et al. stuff to be handled first.

Which is why the host-tar and al. stuff is earlier. What I meant with
this sentence is that reworking the host RPATH handling can be applied
*before* we switch to per-package SDK.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2017-12-29 20:20     ` Thomas Petazzoni
@ 2017-12-30 13:56       ` Yann E. MORIN
  2018-02-06  0:00       ` Matthew Weber
  1 sibling, 0 replies; 56+ messages in thread
From: Yann E. MORIN @ 2017-12-30 13:56 UTC (permalink / raw)
  To: buildroot

On 2017-12-29 21:20 +0100, Thomas Petazzoni spake thusly:
> Hello,
> 
> On Fri, 29 Dec 2017 18:31:18 +0100, Yann E. MORIN wrote:
> 
> > > In order to achieve this, the following changes are made:
> > > 
> > >  - HOST_LDFLAGS no longer contains -Wl,-rpath,$(HOST_DIR)
> > > 
> > >  - The hook that ran at the end of every package installation
> > >    (check_host_rpath) to verify that our hardcoded RPATH is indeed
> > >    present in all binaries is removed, as it is no longer needed: we
> > >    will force $ORIGIN/../LIB as an RPATH in all binaries.  
> > 
> > Then you can just get rid of support/scripts/check-host-rpath that is
> > now no longer used.
> 
> True.
> 
> > 
> > >  - host-patchelf is added as a dependency of all packages, except
> > >    itself. This is necessary as all packages now need to run patchelf
> > >    at the end of their installation process.
> > > 
> > >    TODO: things like host-tar, host-lzip and host-ccache will have to
> > >    be handled properly.  
> 
> In fact, this TODO was no longer accurate: I did take care of host-tar,
> host-lzip and host-ccache as part of this RFCv3. However...

You said on IRC that the uniq-file check did not trigger for host-tar,
and I think I know why: we're looking at installed files in the step
hook named 'step_pkg_size' which gets run as part of the 'install-host'
step.

But the patchelf will only happen in the new step named 'install', which
you introduced in patch 05/15 (pkg-generic: add .stamp_installed step),
and which is called after 'install-host' (guaranteed, because 'install'
depends on 'install-host').

So, the uniq-file check will not get triggered for host-patchelf.

It will not even be trigger by a later package either, because the
hashes are taken before and after the install step, not from a previous
install step.

So, we are lucky.

However, I don't think this is nice...

> > Indeed, because the very moment that patchelf is available, the next
> > package will trigger patchelf on all executables, which means that
> > package will be responsible for touching files from previous packages,
> > and thus will trigger the no-two-packages-should-touch-the-same-file
> > check.
> 
> ... I indeed didn't think of this problem. I see two possibilities to
> address this:
> 
>  1. Have host-patchelf built earlier. I.e have only host-tar be a
>     dependency of host-patchelf and nothing else (in my current patch
>     series, host-patchelf is built after host-tar, host-xz, host-lzip,
>     host-ccache). This requires using HOSTCC_NOCCACHE to build host-tar
>     and host-patchelf, but I don't think it's a big deal. And then do a
>     special case in fix-rpath to not fix the rpath of the tar program,
>     since it should not have dependencies on libraries in HOST_DIR/lib
> 
>  2. Make the RPATH fixing logic smarter, and don't set a RPATH to
>     $ORIGIN/../lib if the program doesn't use a library that is in
>     HOST_DIR/lib. This is perhaps the most pretty solution.

 3. limit patchelf on the files actually installed by the current
    package, looking at $(BUILD_DIR)/packages-file-list-host.txt to
    get the list...

Actually, this might prompt for storing a per-package list of installed
file, something like (untested):

    diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
    index b4aa22e38c..4a086600dd 100644
    --- a/package/pkg-generic.mk
    +++ b/package/pkg-generic.mk
    @@ -89,8 +89,11 @@ define step_pkg_size_end
     	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after,$(2))
     	comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \
     		while read hash file ; do \
    -			echo "$(1),$${file}" ; \
    -		done >> $(BUILD_DIR)/packages-file-list$(3).txt
    +			echo "$${file}" ; \
    +		done >> $($(PKG)_DIR)/.br_filelist.txt
    +		sed -r -e 's/^/$(1),/' \
    +			< $($(PKG)_DIR)/.br_filelist.txt \
    +			>> $(BUILD_DIR)/packages-file-list$(3).txt
     	rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after
     endef

... and then re-using $($(PKG)_DIR)/.br_filelist.txt as the list of
files installed by the current package, and only patchelf thos liste in
there.

My 2ct...

Regards,
Yann E. MORIN.

> > > This change is independent from the per-package SDK functionality, and
> > > could be applied separately.  
> > 
> > Except it requires the host-tar et al. stuff to be handled first.
> 
> Which is why the host-tar and al. stuff is earlier. What I meant with
> this sentence is that reworking the host RPATH handling can be applied
> *before* we switch to per-package SDK.
> 
> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path
  2017-12-01 20:53 ` [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path Thomas Petazzoni
  2017-12-29 15:22   ` Yann E. MORIN
@ 2017-12-31 17:24   ` Thomas Petazzoni
  1 sibling, 0 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-31 17:24 UTC (permalink / raw)
  To: buildroot

Hello,

On Fri,  1 Dec 2017 21:53:38 +0100, Thomas Petazzoni wrote:
> The pkg-config wrapper script is currently generated with absolute
> paths to $(STAGING_DIR). However, this will not work properly with
> per-package SDK, and each package will be built with a different
> STAGING_DIR value.
> 
> In order to fix this, we adjust how the pkg-config wrapper script is
> generated, so that it uses a relative path to itself: the sysroot (i.e
> STAGING_DIR) is always located in $(path of
> pkg-config)/../$(STAGING_SUBDIR).
> 
> This change is independent from the per-package SDK work, and could be
> applied independently from it.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
> Changes since v2:
>  - None
> Changes since v1:
>  - As suggested by Arnout, simplify the generation of the pkg-config
>    script by doing only a replacement on @STAGING_SUBDIR@, the rest
>    being encoded inside the pkg-config script.
> ---
>  package/pkgconf/pkg-config.in | 5 ++++-
>  package/pkgconf/pkgconf.mk    | 3 +--
>  2 files changed, 5 insertions(+), 3 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN
  2017-12-01 20:53 ` [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN Thomas Petazzoni
  2017-12-29 15:22   ` Yann E. MORIN
@ 2017-12-31 17:24   ` Thomas Petazzoni
  1 sibling, 0 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2017-12-31 17:24 UTC (permalink / raw)
  To: buildroot

Hello,

On Fri,  1 Dec 2017 21:53:39 +0100, Thomas Petazzoni wrote:
> The upcoming per-package SDK functionality is heavily based on the
> fact that HOST_DIR, STAGING_DIR and TARGET_DIR are evaluated during
> the configure/build/install steps of the packages. Therefore, any
> evaluation-during-assignment using := is going to cause problems, and
> need to be turned into evaluation-during-use using =.
> 
> This patch fix up one such instance in the external toolchain code.
> 
> This change is independent from the per-package SDK functionality, and
> could be applied separately.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
> Changes since v2:
>  - None
> Changes since v1:
>  - Added Arnout Reviewed-by.
> ---
>  toolchain/toolchain-external/pkg-toolchain-external.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 00/15] Per-package host/target directory support
  2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
                   ` (14 preceding siblings ...)
  2017-12-01 20:53 ` [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target Thomas Petazzoni
@ 2018-01-08 22:41 ` Thomas Petazzoni
  15 siblings, 0 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2018-01-08 22:41 UTC (permalink / raw)
  To: buildroot

Peter, Arnout,

I'd like to apply some of the patches in this series in the current
cycle. See below.

On Fri,  1 Dec 2017 21:53:37 +0100, Thomas Petazzoni wrote:

> Thomas Petazzoni (15):
>   pkgconf: use relative path to STAGING_DIR instead of absolute path
>   toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN

These two are already applied.

>   Makefile, skeleton: move the host skeleton logic to host-skeleton
>     package
>   pkg-cmake: install CMake files as part of a package

I'd like to apply those two.

>   pkg-generic: add .stamp_installed step

This one should instead be just before "core: change host RPATH
handling", it is not needed before.

>   package/pkg-generic: add the concept of extract dependency
>   package/pkg-generic: handle host-tar as an extract dependency
>   package/pkg-generic: handle host-xz as an extract dependency
>   package/pkg-generic: handle host-lzip as an extract dependency
>   package/pkg-generic: handle host-ccache as a regular dependency
>   package/pkg-generic: handle host-fakedate as a regular dependency
>   core: kill DEPENDENCIES_HOST_PREREQ

And I'd like to apply all those ones, i.e the extract dependency idea
and killing DEPENDENCIES_HOST_PREREQ.

Are you OK with that ? If so, I'll respin just those ones after taking
Yann's feedback into account.

So if you have anything against the approach used in those patches,
please speak up ! :-)

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2017-12-29 20:20     ` Thomas Petazzoni
  2017-12-30 13:56       ` Yann E. MORIN
@ 2018-02-06  0:00       ` Matthew Weber
  1 sibling, 0 replies; 56+ messages in thread
From: Matthew Weber @ 2018-02-06  0:00 UTC (permalink / raw)
  To: buildroot

Thomas, Yann,

On Fri, Dec 29, 2017 at 9:20 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Fri, 29 Dec 2017 18:31:18 +0100, Yann E. MORIN wrote:
>
[snip]
>  2. Make the RPATH fixing logic smarter, and don't set a RPATH to
>     $ORIGIN/../lib if the program doesn't use a library that is in
>     HOST_DIR/lib. This is perhaps the most pretty solution.
>

What about the case of autotools tests which now don't have an
absolute path to the library?  I ran into an issue with this and the
host-libglibc2 testing for PCRE unicode.  The test worked fine once I
set the LD LIBRARY PATH.

Matt

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target
  2017-12-01 20:53 ` [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target Thomas Petazzoni
@ 2018-02-06 14:45   ` Matthew Weber
  2018-02-06 22:00     ` Arnout Vandecappelle
  0 siblings, 1 reply; 56+ messages in thread
From: Matthew Weber @ 2018-02-06 14:45 UTC (permalink / raw)
  To: buildroot

Thomas,

On Fri, Dec 1, 2017 at 9:53 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> This commit implemnts the core of the move to per-package SDK and
> target directories. The main idea is that instead of having a global
> output/host and output/target in which all packages install files, we
> switch to per-package host and target folders, that only contain their
> explicit dependencies.
>
> There are two main benefits:
>
>  - Packages will no longer discover dependencies that they do not
>    explicitly indicate in their <pkg>_DEPENDENCIES variable.
>
>  - We can support top-level parallel build properly, because a package
>    only "sees" its own host directory and target directory, isolated
>    from the build of other packages that can happen in parallel.
>
> It works as follows:
>
>  - A new output/per-package/ folder is created, which will contain one
>    sub-folder per package, and inside it, a "host" folder and a
>    "target" folder:
>
>    output/per-package/busybox/target
>    output/per-package/busybox/host
>    output/per-package/host-fakeroot/target
>    output/per-package/host-fakeroot/host
>
>    This output/per-package/ folder is PER_PACKAGE_DIR.
>
>  - The global TARGET_DIR and HOST_DIR variable now automatically point
>    to the per-package directory when PKG is defined. So whenever a
>    package references $(HOST_DIR) or $(TARGET_DIR) in its build
>    process, it effectively references the per-package host/target
>    directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
>    is handled as well.
>

Using this patchset without any -j.  I found a case with host-setools
where it uses the python-package framework to build and using
setuptools puts a default -I<abs
path>/per-package/host-python/host/include.  Previously this would
have pointed at the full sysroot so now there is a missing include.
I've added it for setools and continued on testing at this point.  I'm
unsure if we would consider this a pkg-python.mk update or each
package.....

Matt

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target
  2018-02-06 14:45   ` Matthew Weber
@ 2018-02-06 22:00     ` Arnout Vandecappelle
  2018-02-06 22:39       ` Matthew Weber
  0 siblings, 1 reply; 56+ messages in thread
From: Arnout Vandecappelle @ 2018-02-06 22:00 UTC (permalink / raw)
  To: buildroot



On 06-02-18 15:45, Matthew Weber wrote:
[snip]
> 
> Using this patchset without any -j.  I found a case with host-setools
> where it uses the python-package framework to build and using
> setuptools puts a default -I<abs
> path>/per-package/host-python/host/include.  Previously this would
> have pointed at the full sysroot so now there is a missing include.

 There should also be a -I<abs path>/per-package/host-setools/host/include
because that's in HOST_CFLAGS. Or does this get hardcoded by setuptools?

 BTW, I guess the problem is that it fails to find libselinux or libsepol includes?

 Regards,
 Arnout

> I've added it for setools and continued on testing at this point.  I'm
> unsure if we would consider this a pkg-python.mk update or each
> package.....
> 
> Matt
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2017-12-01 20:53 ` [Buildroot] [RFCv3 13/15] core: change host RPATH handling Thomas Petazzoni
  2017-12-29 17:31   ` Yann E. MORIN
@ 2018-02-06 22:04   ` Matthew Weber
  2018-02-06 23:19     ` Arnout Vandecappelle
  1 sibling, 1 reply; 56+ messages in thread
From: Matthew Weber @ 2018-02-06 22:04 UTC (permalink / raw)
  To: buildroot

Thomas,

On Fri, Dec 1, 2017 at 9:53 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
[snip]
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -220,7 +220,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
>  HOST_CFLAGS   ?= -O2
>  HOST_CFLAGS   += $(HOST_CPPFLAGS)
>  HOST_CXXFLAGS += $(HOST_CFLAGS)
> -HOST_LDFLAGS  += -L$(HOST_DIR)/lib -Wl,-rpath,$(HOST_DIR)/lib
> +HOST_LDFLAGS  += -L$(HOST_DIR)/lib

Reproduced with the following config
BR2_aarch64=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
BR2_SYSTEM_DHCP="eth0"
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_VERSION=y
BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.13.6"
BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux-4.13.config"
BR2_PACKAGE_CHECKPOLICY=y
BR2_PACKAGE_POLICYCOREUTILS=y
BR2_PACKAGE_SETOOLS=y
BR2_PACKAGE_HOST_CHECKPOLICY=y

That build will first fail at host-libglib2 (mentioned in previous
email on this thread) because the autotools test of pcre unicode will
fail as it can't find the libraries.  See patch
https://patchwork.ozlabs.org/patch/870130/

Next it will fail at setools because of a python header include now
only pointing at the python host/include vs the full host/include. It
seems setuptools by default inherits an include path to python headers
install location.  See patch
https://patchwork.ozlabs.org/patch/870131/.

Last it looks like the rpath still needs to be set so that the linker
can evaluate the shared library's shared library dependency.  I tested
the changes below and it builds, plus doesn't leave an rpath in the
executable (Thanks Roman).
<<<<<<<<<  CUT
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -238,7 +238,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
 HOST_CFLAGS   ?= -O2
 HOST_CFLAGS   += $(HOST_CPPFLAGS)
 HOST_CXXFLAGS += $(HOST_CFLAGS)
-HOST_LDFLAGS  += -L$(HOST_DIR)/lib
+HOST_LDFLAGS  += -L$(HOST_DIR)/lib  -Wl,-rpath-link,$(HOST_DIR)/lib

 # The macros below are taken from linux 4.11 and adapted slightly.
 # Copy more when needed.
<<<<<<<<<  CUT
The use of LD_LIBRARY_PATH set to the same value also works.  Here's
my understanding as pulled from the ld man page.

When using ELF or SunOS, one shared library may require another. This
happens when an ld -shared link includes a shared library as one of
the input files. When the linker encounters such a dependency when
doing a non-shared, non-relocateable link, it will automatically try
to locate the required shared library and include it in the link, if
it is not included explicitly. In such a case, the -rpath-link option
specifies the first set of directories to search. The -rpath-link
option may specify a sequence of directory names either by specifying
a list of names separated by colons, or by appearing multiple times.
The linker uses the following search paths to locate required shared
libraries.
1) Any directories specified by -rpath-link options.
2) Any directories specified by -rpath options. The difference between
-rpath and -rpath-link is that directories specified by -rpath options
are included in the executable and used at runtime, whereas the
-rpath-link option is only effective@link time.
3) On an ELF system, if the -rpath and rpath-link options were not
used, search the contents of the environment variable LD_RUN_PATH.
4) On SunOS, if the -rpath option was not used, search any directories
specified using -L options.
5) For a native linker, the contents of the environment variable
LD_LIBRARY_PATH.
6) The default directories, normally `/lib' and `/usr/lib'.


Matt

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target
  2018-02-06 22:00     ` Arnout Vandecappelle
@ 2018-02-06 22:39       ` Matthew Weber
  0 siblings, 0 replies; 56+ messages in thread
From: Matthew Weber @ 2018-02-06 22:39 UTC (permalink / raw)
  To: buildroot

Arnout,

On Tue, Feb 6, 2018 at 11:00 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
> On 06-02-18 15:45, Matthew Weber wrote:
> [snip]
>>
>> Using this patchset without any -j.  I found a case with host-setools
>> where it uses the python-package framework to build and using
>> setuptools puts a default -I<abs
>> path>/per-package/host-python/host/include.  Previously this would
>> have pointed at the full sysroot so now there is a missing include.
>
>  There should also be a -I<abs path>/per-package/host-setools/host/include
> because that's in HOST_CFLAGS. Or does this get hardcoded by setuptools?
>
I haven't been able to verify where the host-python include is coming
from.  It looks like it might be getting pulled as a default based on
setuptools/python install location.

>  BTW, I guess the problem is that it fails to find libselinux or libsepol includes?

Correct here's the patch I'm using so far
https://patchwork.ozlabs.org/patch/870133/

Once I got past this issue, now setools when building for target has
linker issues but it looks like this packages includes are solved.

Matt

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2018-02-06 22:04   ` Matthew Weber
@ 2018-02-06 23:19     ` Arnout Vandecappelle
  2018-02-07  6:30       ` Matthew Weber
  0 siblings, 1 reply; 56+ messages in thread
From: Arnout Vandecappelle @ 2018-02-06 23:19 UTC (permalink / raw)
  To: buildroot



On 06-02-18 23:04, Matthew Weber wrote:
> Thomas,
> 
> On Fri, Dec 1, 2017 at 9:53 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
> [snip]
>> --- a/package/Makefile.in
>> +++ b/package/Makefile.in
>> @@ -220,7 +220,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
>>  HOST_CFLAGS   ?= -O2
>>  HOST_CFLAGS   += $(HOST_CPPFLAGS)
>>  HOST_CXXFLAGS += $(HOST_CFLAGS)
>> -HOST_LDFLAGS  += -L$(HOST_DIR)/lib -Wl,-rpath,$(HOST_DIR)/lib
>> +HOST_LDFLAGS  += -L$(HOST_DIR)/lib
> 
> Reproduced with the following config
> BR2_aarch64=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
> BR2_SYSTEM_DHCP="eth0"
> BR2_LINUX_KERNEL=y
> BR2_LINUX_KERNEL_CUSTOM_VERSION=y
> BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.13.6"
> BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
> BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux-4.13.config"
> BR2_PACKAGE_CHECKPOLICY=y
> BR2_PACKAGE_POLICYCOREUTILS=y
> BR2_PACKAGE_SETOOLS=y
> BR2_PACKAGE_HOST_CHECKPOLICY=y
> 
> That build will first fail at host-libglib2 (mentioned in previous
> email on this thread) because the autotools test of pcre unicode will
> fail as it can't find the libraries.  See patch
> https://patchwork.ozlabs.org/patch/870130/

 IIUC this patch should no longer be needed if we do -Wl,-rpath-link instead, right?


> Next it will fail at setools because of a python header include now
> only pointing at the python host/include vs the full host/include. It
> seems setuptools by default inherits an include path to python headers
> install location.  See patch
> https://patchwork.ozlabs.org/patch/870131/.

 If setuptools completely ignores the CFLAGS we pass to it, then it should be
fixed in setuptools, not in a per-package patch...


> Last it looks like the rpath still needs to be set so that the linker
> can evaluate the shared library's shared library dependency.  I tested
> the changes below and it builds, plus doesn't leave an rpath in the
> executable (Thanks Roman).
> <<<<<<<<<  CUT
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -238,7 +238,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
>  HOST_CFLAGS   ?= -O2
>  HOST_CFLAGS   += $(HOST_CPPFLAGS)
>  HOST_CXXFLAGS += $(HOST_CFLAGS)
> -HOST_LDFLAGS  += -L$(HOST_DIR)/lib
> +HOST_LDFLAGS  += -L$(HOST_DIR)/lib  -Wl,-rpath-link,$(HOST_DIR)/lib
> 
>  # The macros below are taken from linux 4.11 and adapted slightly.
>  # Copy more when needed.
> <<<<<<<<<  CUT
> The use of LD_LIBRARY_PATH set to the same value also works.  Here's
> my understanding as pulled from the ld man page.
> 
> When using ELF or SunOS, one shared library may require another. This
> happens when an ld -shared link includes a shared library as one of
> the input files. When the linker encounters such a dependency when
> doing a non-shared, non-relocateable link, it will automatically try
> to locate the required shared library and include it in the link, if
> it is not included explicitly. In such a case, the -rpath-link option
> specifies the first set of directories to search. The -rpath-link
> option may specify a sequence of directory names either by specifying
> a list of names separated by colons, or by appearing multiple times.
> The linker uses the following search paths to locate required shared
> libraries.
> 1) Any directories specified by -rpath-link options.
> 2) Any directories specified by -rpath options. The difference between
> -rpath and -rpath-link is that directories specified by -rpath options
> are included in the executable and used at runtime, whereas the
> -rpath-link option is only effective at link time.
> 3) On an ELF system, if the -rpath and rpath-link options were not
> used, search the contents of the environment variable LD_RUN_PATH.
> 4) On SunOS, if the -rpath option was not used, search any directories
> specified using -L options.
> 5) For a native linker, the contents of the environment variable
> LD_LIBRARY_PATH.
> 6) The default directories, normally `/lib' and `/usr/lib'.

 I have read this section of the manpage several times over the years, and now
is the first time it makes sense to me :-)

 So indeed, for NEEDED libraries within libraries that you link with, ld will
not look at -L but it will only look at -rpath-link and -rpath (the rest is not
relevant in our context). So if we remove -rpath, it does make sense that we
pass -rpath-link instead (we already pass -L as well).

 That said, I still don't understand why it didn't work without -rpath-link.
Indeed, there *is* a DT_RUNPATH set in the library (at least according to the
commit log of this patch, since fix-rpath is run at the end of installation),
but it's set to $ORIGIN/../lib instead of to an absolute path. So, does ld not
look at DT_RUNPATH? Or does it evaluate $ORIGIN in an unexpected way?


 Finally, note that we said in the Buildroot meeting that this patch is in fact
probably NOT needed for per-package SDK, because the NEEDed libraries are in
fact installed in the location specified by the absolute RPATH. It's just weird
to have an RPATH referring outside of the per-package staging, but it's not an
actual problem.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 13/15] core: change host RPATH handling
  2018-02-06 23:19     ` Arnout Vandecappelle
@ 2018-02-07  6:30       ` Matthew Weber
  0 siblings, 0 replies; 56+ messages in thread
From: Matthew Weber @ 2018-02-07  6:30 UTC (permalink / raw)
  To: buildroot

Arnout,

On Wed, Feb 7, 2018 at 12:19 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
> On 06-02-18 23:04, Matthew Weber wrote:
>> Thomas,
>>
>> On Fri, Dec 1, 2017 at 9:53 PM, Thomas Petazzoni
>> <thomas.petazzoni@free-electrons.com> wrote:
>> [snip]
>>> --- a/package/Makefile.in
>>> +++ b/package/Makefile.in
>>> @@ -220,7 +220,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
>>>  HOST_CFLAGS   ?= -O2
>>>  HOST_CFLAGS   += $(HOST_CPPFLAGS)
>>>  HOST_CXXFLAGS += $(HOST_CFLAGS)
>>> -HOST_LDFLAGS  += -L$(HOST_DIR)/lib -Wl,-rpath,$(HOST_DIR)/lib
>>> +HOST_LDFLAGS  += -L$(HOST_DIR)/lib
>>
>> Reproduced with the following config
>> BR2_aarch64=y
>> BR2_TOOLCHAIN_EXTERNAL=y
>> BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
>> BR2_SYSTEM_DHCP="eth0"
>> BR2_LINUX_KERNEL=y
>> BR2_LINUX_KERNEL_CUSTOM_VERSION=y
>> BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.13.6"
>> BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG=y
>> BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/aarch64-virt/linux-4.13.config"
>> BR2_PACKAGE_CHECKPOLICY=y
>> BR2_PACKAGE_POLICYCOREUTILS=y
>> BR2_PACKAGE_SETOOLS=y
>> BR2_PACKAGE_HOST_CHECKPOLICY=y
>>
>> That build will first fail at host-libglib2 (mentioned in previous
>> email on this thread) because the autotools test of pcre unicode will
>> fail as it can't find the libraries.  See patch
>> https://patchwork.ozlabs.org/patch/870130/
>
>  IIUC this patch should no longer be needed if we do -Wl,-rpath-link instead, right?
>
>
>> Next it will fail at setools because of a python header include now
>> only pointing at the python host/include vs the full host/include. It
>> seems setuptools by default inherits an include path to python headers
>> install location.  See patch
>> https://patchwork.ozlabs.org/patch/870131/.
>
>  If setuptools completely ignores the CFLAGS we pass to it, then it should be
> fixed in setuptools, not in a per-package patch...
>
>
>> Last it looks like the rpath still needs to be set so that the linker
>> can evaluate the shared library's shared library dependency.  I tested
>> the changes below and it builds, plus doesn't leave an rpath in the
>> executable (Thanks Roman).
>> <<<<<<<<<  CUT
>> --- a/package/Makefile.in
>> +++ b/package/Makefile.in
>> @@ -238,7 +238,7 @@ HOST_CPPFLAGS  = -I$(HOST_DIR)/include
>>  HOST_CFLAGS   ?= -O2
>>  HOST_CFLAGS   += $(HOST_CPPFLAGS)
>>  HOST_CXXFLAGS += $(HOST_CFLAGS)
>> -HOST_LDFLAGS  += -L$(HOST_DIR)/lib
>> +HOST_LDFLAGS  += -L$(HOST_DIR)/lib  -Wl,-rpath-link,$(HOST_DIR)/lib
>>
>>  # The macros below are taken from linux 4.11 and adapted slightly.
>>  # Copy more when needed.
>> <<<<<<<<<  CUT
>> The use of LD_LIBRARY_PATH set to the same value also works.  Here's
>> my understanding as pulled from the ld man page.
>>
>> When using ELF or SunOS, one shared library may require another. This
>> happens when an ld -shared link includes a shared library as one of
>> the input files. When the linker encounters such a dependency when
>> doing a non-shared, non-relocateable link, it will automatically try
>> to locate the required shared library and include it in the link, if
>> it is not included explicitly. In such a case, the -rpath-link option
>> specifies the first set of directories to search. The -rpath-link
>> option may specify a sequence of directory names either by specifying
>> a list of names separated by colons, or by appearing multiple times.
>> The linker uses the following search paths to locate required shared
>> libraries.
>> 1) Any directories specified by -rpath-link options.
>> 2) Any directories specified by -rpath options. The difference between
>> -rpath and -rpath-link is that directories specified by -rpath options
>> are included in the executable and used at runtime, whereas the
>> -rpath-link option is only effective at link time.
>> 3) On an ELF system, if the -rpath and rpath-link options were not
>> used, search the contents of the environment variable LD_RUN_PATH.
>> 4) On SunOS, if the -rpath option was not used, search any directories
>> specified using -L options.
>> 5) For a native linker, the contents of the environment variable
>> LD_LIBRARY_PATH.
>> 6) The default directories, normally `/lib' and `/usr/lib'.
>
>  I have read this section of the manpage several times over the years, and now
> is the first time it makes sense to me :-)
>
>  So indeed, for NEEDED libraries within libraries that you link with, ld will
> not look at -L but it will only look at -rpath-link and -rpath (the rest is not
> relevant in our context). So if we remove -rpath, it does make sense that we
> pass -rpath-link instead (we already pass -L as well).
>
>  That said, I still don't understand why it didn't work without -rpath-link.
> Indeed, there *is* a DT_RUNPATH set in the library (at least according to the
> commit log of this patch, since fix-rpath is run at the end of installation),
> but it's set to $ORIGIN/../lib instead of to an absolute path. So, does ld not
> look at DT_RUNPATH? Or does it evaluate $ORIGIN in an unexpected way?
>

It seems $ORIGIN is being evaluated against default lib search dirs
(doesn't include -L paths) so it's going to need the additional base
path from -rpath-link or LD_LIBRARY_PATH (either seemed to behave the
same).

>
>  Finally, note that we said in the Buildroot meeting that this patch is in fact
> probably NOT needed for per-package SDK, because the NEEDed libraries are in
> fact installed in the location specified by the absolute RPATH. It's just weird
> to have an RPATH referring outside of the per-package staging, but it's not an
> actual problem.

Ah now that I reread the initial patch, I see that note at the bottom
about optional.  True.

Matt

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step
  2017-12-01 20:53 ` [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step Thomas Petazzoni
  2017-12-29 15:30   ` Yann E. MORIN
@ 2018-02-22 21:48   ` Matthew Weber
  2018-02-22 22:27     ` Thomas Petazzoni
  1 sibling, 1 reply; 56+ messages in thread
From: Matthew Weber @ 2018-02-22 21:48 UTC (permalink / raw)
  To: buildroot

Thomas,

On Fri, Dec 1, 2017 at 9:53 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:

> +# Some packages use install-staging stuff for install-target
> +$$($(2)_TARGET_INSTALL_TARGET):                $$($(2)_TARGET_INSTALL_STAGING)
> +
>  $(1)-build:            $$($(2)_TARGET_BUILD)
>  $$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE)
>
> @@ -813,6 +822,7 @@ $(1)-reconfigure:   $(1)-clean-for-reconfigure $(1)
>

Should the pkgname-dirclean target also remove its per-package folder ?

Matt

^ permalink raw reply	[flat|nested] 56+ messages in thread

* [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step
  2018-02-22 21:48   ` Matthew Weber
@ 2018-02-22 22:27     ` Thomas Petazzoni
  0 siblings, 0 replies; 56+ messages in thread
From: Thomas Petazzoni @ 2018-02-22 22:27 UTC (permalink / raw)
  To: buildroot

Hello,

On Thu, 22 Feb 2018 15:48:46 -0600, Matthew Weber wrote:

> On Fri, Dec 1, 2017 at 9:53 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
> 
> > +# Some packages use install-staging stuff for install-target
> > +$$($(2)_TARGET_INSTALL_TARGET):                $$($(2)_TARGET_INSTALL_STAGING)
> > +
> >  $(1)-build:            $$($(2)_TARGET_BUILD)
> >  $$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE)
> >
> > @@ -813,6 +822,7 @@ $(1)-reconfigure:   $(1)-clean-for-reconfigure $(1)
> >  
> 
> Should the pkgname-dirclean target also remove its per-package folder ?

Yes, I would think so indeed.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com

^ permalink raw reply	[flat|nested] 56+ messages in thread

end of thread, other threads:[~2018-02-22 22:27 UTC | newest]

Thread overview: 56+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-12-01 20:53 [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni
2017-12-01 20:53 ` [Buildroot] [RFCv3 01/15] pkgconf: use relative path to STAGING_DIR instead of absolute path Thomas Petazzoni
2017-12-29 15:22   ` Yann E. MORIN
2017-12-31 17:24   ` Thomas Petazzoni
2017-12-01 20:53 ` [Buildroot] [RFCv3 02/15] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN Thomas Petazzoni
2017-12-29 15:22   ` Yann E. MORIN
2017-12-31 17:24   ` Thomas Petazzoni
2017-12-01 20:53 ` [Buildroot] [RFCv3 03/15] Makefile, skeleton: move the host skeleton logic to host-skeleton package Thomas Petazzoni
2017-12-03 22:22   ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 04/15] pkg-cmake: install CMake files as part of a package Thomas Petazzoni
2017-12-03 22:34   ` Yann E. MORIN
2017-12-03 22:55     ` Thomas Petazzoni
2017-12-29 15:27       ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 05/15] pkg-generic: add .stamp_installed step Thomas Petazzoni
2017-12-29 15:30   ` Yann E. MORIN
2018-02-22 21:48   ` Matthew Weber
2018-02-22 22:27     ` Thomas Petazzoni
2017-12-01 20:53 ` [Buildroot] [RFCv3 06/15] package/pkg-generic: add the concept of extract dependency Thomas Petazzoni
2017-12-29 15:33   ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 07/15] package/pkg-generic: handle host-tar as an " Thomas Petazzoni
2017-12-02 15:06   ` Yann E. MORIN
2017-12-02 20:19     ` Thomas Petazzoni
2017-12-29 15:34       ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 08/15] package/pkg-generic: handle host-xz " Thomas Petazzoni
2017-12-02 15:08   ` Yann E. MORIN
2017-12-02 20:16     ` Thomas Petazzoni
2017-12-03  8:32       ` Yann E. MORIN
2017-12-03  9:29         ` Thomas Petazzoni
2017-12-03  9:34           ` Yann E. MORIN
2017-12-29 15:36   ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 09/15] package/pkg-generic: handle host-lzip " Thomas Petazzoni
2017-12-02 15:12   ` Yann E. MORIN
2017-12-02 20:13     ` Thomas Petazzoni
2017-12-29 15:36   ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 10/15] package/pkg-generic: handle host-ccache as a regular dependency Thomas Petazzoni
2017-12-29 15:42   ` Yann E. MORIN
2017-12-29 15:48     ` Thomas Petazzoni
2017-12-01 20:53 ` [Buildroot] [RFCv3 11/15] package/pkg-generic: handle host-fakedate " Thomas Petazzoni
2017-12-29 15:50   ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 12/15] core: kill DEPENDENCIES_HOST_PREREQ Thomas Petazzoni
2017-12-29 15:53   ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 13/15] core: change host RPATH handling Thomas Petazzoni
2017-12-29 17:31   ` Yann E. MORIN
2017-12-29 20:20     ` Thomas Petazzoni
2017-12-30 13:56       ` Yann E. MORIN
2018-02-06  0:00       ` Matthew Weber
2018-02-06 22:04   ` Matthew Weber
2018-02-06 23:19     ` Arnout Vandecappelle
2018-02-07  6:30       ` Matthew Weber
2017-12-01 20:53 ` [Buildroot] [RFCv3 14/15] Makefile: evaluate CCACHE and HOST{CC, CXX} at time of use Thomas Petazzoni
2017-12-29 17:32   ` Yann E. MORIN
2017-12-01 20:53 ` [Buildroot] [RFCv3 15/15] core: implement per-package SDK and target Thomas Petazzoni
2018-02-06 14:45   ` Matthew Weber
2018-02-06 22:00     ` Arnout Vandecappelle
2018-02-06 22:39       ` Matthew Weber
2018-01-08 22:41 ` [Buildroot] [RFCv3 00/15] Per-package host/target directory support Thomas Petazzoni

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox