[Buildroot] [PATCH 0 of 5 RFC] dependencies: some improvements

[Buildroot] [PATCH 0 of 5 RFC] dependencies: some improvements

[Thomas De Schampheleire]

This patch series makes some changes in the dependencies code.

   dependencies: check core dependencies before anything else
   dependencies: add function suitable-host-package
   dependencies: build a host-tar if no suitable tar can be found
   dependencies: remove unused lzma checking scripts
   dirs and dependencies should be executed before every package

Although comments on all patches would be fine, I'm mainly troubled with the
last one ('dirs and dependencies...'). Please see the patch description there.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

[Buildroot] [PATCH 1 of 5 RFC] dependencies: check core dependencies before anything else

[Thomas De Schampheleire]

Currently, the dependencies target (that runs dependencies.sh) depends on
DEPENDENCIES_HOST_PREREQ. This means that the dependencies listed in
DEPENDENCIES_HOST_PREREQ (currently host-sstrip if sstrip is selected) are
built *before* the dependencies.sh script is run.

As a result, if e.g. there is no gcc compiler present on the build system, the
dependencies in DEPENDENCIES_HOST_PREREQ will fail to build, and buildroot
will fail non-gracefully.

This patch makes sure that the DEPENDENCIES_HOST_PREREQ are checked *after* the
dependencies.sh script, so that any problem in the build system is reported in
a clean way by dependencies.sh.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 toolchain/dependencies/dependencies.mk |  6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/toolchain/dependencies/dependencies.mk b/toolchain/dependencies/dependencies.mk
--- a/toolchain/dependencies/dependencies.mk
+++ b/toolchain/dependencies/dependencies.mk
@@ -16,12 +16,14 @@ DL_TOOLS = \
 	$(findstring git,$(DL_TOOLS_DEPENDENCIES)) \
 	$(findstring bzr,$(DL_TOOLS_DEPENDENCIES))
 
-dependencies: $(DEPENDENCIES_HOST_PREREQ)
+core-dependencies:
 	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
 		CONFIG_FILE="$(CONFIG_DIR)/.config" \
 		DL_TOOLS="$(DL_TOOLS)" \
 		$(TOPDIR)/toolchain/dependencies/dependencies.sh
 
+dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
+
 dependencies-source:
 
 dependencies-clean:
@@ -35,5 +37,5 @@ dependencies-dirclean:
 # Toplevel Makefile options
 #
 #############################################################
-.PHONY: dependencies
+.PHONY: dependencies core-dependencies
 

[Buildroot] [PATCH 2 of 5 RFC] dependencies: add function suitable-host-package

[Thomas De Schampheleire]

Sometimes, buildroot needs a certain host tool to do its job, e.g. tar. In
many cases, we expect this tool to be present on the host system, but this is
not always the case. Or maybe, the version on the host system is not
suitable, and we need a more recent one.

In some of these cases, instead of bailing out, buildroot could build the
package first (but only if the existing system package is not suitable).

To aid in detecting if a host package is suitable or not, this patch adds a
function suitable-host-package. When called with parameter foo, it will
execute check-host-foo.sh, or if absent, make function check-host-foo. These
should return either the path to the suitable host package, or the empty
string if no suitable package can be found.
The rules to determine whether something is suitable or not is left to the
check-host-foo(.sh) function and depends on foo.

An example usage of suitable-host-package is:
DEPENDENCIES_HOST_PREREQ += $(if $(call suitable-host-package,foo),,host-foo)

To avoid cluttering the existing dependencies.mk file, it includes any
check-host-foo.mk file. These files can be used to hold the appropriate
functions or definitions.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 toolchain/dependencies/dependencies.mk |  10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/toolchain/dependencies/dependencies.mk b/toolchain/dependencies/dependencies.mk
--- a/toolchain/dependencies/dependencies.mk
+++ b/toolchain/dependencies/dependencies.mk
@@ -6,6 +6,16 @@
 ######################################################################
 
 DEPENDENCIES_HOST_PREREQ:=
+
+# suitable-host-pkg: calls check-host-$(1).sh shell script if it exists,
+# otherwise calls check-host-$(1) make function. This script/function
+# should return the path to the suitable host tool, or nothing if no
+# suitable tool was found.
+define suitable-host-package
+$(if $(wildcard toolchain/dependencies/check-host-$(1).sh),$(shell toolchain/dependencies/check-host-$(1).sh),$(call check-host-$(1)))
+endef
+include toolchain/dependencies/check-host-*.mk
+
 ifeq ($(BR2_STRIP_sstrip),y)
 DEPENDENCIES_HOST_PREREQ+=host-sstrip
 endif

[Buildroot] [PATCH 3 of 5 RFC] dependencies: build a host-tar if no suitable tar can be found

[Thomas De Schampheleire]

Some toolchains, like the one built with buildroot itself, use hardlinks (for
example to link between the c++ and g++ binary). Unpacking such a toolchain
with the --strip-components options does not work correctly if the system tar
is too old (<1.17). Even recent releases of RedHat/CentOS still ship with
tar 1.15.

This patch checks for a suitable tar version (tar 1.17+) on the host system,
and adds host-tar to the host dependencies if none can be found.

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a
chicken-and-egg problem), so that all packages use the host-tar if no suitable
tar was found.

Along with this patch, the definition of TAR_STRIP_COMPONENTS was moved to
check-host-tar.mk.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 package/Makefile.package.in              |  12 ------------
 package/tar/tar.mk                       |   4 ++++
 toolchain/dependencies/check-host-tar.mk |  24 ++++++++++++++++++++++++
 toolchain/dependencies/check-host-tar.sh |  29 +++++++++++++++++++++++++++++
 4 files changed, 57 insertions(+), 12 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -235,18 +235,6 @@ define DOWNLOAD
 	exit 1
 endef
 
-# Utility programs used to build packages
-TAR ?= tar
-
-# Automatically detect tar --strip-path/components option
-TAR_STRIP_COMPONENTS := \
-  $(shell $(TAR) --help | grep strip-path > /dev/null ; \
-  if test $$? = 0 ; then \
-   echo '--strip-path' ; \
-  else \
-   echo '--strip-components' ; \
-  fi)
-
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.
 define sep
diff --git a/package/tar/tar.mk b/package/tar/tar.mk
--- a/package/tar/tar.mk
+++ b/package/tar/tar.mk
@@ -8,3 +8,7 @@ TAR_VERSION = 1.26
 TAR_SITE = $(BR2_GNU_MIRROR)/tar
 
 $(eval $(call AUTOTARGETS))
+
+TAR := $(SYSTEM_TAR) # We need a real tar to extract tar
+$(eval $(call AUTOTARGETS,host))
+TAR := $(HOST_TAR)
diff --git a/toolchain/dependencies/check-host-tar.mk b/toolchain/dependencies/check-host-tar.mk
new file mode 100644
--- /dev/null
+++ b/toolchain/dependencies/check-host-tar.mk
@@ -0,0 +1,24 @@
+TAR ?= tar
+SYSTEM_TAR := $(TAR)
+
+ifneq (,$(call suitable-host-package,tar))
+  HOST_TAR := $(TAR)
+
+  # Automatically detect tar --strip-path/components option
+  # --strip-path was renamed to --strip-components in tar 1.15
+  TAR_STRIP_COMPONENTS := \
+    $(shell $(TAR) --help | grep strip-path > /dev/null ; \
+    if test $$? = 0 ; then \
+     echo '--strip-path' ; \
+    else \
+     echo '--strip-components' ; \
+    fi)
+
+else
+  DEPENDENCIES_HOST_PREREQ += host-tar
+  HOST_TAR := $(HOST_DIR)/usr/bin/tar
+  TAR_STRIP_COMPONENTS := --strip-components
+
+endif
+
+TAR := $(HOST_TAR)
diff --git a/toolchain/dependencies/check-host-tar.sh b/toolchain/dependencies/check-host-tar.sh
new file mode 100755
--- /dev/null
+++ b/toolchain/dependencies/check-host-tar.sh
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+tar=`which tar`
+if [ -z "$tar" ]; then
+	return
+fi
+
+# Output of 'tar --version' examples:
+# tar (GNU tar) 1.15.1 
+# tar (GNU tar) 1.25
+version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+bugfix=`echo "$version" | cut -d. -f3`
+
+# Minimal version = 1.17 (previous versions do not correctly unpack archives
+# containing hard-links if the --strip-components option is used).
+major_min=1
+minor_min=17
+if [ $major -gt $major_min ]; then
+	echo $tar
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $tar
+	else
+		false
+		# echo nothing: no suitable tar found
+	fi
+fi

[Buildroot] [PATCH 4 of 5 RFC] dependencies: remove unused lzma checking scripts

[Thomas De Schampheleire]

Files package/lzma/lzmacheck.sh and toolchain/dependencies/check-host-lzma.sh
are present since the very beginning of buildroot, but do not appear to be
used (anymore). Let's remove them.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 package/lzma/lzmacheck.sh                 |   9 ---------
 toolchain/dependencies/check-host-lzma.sh |  13 -------------
 2 files changed, 0 insertions(+), 22 deletions(-)

diff --git a/package/lzma/lzmacheck.sh b/package/lzma/lzmacheck.sh
deleted file mode 100755
--- a/package/lzma/lzmacheck.sh
+++ /dev/null
@@ -1,9 +0,0 @@
-#!/bin/sh
-
-bin=$(toolchain/dependencies/check-host-lzma.sh)
-if [ "x$bin" = "x" ] ; then
-  echo build-lzma-host-binary
-else
-  echo use-lzma-host-binary
-fi
-
diff --git a/toolchain/dependencies/check-host-lzma.sh b/toolchain/dependencies/check-host-lzma.sh
deleted file mode 100755
--- a/toolchain/dependencies/check-host-lzma.sh
+++ /dev/null
@@ -1,13 +0,0 @@
-#!/bin/sh
-
-ok=""
-
-for bin in /usr/bin/lzma $LZMA
-do
-# TODO: add check for proper functionality here..
-  $bin --version > /dev/null 2>&1 && ok="$bin"
-  if test "x$ok" != "x" ; then
-    break
-  fi
-done
-echo "$ok"

[Buildroot] [PATCH 5 of 5 RFC] dirs and dependencies should be executed before every package

[Thomas De Schampheleire]

This patch attempts to solve two problems:
1. download directory not created before downloading
If the first download is handled by wget, then the download directory will be
created by wget itself and all is fine. However, if the first download is
handled by another tool, e.g. git/bzr/hg/scp etc., then there is no automatic
creation of this directory.

In most cases, you just execute 'make', and the 'dirs' target will be executed
automatically. But, if you give an explicit target the first time, e.g. 'make
ctng-menuconfig', there is no direct dependency to 'dirs'

A similar problem was detected and fixed earlier: see
'source' target should depend on 'dirs'
http://git.buildroot.net/buildroot/commit/?id=4eb982cf90f16927442f4e7692ce51dd5d21f37b
In that case, the fix was made specifically for the 'source' target, but in
fact the problem is generic.

2. dirs not created and dependencies not met before packages are built
The second problem is new. If there are dependencies that are needed to build
packages, e.g. a dependency to host-tar, then these should be executed before
trying to build any other package. Moreover, the build directory etc. should
also be made before building any package. This is not always the case.

For example, ctng-menuconfig depends on 'host-gawk'. If you run
ctng-menuconfig directly, host-gawk will be built first, but the general
'dirs' and 'dependencies' targets will be skipped! This is problematic in the
mentioned 'tar' case.
The same would occur if you explicitly execute 'make foo' as first command.


To solve both problems, this patch changes the gentargets 'foo-source' target
to explicitly depend on 'dirs' and 'dependencies'. This way, every package
built with gentargets/autotargets/cmake will be built after dirs and
dependencies.

But, certain problems remain:
* When a dependency package is not using gentargets.
* In some cases, this patch introduces a circular make dependency, because e.g.
  dependencies <-- host-tar
  <any package>  <-- dirs dependencies

Although make gracefully drops this circular dependency when present, it is
not nice to have. Maybe we need to treat the 'tar' case in a special manner. For
example by adding only a dependency to host-tar (and not 'dependencies' in
general) to the extraction step for all packages, except for host-tar itself.
Or, alternatively, we may decide that we won't support direct calls to 'make
ctng-menuconfig' or similar, without the user having called 'make dependencies'
first.

With respect to the downloads problem, we may also want to change it
differently: either let the download step depend on DL_DIR, or have the download
helpers create the download directory first.

Any thoughts are welcome!

[not to be merged in current state]
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

---
 package/Makefile.package.in |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -558,7 +558,7 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
-$(1)-source:		$$($(2)_TARGET_SOURCE)
+$(1)-source:		dirs dependencies $$($(2)_TARGET_SOURCE)
 else
 # In the package override case, the sequence of steps
 #  source, by rsyncing
@@ -571,7 +571,7 @@ else
 
 $(1)-rsync:		$$($(2)_TARGET_RSYNC)
 
-$(1)-source:		$$($(2)_TARGET_RSYNC_SOURCE)
+$(1)-source:		dirs dependencies $$($(2)_TARGET_RSYNC_SOURCE)
 endif
 
 $(1)-show-depends:

[Buildroot] [PATCH 1 of 5 RFC] dependencies: check core dependencies before anything else

[Arnout Vandecappelle]

On Thursday 10 November 2011 19:31:51 Thomas De Schampheleire wrote:
> Currently, the dependencies target (that runs dependencies.sh) depends on
> DEPENDENCIES_HOST_PREREQ. This means that the dependencies listed in
> DEPENDENCIES_HOST_PREREQ (currently host-sstrip if sstrip is selected) are
> built *before* the dependencies.sh script is run.
> 
> As a result, if e.g. there is no gcc compiler present on the build system,
> the dependencies in DEPENDENCIES_HOST_PREREQ will fail to build, and
> buildroot will fail non-gracefully.
> 
> This patch makes sure that the DEPENDENCIES_HOST_PREREQ are checked *after*
> the dependencies.sh script, so that any problem in the build system is
> reported in a clean way by dependencies.sh.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

> 
> ---
>  toolchain/dependencies/dependencies.mk |  6 ++++--
>  1 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/toolchain/dependencies/dependencies.mk
> b/toolchain/dependencies/dependencies.mk ---
> a/toolchain/dependencies/dependencies.mk
> +++ b/toolchain/dependencies/dependencies.mk
> @@ -16,12 +16,14 @@ DL_TOOLS = \
>  	$(findstring git,$(DL_TOOLS_DEPENDENCIES)) \
>  	$(findstring bzr,$(DL_TOOLS_DEPENDENCIES))
> 
> -dependencies: $(DEPENDENCIES_HOST_PREREQ)
> +core-dependencies:
>  	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
>  		CONFIG_FILE="$(CONFIG_DIR)/.config" \
>  		DL_TOOLS="$(DL_TOOLS)" \
>  		$(TOPDIR)/toolchain/dependencies/dependencies.sh
> 
> +dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
> +
>  dependencies-source:
> 
>  dependencies-clean:
> @@ -35,5 +37,5 @@ dependencies-dirclean:
>  # Toplevel Makefile options
>  #
>  #############################################################
> -.PHONY: dependencies
> +.PHONY: dependencies core-dependencies
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] [PATCH 4 of 5 RFC] dependencies: remove unused lzma checking scripts

[Arnout Vandecappelle]

On Thursday 10 November 2011 19:32:00 Thomas De Schampheleire wrote:
> Files package/lzma/lzmacheck.sh and
> toolchain/dependencies/check-host-lzma.sh are present since the very
> beginning of buildroot, but do not appear to be used (anymore). Let's
> remove them.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

> 
> ---
>  package/lzma/lzmacheck.sh                 |   9 ---------
>  toolchain/dependencies/check-host-lzma.sh |  13 -------------
>  2 files changed, 0 insertions(+), 22 deletions(-)
> 
> diff --git a/package/lzma/lzmacheck.sh b/package/lzma/lzmacheck.sh
> deleted file mode 100755
> --- a/package/lzma/lzmacheck.sh
> +++ /dev/null
> @@ -1,9 +0,0 @@
> -#!/bin/sh
> -
> -bin=$(toolchain/dependencies/check-host-lzma.sh)
> -if [ "x$bin" = "x" ] ; then
> -  echo build-lzma-host-binary
> -else
> -  echo use-lzma-host-binary
> -fi
> -
> diff --git a/toolchain/dependencies/check-host-lzma.sh
> b/toolchain/dependencies/check-host-lzma.sh deleted file mode 100755
> --- a/toolchain/dependencies/check-host-lzma.sh
> +++ /dev/null
> @@ -1,13 +0,0 @@
> -#!/bin/sh
> -
> -ok=""
> -
> -for bin in /usr/bin/lzma $LZMA
> -do
> -# TODO: add check for proper functionality here..
> -  $bin --version > /dev/null 2>&1 && ok="$bin"
> -  if test "x$ok" != "x" ; then
> -    break
> -  fi
> -done
> -echo "$ok"
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] [PATCH 2 of 5 RFC] dependencies: add function suitable-host-package

[Arnout Vandecappelle]

On Thursday 10 November 2011 19:31:54 Thomas De Schampheleire wrote:
> Sometimes, buildroot needs a certain host tool to do its job, e.g. tar. In
> many cases, we expect this tool to be present on the host system, but this is
> not always the case. Or maybe, the version on the host system is not
> suitable, and we need a more recent one.
> 
> In some of these cases, instead of bailing out, buildroot could build the
> package first (but only if the existing system package is not suitable).
> 
> To aid in detecting if a host package is suitable or not, this patch adds a
> function suitable-host-package. When called with parameter foo, it will
> execute check-host-foo.sh, or if absent, make function check-host-foo. These
> should return either the path to the suitable host package, or the empty
> string if no suitable package can be found.
> The rules to determine whether something is suitable or not is left to the
> check-host-foo(.sh) function and depends on foo.
> 
> An example usage of suitable-host-package is:
> DEPENDENCIES_HOST_PREREQ += $(if $(call suitable-host-package,foo),,host-foo)
> 
> To avoid cluttering the existing dependencies.mk file, it includes any
> check-host-foo.mk file. These files can be used to hold the appropriate
> functions or definitions.

While you're at it, perhaps move the toolchain/dependencies directory to 
support/dependencies?  And also it can be included from the top-level
Makefile instead of the toolchain makefiles.  Not that this is in any way
related to this patch :-)

> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

[ I just realized that I agree with Thomas DS's conclusion that we should 
use Reviewed-by, not Acked-by... So my previous Acks should be Reviews.]

> 
> ---
>  toolchain/dependencies/dependencies.mk |  10 ++++++++++
>  1 files changed, 10 insertions(+), 0 deletions(-)
> 
> diff --git a/toolchain/dependencies/dependencies.mk b/toolchain/dependencies/dependencies.mk
> --- a/toolchain/dependencies/dependencies.mk
> +++ b/toolchain/dependencies/dependencies.mk
> @@ -6,6 +6,16 @@
>  ######################################################################
>  
>  DEPENDENCIES_HOST_PREREQ:=
> +
> +# suitable-host-pkg: calls check-host-$(1).sh shell script if it exists,
> +# otherwise calls check-host-$(1) make function. This script/function
> +# should return the path to the suitable host tool, or nothing if no
> +# suitable tool was found.
> +define suitable-host-package
> +$(if $(wildcard toolchain/dependencies/check-host-$(1).sh),$(shell toolchain/dependencies/check-host-$(1).sh),$(call check-host-$(1)))

 I would remove the possibility of having check-host-xxx as a function
unless there is a use case for it.  It's fairly easy to add later, and this
function becomes much more readable without it:
+$(shell toolchain/dependencies/check-host-$(1).sh)

> +endef
> +include toolchain/dependencies/check-host-*.mk
> +
>  ifeq ($(BR2_STRIP_sstrip),y)
>  DEPENDENCIES_HOST_PREREQ+=host-sstrip
>  endif
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111115/738a076b/attachment-0001.html>

[Buildroot] [PATCH 3 of 5 RFC] dependencies: build a host-tar if no suitable tar can be found

[Arnout Vandecappelle]

On Thursday 10 November 2011 19:31:58 Thomas De Schampheleire wrote:
> Some toolchains, like the one built with buildroot itself, use hardlinks (for
> example to link between the c++ and g++ binary). Unpacking such a toolchain
> with the --strip-components options does not work correctly if the system tar
> is too old (<1.17). Even recent releases of RedHat/CentOS still ship with
> tar 1.15.
> 
> This patch checks for a suitable tar version (tar 1.17+) on the host system,
> and adds host-tar to the host dependencies if none can be found.
> 
> TAR is redefined to HOST_TAR, except when extracting host-tar (this is a
> chicken-and-egg problem), so that all packages use the host-tar if no suitable
> tar was found.
> 
> Along with this patch, the definition of TAR_STRIP_COMPONENTS was moved to
> check-host-tar.mk.

 Since tar >= 1.17 is required, this can be removed.

> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

> 
> ---
>  package/Makefile.package.in              |  12 ------------
>  package/tar/tar.mk                       |   4 ++++
>  toolchain/dependencies/check-host-tar.mk |  24 ++++++++++++++++++++++++
>  toolchain/dependencies/check-host-tar.sh |  29 +++++++++++++++++++++++++++++
>  4 files changed, 57 insertions(+), 12 deletions(-)
> 
> diff --git a/package/Makefile.package.in b/package/Makefile.package.in
> --- a/package/Makefile.package.in
> +++ b/package/Makefile.package.in
> @@ -235,18 +235,6 @@ define DOWNLOAD
>  	exit 1
>  endef
>  
> -# Utility programs used to build packages
> -TAR ?= tar
> -
> -# Automatically detect tar --strip-path/components option
> -TAR_STRIP_COMPONENTS := \
> -  $(shell $(TAR) --help | grep strip-path > /dev/null ; \
> -  if test $$? = 0 ; then \
> -   echo '--strip-path' ; \
> -  else \
> -   echo '--strip-components' ; \
> -  fi)
> -
>  # Needed for the foreach loops to loop over the list of hooks, so that
>  # each hook call is properly separated by a newline.
>  define sep
> diff --git a/package/tar/tar.mk b/package/tar/tar.mk
> --- a/package/tar/tar.mk
> +++ b/package/tar/tar.mk
> @@ -8,3 +8,7 @@ TAR_VERSION = 1.26
>  TAR_SITE = $(BR2_GNU_MIRROR)/tar
>  
>  $(eval $(call AUTOTARGETS))
> +
> +TAR := $(SYSTEM_TAR) # We need a real tar to extract tar

 Here you still need the TAR_STRIP_COMPONENTS, because SYSTEM_TAR
may be < 1.15.

> +$(eval $(call AUTOTARGETS,host))
> +TAR := $(HOST_TAR)
> diff --git a/toolchain/dependencies/check-host-tar.mk b/toolchain/dependencies/check-host-tar.mk
> new file mode 100644
> --- /dev/null
> +++ b/toolchain/dependencies/check-host-tar.mk
> @@ -0,0 +1,24 @@
> +TAR ?= tar
> +SYSTEM_TAR := $(TAR)
 Add a comment why this has to be :=, e.g.

# TAR will be overridden again when tar itself is extracted, so we must
# finalize the value of SYSTEM_TAR.

> +
> +ifneq (,$(call suitable-host-package,tar))
> +  HOST_TAR := $(TAR)
 I would write
  HOST_TAR = $(SYSTEM_TAR)
(no := is needed in that case).

> +
> +  # Automatically detect tar --strip-path/components option
> +  # --strip-path was renamed to --strip-components in tar 1.15
> +  TAR_STRIP_COMPONENTS := \
> +    $(shell $(TAR) --help | grep strip-path > /dev/null ; \
> +    if test $$? = 0 ; then \
> +     echo '--strip-path' ; \
> +    else \
> +     echo '--strip-components' ; \
> +    fi)
> +
> +else
> +  DEPENDENCIES_HOST_PREREQ += host-tar
> +  HOST_TAR := $(HOST_DIR)/usr/bin/tar
> +  TAR_STRIP_COMPONENTS := --strip-components

 = is fine for these two assignments.

> +
> +endif
> +
> +TAR := $(HOST_TAR)

 Also here = should be OK.

> diff --git a/toolchain/dependencies/check-host-tar.sh b/toolchain/dependencies/check-host-tar.sh
> new file mode 100755
> --- /dev/null
> +++ b/toolchain/dependencies/check-host-tar.sh
> @@ -0,0 +1,29 @@
> +#!/bin/sh
> +
> +tar=`which tar`
> +if [ -z "$tar" ]; then
> +	return
> +fi
> +
> +# Output of 'tar --version' examples:
> +# tar (GNU tar) 1.15.1 
> +# tar (GNU tar) 1.25
> +version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'`
> +major=`echo "$version" | cut -d. -f1`
> +minor=`echo "$version" | cut -d. -f2`
> +bugfix=`echo "$version" | cut -d. -f3`
> +
> +# Minimal version = 1.17 (previous versions do not correctly unpack archives
> +# containing hard-links if the --strip-components option is used).
> +major_min=1
> +minor_min=17
> +if [ $major -gt $major_min ]; then
> +	echo $tar
> +else
> +	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
> +		echo $tar
> +	else
> +		false
> +		# echo nothing: no suitable tar found
> +	fi
> +fi
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] [PATCH 5 of 5 RFC] dirs and dependencies should be executed before every package

[Arnout Vandecappelle]

On Thursday 10 November 2011 19:32:02 Thomas De Schampheleire wrote:
[snip]
> To solve both problems, this patch changes the gentargets 'foo-source' target
> to explicitly depend on 'dirs' and 'dependencies'. This way, every package
> built with gentargets/autotargets/cmake will be built after dirs and
> dependencies.
> 
> But, certain problems remain:
> * When a dependency package is not using gentargets.

 packets not using gentargets are deprecated.  AFAICS there are only 14 left:
cups/cups.mk
customize/customize.mk
fis/fis.mk
games/doom-wad/doom-wads.mk
gettext/gettext.mk
java/concierge/concierge.mk
microperl/microperl.mk
netkitbase/netkitbase.mk
netkittelnet/netkittelnet.mk
newt/newt.mk
tinyhttpd/tinyhttpd.mk
ttcp/ttcp.mk
uemacs/uemacs.mk
vpnc/vpnc.mk
xfsprogs/xfsprogs.mk

 The only worrysome one is gettext.

> * In some cases, this patch introduces a circular make dependency, because e.g.
>   dependencies <-- host-tar
>   <any package>  <-- dirs dependencies

 This is of course much more important.

 For dirs it certainly isn't a problem.

 For the dependencies, I'm actually thinking that a much better solution
would be to add a dependencies step at the first build, that checks for all
potentially needed host packages.  It creates a hostpackages.mk file that
contains variable assignments for all potential dependencies.  Specifying
a rule for hostpackages.mk in the top-level Makefile will make sure that it is
generated if it doesn't exist.  This has the advantage that dependencies
have to be checked only at the first build and not every single time.  But
it is a lot more work to implement of course...

 Regards,
 Arnout
-- 
Arnout Vandecappelle                               arnout@mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

[Buildroot] [PATCH 1 of 5 RFC] dependencies: check core dependencies before anything else

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:

 Thomas> Currently, the dependencies target (that runs dependencies.sh)
 Thomas> depends on DEPENDENCIES_HOST_PREREQ. This means that the
 Thomas> dependencies listed in DEPENDENCIES_HOST_PREREQ (currently
 Thomas> host-sstrip if sstrip is selected) are built *before* the
 Thomas> dependencies.sh script is run.

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 4 of 5 RFC] dependencies: remove unused lzma checking scripts

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:

 Thomas> Files package/lzma/lzmacheck.sh and
 Thomas> toolchain/dependencies/check-host-lzma.sh are present since the
 Thomas> very beginning of buildroot, but do not appear to be used
 Thomas> (anymore). Let's remove them.

Committed, thanks.

-- 
Bye, Peter Korsgaard