[Buildroot] [PATCH 0 of 3 v2] dependencies: some improvements

[Buildroot] [PATCH 0 of 3 v2] dependencies: some improvements

[Thomas De Schampheleire]

The original series had the following patches:
 1 dependencies: check core dependencies before anything else
 2 dependencies: add function suitable-host-package
 3 dependencies: build a host-tar if no suitable tar can be found
 4 dependencies: remove unused lzma checking scripts
 5 dirs and dependencies should be executed before every package

Patches 1 and 4 are already merged (thanks Peter).

Patch 5 tried to fix some non-default scenarios, like running 'make
ctng-menuconfig'
without having run 'make dependencies' or plain 'make' first. But, this
attempt mainly suffered from the introduction of circular dependencies, which
are not easy to solve. I therefore decided not to support these scenarios in
this series.

Patch 2 and 3 are updated in this series based on the input from Arnout
(thanks!)

Additionally, a new patch moves toolchain/dependencies to
support/dependencies.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

[Buildroot] [PATCH 1 of 3 v2] dependencies: add function suitable-host-package

[Thomas De Schampheleire]

Sometimes, buildroot needs a certain host tool to do its job, e.g. tar. In
many cases, we expect this tool to be present on the host system, but this is
not always the case. Or maybe, the version on the host system is not
suitable, and we need a more recent one.

In some of these cases, instead of bailing out, buildroot could build the
package first (but only if the existing system package is not suitable).

To aid in detecting if a host package is suitable or not, this patch adds a
function suitable-host-package. When called with parameter foo, it will
execute check-host-foo.sh. This script should return either the path to the
suitable host package, or the empty string if no suitable package can be found.
The rules to determine whether something is suitable or not is left to
check-host-foo.sh and depends on foo.

An example usage of suitable-host-package is:
DEPENDENCIES_HOST_PREREQ += $(if $(call suitable-host-package,foo),,host-foo)

To avoid cluttering the existing dependencies.mk file, it includes any
check-host-foo.mk file. These files can be used to hold appropriate
dependency-related actions for foo.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

---
 toolchain/dependencies/dependencies.mk |  9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

diff --git a/toolchain/dependencies/dependencies.mk b/toolchain/dependencies/dependencies.mk
--- a/toolchain/dependencies/dependencies.mk
+++ b/toolchain/dependencies/dependencies.mk
@@ -6,6 +6,15 @@
 ######################################################################
 
 DEPENDENCIES_HOST_PREREQ:=
+
+# suitable-host-pkg: calls check-host-$(1).sh shell script, which
+# should return the path to the suitable host tool, or nothing if no
+# suitable tool was found.
+define suitable-host-package
+$(shell toolchain/dependencies/check-host-$(1).sh)
+endef
+include toolchain/dependencies/check-host-*.mk
+
 ifeq ($(BR2_STRIP_sstrip),y)
 DEPENDENCIES_HOST_PREREQ+=host-sstrip
 endif

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Thomas De Schampheleire]

Some toolchains, like the one built with buildroot itself, use hardlinks (for
example to link between the c++ and g++ binary). Unpacking such a toolchain
with the --strip-components options does not work correctly if the system tar
is too old (<1.17). Even recent releases of RedHat/CentOS still ship with
tar 1.15.

This patch checks for a suitable tar version (tar 1.17+) on the host system,
and adds host-tar to the host dependencies if none can be found.

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a
chicken-and-egg problem), so that all packages use the host-tar if no suitable
tar was found.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

---
 package/Makefile.package.in              |  12 ------------
 package/tar/tar.mk                       |   6 ++++++
 toolchain/dependencies/check-host-tar.mk |  32 ++++++++++++++++++++++++++++++++
 toolchain/dependencies/check-host-tar.sh |  29 +++++++++++++++++++++++++++++
 4 files changed, 67 insertions(+), 12 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -235,18 +235,6 @@ define DOWNLOAD
 	exit 1
 endef
 
-# Utility programs used to build packages
-TAR ?= tar
-
-# Automatically detect tar --strip-path/components option
-TAR_STRIP_COMPONENTS := \
-  $(shell $(TAR) --help | grep strip-path > /dev/null ; \
-  if test $$? = 0 ; then \
-   echo '--strip-path' ; \
-  else \
-   echo '--strip-components' ; \
-  fi)
-
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.
 define sep
diff --git a/package/tar/tar.mk b/package/tar/tar.mk
--- a/package/tar/tar.mk
+++ b/package/tar/tar.mk
@@ -13,3 +13,9 @@ TAR_DEPENDENCIES += busybox
 endif
 
 $(eval $(call AUTOTARGETS))
+
+TAR = $(SYSTEM_TAR) # We need a real tar to extract tar
+TAR_STRIP_COMPONENTS = $(SYSTEM_TAR_STRIP_COMPONENTS)
+$(eval $(call AUTOTARGETS,host))
+TAR = $(HOST_TAR)
+TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
diff --git a/toolchain/dependencies/check-host-tar.mk b/toolchain/dependencies/check-host-tar.mk
new file mode 100644
--- /dev/null
+++ b/toolchain/dependencies/check-host-tar.mk
@@ -0,0 +1,32 @@
+TAR ?= tar
+
+# To extract tar, we always need a real (system) tar. Provide the necessary
+# definitions here to avoid cluttering package/tar/tar.mk .
+# TAR will be overridden again when the tar .mk file is included, so we must
+# finalize the value of SYSTEM_TAR by using :=
+SYSTEM_TAR := $(TAR)
+# Automatically detect tar --strip-path/components option
+# --strip-path was renamed to --strip-components in tar 1.15
+SYSTEM_TAR_STRIP_COMPONENTS := \
+    $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
+    if test $$? = 0 ; then \
+     echo '--strip-path' ; \
+    else \
+     echo '--strip-components' ; \
+    fi)
+
+# Determine correct HOST_TAR
+ifneq (,$(call suitable-host-package,tar))
+  HOST_TAR = $(SYSTEM_TAR)
+else
+  DEPENDENCIES_HOST_PREREQ += host-tar
+  HOST_TAR = $(HOST_DIR)/usr/bin/tar
+endif
+
+# Since HOST_TAR is at least 1.17, it will certainly support --strip-components
+HOST_TAR_STRIP_COMPONENTS = --strip-components
+
+# Set definitions used by all .mk files included after this one and before
+# package/tar/tar.mk
+TAR = $(HOST_TAR)
+TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
diff --git a/toolchain/dependencies/check-host-tar.sh b/toolchain/dependencies/check-host-tar.sh
new file mode 100755
--- /dev/null
+++ b/toolchain/dependencies/check-host-tar.sh
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+tar=`which tar`
+if [ -z "$tar" ]; then
+	return
+fi
+
+# Output of 'tar --version' examples:
+# tar (GNU tar) 1.15.1 
+# tar (GNU tar) 1.25
+version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+bugfix=`echo "$version" | cut -d. -f3`
+
+# Minimal version = 1.17 (previous versions do not correctly unpack archives
+# containing hard-links if the --strip-components option is used).
+major_min=1
+minor_min=17
+if [ $major -gt $major_min ]; then
+	echo $tar
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $tar
+	else
+		false
+		# echo nothing: no suitable tar found
+	fi
+fi

[Buildroot] [PATCH 3 of 3 v2] Move toolchain/dependencies to support/dependencies

[Thomas De Schampheleire]

As suggested by Arnout Vandecappelle, move toolchain/dependencies to
support/dependencies, as it really is not toolchain-specific anymore.

Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -298,6 +298,8 @@ include package/Makefile.in
 
 all: world
 
+include support/dependencies/dependencies.mk
+
 # We also need the various per-package makefiles, which also add
 # each selected package to TARGETS if that package was selected
 # in the .config file.
diff --git a/toolchain/dependencies/check-host-tar.mk b/support/dependencies/check-host-tar.mk
rename from toolchain/dependencies/check-host-tar.mk
rename to support/dependencies/check-host-tar.mk
diff --git a/toolchain/dependencies/check-host-tar.sh b/support/dependencies/check-host-tar.sh
rename from toolchain/dependencies/check-host-tar.sh
rename to support/dependencies/check-host-tar.sh
diff --git a/toolchain/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
rename from toolchain/dependencies/dependencies.mk
rename to support/dependencies/dependencies.mk
--- a/toolchain/dependencies/dependencies.mk
+++ b/support/dependencies/dependencies.mk
@@ -11,9 +11,9 @@ DEPENDENCIES_HOST_PREREQ:=
 # should return the path to the suitable host tool, or nothing if no
 # suitable tool was found.
 define suitable-host-package
-$(shell toolchain/dependencies/check-host-$(1).sh)
+$(shell support/dependencies/check-host-$(1).sh)
 endef
-include toolchain/dependencies/check-host-*.mk
+include support/dependencies/check-host-*.mk
 
 ifeq ($(BR2_STRIP_sstrip),y)
 DEPENDENCIES_HOST_PREREQ+=host-sstrip
@@ -29,7 +29,7 @@ core-dependencies:
 	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
 		CONFIG_FILE="$(CONFIG_DIR)/.config" \
 		DL_TOOLS="$(DL_TOOLS)" \
-		$(TOPDIR)/toolchain/dependencies/dependencies.sh
+		$(TOPDIR)/support/dependencies/dependencies.sh
 
 dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
 
diff --git a/toolchain/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
rename from toolchain/dependencies/dependencies.sh
rename to support/dependencies/dependencies.sh
diff --git a/toolchain/toolchain-buildroot.mk b/toolchain/toolchain-buildroot.mk
--- a/toolchain/toolchain-buildroot.mk
+++ b/toolchain/toolchain-buildroot.mk
@@ -1,6 +1,5 @@
 # Include files required for the internal toolchain backend
 
-include toolchain/dependencies/dependencies.mk
 include toolchain/elf2flt/elf2flt.mk
 include toolchain/gcc/gcc-uclibc-4.x.mk
 include toolchain/gdb/gdb.mk
diff --git a/toolchain/toolchain-crosstool-ng.mk b/toolchain/toolchain-crosstool-ng.mk
--- a/toolchain/toolchain-crosstool-ng.mk
+++ b/toolchain/toolchain-crosstool-ng.mk
@@ -2,7 +2,6 @@
 
 # Explicit ordering:
 include toolchain/helpers.mk
-include toolchain/dependencies/dependencies.mk
 include toolchain/elf2flt/elf2flt.mk
 include toolchain/gcc/gcc-uclibc-4.x.mk
 include toolchain/gdb/gdb.mk
diff --git a/toolchain/toolchain-external.mk b/toolchain/toolchain-external.mk
--- a/toolchain/toolchain-external.mk
+++ b/toolchain/toolchain-external.mk
@@ -1,7 +1,6 @@
 # Required includes for the external toolchain backend
 
 include toolchain/helpers.mk
-include toolchain/dependencies/dependencies.mk
 include toolchain/elf2flt/elf2flt.mk
 include toolchain/gcc/gcc-uclibc-4.x.mk
 include toolchain/gdb/gdb.mk

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Thomas Petazzoni]

Le Wed, 23 Nov 2011 14:59:59 +0100,
Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :

> Some toolchains, like the one built with buildroot itself, use
> hardlinks (for example to link between the c++ and g++ binary).
> Unpacking such a toolchain with the --strip-components options does
> not work correctly if the system tar is too old (<1.17). Even recent
> releases of RedHat/CentOS still ship with tar 1.15.
> 
> This patch checks for a suitable tar version (tar 1.17+) on the host
> system, and adds host-tar to the host dependencies if none can be
> found.
> 
> TAR is redefined to HOST_TAR, except when extracting host-tar (this
> is a chicken-and-egg problem), so that all packages use the host-tar
> if no suitable tar was found.

I am still puzzled by the fact that tar is needed to extract this
host-tar, so there's like a chicken-and-egg problem. I understand that
the real host tar is used to extract the host-tar, which is then use to
extract all other packages. But that sounds really nasty to me.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 3 of 3 v2] Move toolchain/dependencies to support/dependencies

[Thomas Petazzoni]

Le Wed, 23 Nov 2011 15:00:02 +0100,
Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :

> As suggested by Arnout Vandecappelle, move toolchain/dependencies to
> support/dependencies, as it really is not toolchain-specific anymore.
> 
> Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>

Acked-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Michael S. Zick]

On Wed November 23 2011, Thomas Petazzoni wrote:
> Le Wed, 23 Nov 2011 14:59:59 +0100,
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :
> 
> > Some toolchains, like the one built with buildroot itself, use
> > hardlinks (for example to link between the c++ and g++ binary).
> > Unpacking such a toolchain with the --strip-components options does
> > not work correctly if the system tar is too old (<1.17). Even recent
> > releases of RedHat/CentOS still ship with tar 1.15.
> > 
> > This patch checks for a suitable tar version (tar 1.17+) on the host
> > system, and adds host-tar to the host dependencies if none can be
> > found.
> > 
> > TAR is redefined to HOST_TAR, except when extracting host-tar (this
> > is a chicken-and-egg problem), so that all packages use the host-tar
> > if no suitable tar was found.
> 
> I am still puzzled by the fact that tar is needed to extract this
> host-tar, so there's like a chicken-and-egg problem. I understand that
> the real host tar is used to extract the host-tar, which is then use to
> extract all other packages. But that sounds really nasty to me.
> 

Maybe:
ftp://ftp.gnu.org/gnu/tar/tar-1.26.cpio.gz

Then you only need the more stable cpio and gzip utilities.
Even Busybox can provide those.

Mike
> Thomas

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Thomas De Schampheleire]

On Wed, Nov 23, 2011 at 3:24 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Le Wed, 23 Nov 2011 14:59:59 +0100,
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :
>
>> Some toolchains, like the one built with buildroot itself, use
>> hardlinks (for example to link between the c++ and g++ binary).
>> Unpacking such a toolchain with the --strip-components options does
>> not work correctly if the system tar is too old (<1.17). Even recent
>> releases of RedHat/CentOS still ship with tar 1.15.
>>
>> This patch checks for a suitable tar version (tar 1.17+) on the host
>> system, and adds host-tar to the host dependencies if none can be
>> found.
>>
>> TAR is redefined to HOST_TAR, except when extracting host-tar (this
>> is a chicken-and-egg problem), so that all packages use the host-tar
>> if no suitable tar was found.
>
> I am still puzzled by the fact that tar is needed to extract this
> host-tar, so there's like a chicken-and-egg problem. I understand that
> the real host tar is used to extract the host-tar, which is then use to
> extract all other packages. But that sounds really nasty to me.

To summarize the problem about tar:
* some distributions (like RedHat/CentOS) still ship an old tar
version (1.15) which is known to have problems when
--tar-strip-components is used on tarballs with hardlinks.
* the problem was first seen when tar'ing up a buildroot toolchain and
use it as an external toolchain later.
* since the problem is not restricted to toolchains, we decided to fix
the problem in a common way, by redefining TAR.

I see several solutions, some more desirable than others:
1. don't change anything and force users of RedHat/CentOS to build
their own tar that is recent enough. This is by far the easiest from a
buildroot perspective, but least user-friendly.
2. use the proposed patch
2. don't download tar as .tar.gz/.tar.bz2, but rather as .cpio.gz
(available on the official mirrors since 1.16.1). It requires cpio and
gzip to be present, but that is already checked in dependencies.sh.
We'd need to update gentargets to cater for .cpio, but that is doable.

Best regards,
Thomas

[Buildroot] [PATCH 3 of 3 v2] Move toolchain/dependencies to support/dependencies

[Arnout Vandecappelle]

On Wednesday 23 November 2011 15:25:26 Thomas Petazzoni wrote:
> Le Wed, 23 Nov 2011 15:00:02 +0100,
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :
> 
> > As suggested by Arnout Vandecappelle, move toolchain/dependencies to
> > support/dependencies, as it really is not toolchain-specific anymore.
> > 
> > Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
> 
> Acked-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/0b7d8209/attachment.html>

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Arnout Vandecappelle]

On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
> > TAR is redefined to HOST_TAR, except when extracting host-tar (this
> > is a chicken-and-egg problem), so that all packages use the host-tar
> > if no suitable tar was found.
> 
> I am still puzzled by the fact that tar is needed to extract this
> host-tar, so there's like a chicken-and-egg problem. I understand that
> the real host tar is used to extract the host-tar, which is then use to
> extract all other packages. But that sounds really nasty to me.

 It's not so strange, it's like a compiler's canadian build.  You need a
compiler to compile the compiler, right?

 The way it is solved for tar is OK for me.  I think adding support for
cpio archives would just complicate things, because then host-tar
can't use GENTARGETS anymore.

 Regards,
 Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/d726b617/attachment-0001.html>

[Buildroot] [PATCH 1 of 3 v2] dependencies: add function suitable-host-package

[Arnout Vandecappelle]

On Wednesday 23 November 2011 14:59:50 Thomas De Schampheleire wrote:
> +# suitable-host-pkg: calls check-host-$(1).sh shell script, which
> +# should return the path to the suitable host tool, or nothing if no
> +# suitable tool was found.
> +define suitable-host-package
> +$(shell toolchain/dependencies/check-host-$(1).sh)
> +endef

 I should have noticed this in the first review, but:  If the function is
this simple, it becomes pointless.  Especially because other check-host-*
mk files may want to implement the checking in a different way, e.g.
$(eval $(shell ...)).

> +include toolchain/dependencies/check-host-*.mk

 And if this is the only thing remaining, perhaps it's better to merge
this patch with the next one :-)


 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/4d32f6c6/attachment.html>

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Arnout Vandecappelle]

On Wednesday 23 November 2011 14:59:59 Thomas De Schampheleire wrote:
> diff --git a/toolchain/dependencies/check-host-tar.mk b/toolchain/dependencies/check-host-tar.mk
> new file mode 100644
> --- /dev/null
> +++ b/toolchain/dependencies/check-host-tar.mk
> @@ -0,0 +1,32 @@
> +TAR ?= tar
> +
> +# To extract tar, we always need a real (system) tar. Provide the necessary
> +# definitions here to avoid cluttering package/tar/tar.mk .
> +# TAR will be overridden again when the tar .mk file is included, so we must
> +# finalize the value of SYSTEM_TAR by using :=
> +SYSTEM_TAR := $(TAR)
> +# Automatically detect tar --strip-path/components option
> +# --strip-path was renamed to --strip-components in tar 1.15
> +SYSTEM_TAR_STRIP_COMPONENTS := \
> +    $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
> +    if test $$? = 0 ; then \
> +     echo '--strip-path' ; \
> +    else \
> +     echo '--strip-components' ; \
> +    fi)

 This can move inside the ifneq below, with a 
SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)
in the else (and moving that one up of course).

> +
> +# Determine correct HOST_TAR
> +ifneq (,$(call suitable-host-package,tar))
> +  HOST_TAR = $(SYSTEM_TAR)
> +else
> +  DEPENDENCIES_HOST_PREREQ += host-tar
> +  HOST_TAR = $(HOST_DIR)/usr/bin/tar
> +endif
> +
> +# Since HOST_TAR is at least 1.17, it will certainly support --strip-components
> +HOST_TAR_STRIP_COMPONENTS = --strip-components
> +
> +# Set definitions used by all .mk files included after this one and before
> +# package/tar/tar.mk
> +TAR = $(HOST_TAR)
> +TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
> diff --git a/toolchain/dependencies/check-host-tar.sh b/toolchain/dependencies/check-host-tar.sh
> new file mode 100755
> --- /dev/null
> +++ b/toolchain/dependencies/check-host-tar.sh
> @@ -0,0 +1,29 @@
> +#!/bin/sh
> +
> +tar=`which tar`

 This contradicts the TAR ?= tar in check-host-tar.mk: if the user provided
their own tar version, it isn't checked and host-tar will still be built...
I guess $TAR should be given as an argument to this script.

> +if [ -z "$tar" ]; then
> +	return
> +fi

 This could be a [ ! -x "$tar" ]
and it should be exit 1, not return.

 Regards,
 Arnout


> +
> +# Output of 'tar --version' examples:
> +# tar (GNU tar) 1.15.1 
> +# tar (GNU tar) 1.25
> +version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'`
> +major=`echo "$version" | cut -d. -f1`
> +minor=`echo "$version" | cut -d. -f2`
> +bugfix=`echo "$version" | cut -d. -f3`
> +
> +# Minimal version = 1.17 (previous versions do not correctly unpack archives
> +# containing hard-links if the --strip-components option is used).
> +major_min=1
> +minor_min=17
> +if [ $major -gt $major_min ]; then
> +	echo $tar
> +else
> +	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
> +		echo $tar
> +	else
> +		false
> +		# echo nothing: no suitable tar found
> +	fi
> +fi
> 
> 

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/4fce0ba6/attachment.html>

[Buildroot] [PATCH 1 of 3 v2] dependencies: add function suitable-host-package

[Thomas De Schampheleire]

Hi Arnout,

On Tue, Dec 6, 2011 at 6:12 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On Wednesday 23 November 2011 14:59:50 Thomas De Schampheleire wrote:
>
>> +# suitable-host-pkg: calls check-host-$(1).sh shell script, which
>
>> +# should return the path to the suitable host tool, or nothing if no
>
>> +# suitable tool was found.
>
>> +define suitable-host-package
>
>> +$(shell toolchain/dependencies/check-host-$(1).sh)
>
>> +endef
>
>
> I should have noticed this in the first review, but: If the function is
>
> this simple, it becomes pointless. Especially because other check-host-*
>
> mk files may want to implement the checking in a different way, e.g.
>
> $(eval $(shell ...)).

I'm not sure if it's pointless. It provides a uniform way of checking
for host packages, and in that sense becomes a sort of
'infrastructure'. Moreover, the syntax is simpler and does not require
knowing the path to the correct check-host file. If anything changes,
it only needs to change in one place.

Compare:
$(if $(call suitable-host-package,foo),,host-foo)
with
$(if $(shell toolchain/dependencies/check-host-foo.sh),,host-foo)

>
>
>> +include toolchain/dependencies/check-host-*.mk
>
>
> And if this is the only thing remaining, perhaps it's better to merge
>
> this patch with the next one :-)
>
>
>

Best regards,
Thomas

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Thomas De Schampheleire]

Hi Arnout,

(First of all: your sending mails in both plaintext and html, messes
up my Gmail sending: extra white lines are added between each line. Am
I the only one with this problem? Does anyone know a fix? (one fix
would be to send pure plain text to this list, but I don't know if
that's an option for you)

On Tue, Dec 6, 2011 at 6:23 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On Wednesday 23 November 2011 14:59:59 Thomas De Schampheleire wrote:
>
>> diff --git a/toolchain/dependencies/check-host-tar.mk
>> b/toolchain/dependencies/check-host-tar.mk
>
>> new file mode 100644
>
>> --- /dev/null
>
>> +++ b/toolchain/dependencies/check-host-tar.mk
>
>> @@ -0,0 +1,32 @@
>
>> +TAR ?= tar
>
>> +
>
>> +# To extract tar, we always need a real (system) tar. Provide the
>> necessary
>
>> +# definitions here to avoid cluttering package/tar/tar.mk .
>
>> +# TAR will be overridden again when the tar .mk file is included, so we
>> must
>
>> +# finalize the value of SYSTEM_TAR by using :=
>
>> +SYSTEM_TAR := $(TAR)
>
>> +# Automatically detect tar --strip-path/components option
>
>> +# --strip-path was renamed to --strip-components in tar 1.15
>
>> +SYSTEM_TAR_STRIP_COMPONENTS := \
>
>> + $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
>
>> + if test $$? = 0 ; then \
>
>> + echo '--strip-path' ; \
>
>> + else \
>
>> + echo '--strip-components' ; \
>
>> + fi)
>
>
> This can move inside the ifneq below, with a
>
> SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)
>
> in the else (and moving that one up of course).
>

I don't think this is true.
For the chicken-egg problem in the current patch, you need SYSTEM_TAR
(a real one) to build host-tar if needed. So in the 'else' case, you
need the full SYSTEM_TAR_STRIP_COMPONENTS.

You could do the following: (maybe this is what you meant)

HOST_TAR_STRIP_COMPONENTS = --strip-components
ifneq (,$(call suitable-host-package,tar))
  HOST_TAR = $(SYSTEM_TAR)
  SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)
else
  DEPENDENCIES_HOST_PREREQ += host-tar
  HOST_TAR = $(HOST_DIR)/usr/bin/tar
  SYSTEM_TAR_STRIP_COMPONENTS := \
    $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
    if test $$? = 0 ; then \
     echo '--strip-path' ; \
    else \
     echo '--strip-components' ; \
    fi)
endif

But to be honest, I don't think we'll need any of this complexity
anymore with the cpio approach (see next mail).


>
>> +
>
>> +# Determine correct HOST_TAR
>
>> +ifneq (,$(call suitable-host-package,tar))
>
>> + HOST_TAR = $(SYSTEM_TAR)
>
>> +else
>
>> + DEPENDENCIES_HOST_PREREQ += host-tar
>
>> + HOST_TAR = $(HOST_DIR)/usr/bin/tar
>
>> +endif
>
>> +
>
>> +# Since HOST_TAR is at least 1.17, it will certainly support
>> --strip-components
>
>> +HOST_TAR_STRIP_COMPONENTS = --strip-components
>
>> +
>
>> +# Set definitions used by all .mk files included after this one and
>> before
>
>> +# package/tar/tar.mk
>
>> +TAR = $(HOST_TAR)
>
>> +TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
>
>> diff --git a/toolchain/dependencies/check-host-tar.sh
>> b/toolchain/dependencies/check-host-tar.sh
>
>> new file mode 100755
>
>> --- /dev/null
>
>> +++ b/toolchain/dependencies/check-host-tar.sh
>
>> @@ -0,0 +1,29 @@
>
>> +#!/bin/sh
>
>> +
>
>> +tar=`which tar`
>
>
> This contradicts the TAR ?= tar in check-host-tar.mk: if the user provided
>
> their own tar version, it isn't checked and host-tar will still be built...
>
> I guess $TAR should be given as an argument to this script.

You're right, I missed this one.

>
>
>> +if [ -z "$tar" ]; then
>
>> + return
>
>> +fi
>
>
> This could be a [ ! -x "$tar" ]

ok

>
> and it should be exit 1, not return.

Why? The convention is that the check-host-foo.sh script should give
as output either nothing (no suitable package found) or the path to
the suitable package. The return code is not checked.
Of course, it wouldn't hurt, but it's not needed for the implementation.

Thanks for your comments.

Best regards,
Thomas

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Thomas De Schampheleire]

On Tue, Dec 6, 2011 at 6:07 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
>
>> > TAR is redefined to HOST_TAR, except when extracting host-tar (this
>
>> > is a chicken-and-egg problem), so that all packages use the host-tar
>
>> > if no suitable tar was found.
>
>>
>
>> I am still puzzled by the fact that tar is needed to extract this
>
>> host-tar, so there's like a chicken-and-egg problem. I understand that
>
>> the real host tar is used to extract the host-tar, which is then use to
>
>> extract all other packages. But that sounds really nasty to me.
>
>
> It's not so strange, it's like a compiler's canadian build. You need a
>
> compiler to compile the compiler, right?
>
>
> The way it is solved for tar is OK for me. I think adding support for
>
> cpio archives would just complicate things, because then host-tar
>
> can't use GENTARGETS anymore.

I don't think that's true. I started adding cpio support to
gentargets. Essentially, here is the code that needs modification:

# Define extractors for different archive suffixes
INFLATE.bz2  = $(BZCAT)
INFLATE.gz   = $(ZCAT)
INFLATE.tbz  = $(BZCAT)
INFLATE.tbz2 = $(BZCAT)
INFLATE.tgz  = $(ZCAT)
INFLATE.xz   = $(XZCAT)
INFLATE.tar  = cat

$(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
$(DL_DIR)/$$($(2)_SOURCE) | \
        $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)


Although the inflate step already handles multiple suffices, the
extract does not yet.

My current approach is this:
+INFLATE.cpio = cat
+
+EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) -
+EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )

 $(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
$(DL_DIR)/$$($(2)_SOURCE) | \
-       $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
+       $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call
EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call
EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))


The only thing I'm struggling with is that cpio does not have a
--strip-components functionality. As a result, the package is
extracted in output/build/host-tar-1.26/tar-1.26 which is not right.
I don't directly know the name 'tar-1.26', because all I have is the
string 'host-tar-1.26'. I see two approaches:
* either try to strip of 'host' from $(2)_DIR
* or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'

Suggestions are much appreciated...

Thanks,
Thomas

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Michael S. Zick]

On Thu December 8 2011, Thomas De Schampheleire wrote:
> On Tue, Dec 6, 2011 at 6:07 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> > On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
> >
> >> > TAR is redefined to HOST_TAR, except when extracting host-tar (this
> >
> >> > is a chicken-and-egg problem), so that all packages use the host-tar
> >
> >> > if no suitable tar was found.
> >
> >>
> >
> >> I am still puzzled by the fact that tar is needed to extract this
> >
> >> host-tar, so there's like a chicken-and-egg problem. I understand that
> >
> >> the real host tar is used to extract the host-tar, which is then use to
> >
> >> extract all other packages. But that sounds really nasty to me.
> >
> >
> > It's not so strange, it's like a compiler's canadian build. You need a
> >
> > compiler to compile the compiler, right?
> >
> >
> > The way it is solved for tar is OK for me. I think adding support for
> >
> > cpio archives would just complicate things, because then host-tar
> >
> > can't use GENTARGETS anymore.
> 
> I don't think that's true. I started adding cpio support to
> gentargets. Essentially, here is the code that needs modification:
> 
> # Define extractors for different archive suffixes
> INFLATE.bz2  = $(BZCAT)
> INFLATE.gz   = $(ZCAT)
> INFLATE.tbz  = $(BZCAT)
> INFLATE.tbz2 = $(BZCAT)
> INFLATE.tgz  = $(ZCAT)
> INFLATE.xz   = $(XZCAT)
> INFLATE.tar  = cat
> 
> $(2)_EXTRACT_CMDS ?= \
>         $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
> $(DL_DIR)/$$($(2)_SOURCE) | \
>         $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
> 
> 
> Although the inflate step already handles multiple suffices, the
> extract does not yet.
> 
> My current approach is this:
> +INFLATE.cpio = cat
> +
> +EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) -
> +EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )
> 
>  $(2)_EXTRACT_CMDS ?= \
>         $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
> $(DL_DIR)/$$($(2)_SOURCE) | \
> -       $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
> +       $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call
> EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call
> EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))
> 
> 
> The only thing I'm struggling with is that cpio does not have a
> --strip-components functionality. 
>

Would: "--no-absolute-filenames" help on the cpio command?

If this will become a general purpose extract.cpio command, it
should have that option even if it doesn't help in this case.

Many cpio archives do have absolute paths (like the initramFS
archive) even if the GNU-TAR cpio archive does not.

Mike
> As a result, the package is 
> extracted in output/build/host-tar-1.26/tar-1.26 which is not right.
> I don't directly know the name 'tar-1.26', because all I have is the
> string 'host-tar-1.26'. I see two approaches:
> * either try to strip of 'host' from $(2)_DIR
> * or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'
> 
> Suggestions are much appreciated...
> 
> Thanks,
> Thomas
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 
> 

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Michael S. Zick]

On Thu December 8 2011, Thomas De Schampheleire wrote:
> Hi Arnout,
> 
> (First of all: your sending mails in both plaintext and html, messes
> up my Gmail sending: extra white lines are added between each line. Am
> I the only one with this problem? Does anyone know a fix? (one fix
> would be to send pure plain text to this list, but I don't know if
> that's an option for you)
>

I see the same thing here.
He is using Kmail for an e-mail client, so "plain text only" is an
option of his e-mail client.
(At least the version of Kmail I am sending this with has the option.)

Mike 
> On Tue, Dec 6, 2011 at 6:23 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> > On Wednesday 23 November 2011 14:59:59 Thomas De Schampheleire wrote:
> >
> >> diff --git a/toolchain/dependencies/check-host-tar.mk
> >> b/toolchain/dependencies/check-host-tar.mk
> >
> >> new file mode 100644
> >
> >> --- /dev/null
> >
> >> +++ b/toolchain/dependencies/check-host-tar.mk
> >
> >> @@ -0,0 +1,32 @@
> >
> >> +TAR ?= tar
> >
> >> +
> >
> >> +# To extract tar, we always need a real (system) tar. Provide the
> >> necessary
> >
> >> +# definitions here to avoid cluttering package/tar/tar.mk .
> >
> >> +# TAR will be overridden again when the tar .mk file is included, so we
> >> must
> >
> >> +# finalize the value of SYSTEM_TAR by using :=
> >
> >> +SYSTEM_TAR := $(TAR)
> >
> >> +# Automatically detect tar --strip-path/components option
> >
> >> +# --strip-path was renamed to --strip-components in tar 1.15
> >
> >> +SYSTEM_TAR_STRIP_COMPONENTS := \
> >
> >> + $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
> >
> >> + if test $$? = 0 ; then \
> >
> >> + echo '--strip-path' ; \
> >
> >> + else \
> >
> >> + echo '--strip-components' ; \
> >
> >> + fi)
> >
> >
> > This can move inside the ifneq below, with a
> >
> > SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)
> >
> > in the else (and moving that one up of course).
> >
> 
> I don't think this is true.
> For the chicken-egg problem in the current patch, you need SYSTEM_TAR
> (a real one) to build host-tar if needed. So in the 'else' case, you
> need the full SYSTEM_TAR_STRIP_COMPONENTS.
> 
> You could do the following: (maybe this is what you meant)
> 
> HOST_TAR_STRIP_COMPONENTS = --strip-components
> ifneq (,$(call suitable-host-package,tar))
>   HOST_TAR = $(SYSTEM_TAR)
>   SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)
> else
>   DEPENDENCIES_HOST_PREREQ += host-tar
>   HOST_TAR = $(HOST_DIR)/usr/bin/tar
>   SYSTEM_TAR_STRIP_COMPONENTS := \
>     $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
>     if test $$? = 0 ; then \
>      echo '--strip-path' ; \
>     else \
>      echo '--strip-components' ; \
>     fi)
> endif
> 
> But to be honest, I don't think we'll need any of this complexity
> anymore with the cpio approach (see next mail).
> 
> 
> >
> >> +
> >
> >> +# Determine correct HOST_TAR
> >
> >> +ifneq (,$(call suitable-host-package,tar))
> >
> >> + HOST_TAR = $(SYSTEM_TAR)
> >
> >> +else
> >
> >> + DEPENDENCIES_HOST_PREREQ += host-tar
> >
> >> + HOST_TAR = $(HOST_DIR)/usr/bin/tar
> >
> >> +endif
> >
> >> +
> >
> >> +# Since HOST_TAR is at least 1.17, it will certainly support
> >> --strip-components
> >
> >> +HOST_TAR_STRIP_COMPONENTS = --strip-components
> >
> >> +
> >
> >> +# Set definitions used by all .mk files included after this one and
> >> before
> >
> >> +# package/tar/tar.mk
> >
> >> +TAR = $(HOST_TAR)
> >
> >> +TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
> >
> >> diff --git a/toolchain/dependencies/check-host-tar.sh
> >> b/toolchain/dependencies/check-host-tar.sh
> >
> >> new file mode 100755
> >
> >> --- /dev/null
> >
> >> +++ b/toolchain/dependencies/check-host-tar.sh
> >
> >> @@ -0,0 +1,29 @@
> >
> >> +#!/bin/sh
> >
> >> +
> >
> >> +tar=`which tar`
> >
> >
> > This contradicts the TAR ?= tar in check-host-tar.mk: if the user provided
> >
> > their own tar version, it isn't checked and host-tar will still be built...
> >
> > I guess $TAR should be given as an argument to this script.
> 
> You're right, I missed this one.
> 
> >
> >
> >> +if [ -z "$tar" ]; then
> >
> >> + return
> >
> >> +fi
> >
> >
> > This could be a [ ! -x "$tar" ]
> 
> ok
> 
> >
> > and it should be exit 1, not return.
> 
> Why? The convention is that the check-host-foo.sh script should give
> as output either nothing (no suitable package found) or the path to
> the suitable package. The return code is not checked.
> Of course, it wouldn't hurt, but it's not needed for the implementation.
> 
> Thanks for your comments.
> 
> Best regards,
> Thomas
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 
> 

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Michael S. Zick]

On Thu December 8 2011, Michael S. Zick wrote:
> On Thu December 8 2011, Thomas De Schampheleire wrote:
> > On Tue, Dec 6, 2011 at 6:07 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> > > On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
> > >
> > >> > TAR is redefined to HOST_TAR, except when extracting host-tar (this
> > >
> > >> > is a chicken-and-egg problem), so that all packages use the host-tar
> > >
> > >> > if no suitable tar was found.
> > >
> > >>
> > >
> > >> I am still puzzled by the fact that tar is needed to extract this
> > >
> > >> host-tar, so there's like a chicken-and-egg problem. I understand that
> > >
> > >> the real host tar is used to extract the host-tar, which is then use to
> > >
> > >> extract all other packages. But that sounds really nasty to me.
> > >
> > >
> > > It's not so strange, it's like a compiler's canadian build. You need a
> > >
> > > compiler to compile the compiler, right?
> > >
> > >
> > > The way it is solved for tar is OK for me. I think adding support for
> > >
> > > cpio archives would just complicate things, because then host-tar
> > >
> > > can't use GENTARGETS anymore.
> > 
> > I don't think that's true. I started adding cpio support to
> > gentargets. Essentially, here is the code that needs modification:
> > 
> > # Define extractors for different archive suffixes
> > INFLATE.bz2  = $(BZCAT)
> > INFLATE.gz   = $(ZCAT)
> > INFLATE.tbz  = $(BZCAT)
> > INFLATE.tbz2 = $(BZCAT)
> > INFLATE.tgz  = $(ZCAT)
> > INFLATE.xz   = $(XZCAT)
> > INFLATE.tar  = cat
> > 
> > $(2)_EXTRACT_CMDS ?= \
> >         $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
> > $(DL_DIR)/$$($(2)_SOURCE) | \
> >         $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
> > 
> > 
> > Although the inflate step already handles multiple suffices, the
> > extract does not yet.
> >

Is the inflate step being used for tar archives?

The reason I ask is because GNU-tar (and BSD-tar) will auto-detect
the compression method from the file and un-archive it without
a prior de-compression.

In fact, if the "host tar" is BSD-tar, then you can unpack *.iso9660
as if it was an archive.

Mike 
> > My current approach is this:
> > +INFLATE.cpio = cat
> > +
> > +EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) -
> > +EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )
> > 
> >  $(2)_EXTRACT_CMDS ?= \
> >         $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
> > $(DL_DIR)/$$($(2)_SOURCE) | \
> > -       $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
> > +       $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call
> > EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call
> > EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))
> > 
> > 
> > The only thing I'm struggling with is that cpio does not have a
> > --strip-components functionality. 
> >
> 
> Would: "--no-absolute-filenames" help on the cpio command?
> 
> If this will become a general purpose extract.cpio command, it
> should have that option even if it doesn't help in this case.
> 
> Many cpio archives do have absolute paths (like the initramFS
> archive) even if the GNU-TAR cpio archive does not.
> 
> Mike
> > As a result, the package is 
> > extracted in output/build/host-tar-1.26/tar-1.26 which is not right.
> > I don't directly know the name 'tar-1.26', because all I have is the
> > string 'host-tar-1.26'. I see two approaches:
> > * either try to strip of 'host' from $(2)_DIR
> > * or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'
> > 
> > Suggestions are much appreciated...
> > 
> > Thanks,
> > Thomas
> > _______________________________________________
> > buildroot mailing list
> > buildroot at busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot
> > 
> > 
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 
> 

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Thomas De Schampheleire]

On Thu, Dec 8, 2011 at 4:05 PM, Michael S. Zick <minimod@morethan.org> wrote:
> On Thu December 8 2011, Michael S. Zick wrote:
>> On Thu December 8 2011, Thomas De Schampheleire wrote:
>> > On Tue, Dec 6, 2011 at 6:07 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>> > > On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
>> > >
>> > >> > TAR is redefined to HOST_TAR, except when extracting host-tar (this
>> > >
>> > >> > is a chicken-and-egg problem), so that all packages use the host-tar
>> > >
>> > >> > if no suitable tar was found.
>> > >
>> > >>
>> > >
>> > >> I am still puzzled by the fact that tar is needed to extract this
>> > >
>> > >> host-tar, so there's like a chicken-and-egg problem. I understand that
>> > >
>> > >> the real host tar is used to extract the host-tar, which is then use to
>> > >
>> > >> extract all other packages. But that sounds really nasty to me.
>> > >
>> > >
>> > > It's not so strange, it's like a compiler's canadian build. You need a
>> > >
>> > > compiler to compile the compiler, right?
>> > >
>> > >
>> > > The way it is solved for tar is OK for me. I think adding support for
>> > >
>> > > cpio archives would just complicate things, because then host-tar
>> > >
>> > > can't use GENTARGETS anymore.
>> >
>> > I don't think that's true. I started adding cpio support to
>> > gentargets. Essentially, here is the code that needs modification:
>> >
>> > # Define extractors for different archive suffixes
>> > INFLATE.bz2 ?= $(BZCAT)
>> > INFLATE.gz ? = $(ZCAT)
>> > INFLATE.tbz ?= $(BZCAT)
>> > INFLATE.tbz2 = $(BZCAT)
>> > INFLATE.tgz ?= $(ZCAT)
>> > INFLATE.xz ? = $(XZCAT)
>> > INFLATE.tar ?= cat
>> >
>> > $(2)_EXTRACT_CMDS ?= \
>> > ? ? ? ? $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
>> > $(DL_DIR)/$$($(2)_SOURCE) | \
>> > ? ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
>> >
>> >
>> > Although the inflate step already handles multiple suffices, the
>> > extract does not yet.
>> >
>
> Is the inflate step being used for tar archives?
>
> The reason I ask is because GNU-tar (and BSD-tar) will auto-detect
> the compression method from the file and un-archive it without
> a prior de-compression.
>

Yes, the inflate step is used for tar archives.
Suppose you have a file called foo.tar.gz, then the commands executed
will be similar to: (left out some details)
zcat foo.tar.gz | tar -xf

For foo.tar.bz2, this will be:
bzcat foo.tar.bz2 | tar -xf

The tar step does not have to bother with compression.

> In fact, if the "host tar" is BSD-tar, then you can unpack *.iso9660
> as if it was an archive.
>
> Mike
>> > My current approach is this:
>> > +INFLATE.cpio = cat
>> > +
>> > +EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) -
>> > +EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )
>> >
>> > ?$(2)_EXTRACT_CMDS ?= \
>> > ? ? ? ? $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
>> > $(DL_DIR)/$$($(2)_SOURCE) | \
>> > - ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
>> > + ? ? ? $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call
>> > EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call
>> > EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))
>> >
>> >
>> > The only thing I'm struggling with is that cpio does not have a
>> > --strip-components functionality.
>> >
>>
>> Would: "--no-absolute-filenames" help on the cpio command?
>>
>> If this will become a general purpose extract.cpio command, it
>> should have that option even if it doesn't help in this case.
>>
>> Many cpio archives do have absolute paths (like the initramFS
>> archive) even if the GNU-TAR cpio archive does not.
>>
>> Mike
>> > As a result, the package is
>> > extracted in output/build/host-tar-1.26/tar-1.26 which is not right.
>> > I don't directly know the name 'tar-1.26', because all I have is the
>> > string 'host-tar-1.26'. I see two approaches:
>> > * either try to strip of 'host' from $(2)_DIR
>> > * or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'
>> >
>> > Suggestions are much appreciated...
>> >
>> > Thanks,
>> > Thomas
>> > _______________________________________________
>> > buildroot mailing list
>> > buildroot at busybox.net
>> > http://lists.busybox.net/mailman/listinfo/buildroot
>> >
>> >
>>
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot at busybox.net
>> http://lists.busybox.net/mailman/listinfo/buildroot
>>
>>
>
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH 2 of 3 v2] dependencies: build a host-tar if no suitable tar can be found

[Thomas De Schampheleire]

On Fri, Dec 9, 2011 at 3:11 PM, Michael S. Zick <minimod@morethan.org> wrote:
> On Fri December 9 2011, you wrote:
>> On Thu, Dec 8, 2011 at 4:05 PM, Michael S. Zick <minimod@morethan.org> wrote:
>> > On Thu December 8 2011, Michael S. Zick wrote:
>> >> On Thu December 8 2011, Thomas De Schampheleire wrote:
>> >> > On Tue, Dec 6, 2011 at 6:07 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>> >> > > On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
>> >> > >
>> >> > >> > TAR is redefined to HOST_TAR, except when extracting host-tar (this
>> >> > >
>> >> > >> > is a chicken-and-egg problem), so that all packages use the host-tar
>> >> > >
>> >> > >> > if no suitable tar was found.
>> >> > >
>> >> > >>
>> >> > >
>> >> > >> I am still puzzled by the fact that tar is needed to extract this
>> >> > >
>> >> > >> host-tar, so there's like a chicken-and-egg problem. I understand that
>> >> > >
>> >> > >> the real host tar is used to extract the host-tar, which is then use to
>> >> > >
>> >> > >> extract all other packages. But that sounds really nasty to me.
>> >> > >
>> >> > >
>> >> > > It's not so strange, it's like a compiler's canadian build. You need a
>> >> > >
>> >> > > compiler to compile the compiler, right?
>> >> > >
>> >> > >
>> >> > > The way it is solved for tar is OK for me. I think adding support for
>> >> > >
>> >> > > cpio archives would just complicate things, because then host-tar
>> >> > >
>> >> > > can't use GENTARGETS anymore.
>> >> >
>> >> > I don't think that's true. I started adding cpio support to
>> >> > gentargets. Essentially, here is the code that needs modification:
>> >> >
>> >> > # Define extractors for different archive suffixes
>> >> > INFLATE.bz2 ?= $(BZCAT)
>> >> > INFLATE.gz ? = $(ZCAT)
>> >> > INFLATE.tbz ?= $(BZCAT)
>> >> > INFLATE.tbz2 = $(BZCAT)
>> >> > INFLATE.tgz ?= $(ZCAT)
>> >> > INFLATE.xz ? = $(XZCAT)
>> >> > INFLATE.tar ?= cat
>> >> >
>> >> > $(2)_EXTRACT_CMDS ?= \
>> >> > ? ? ? ? $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
>> >> > $(DL_DIR)/$$($(2)_SOURCE) | \
>> >> > ? ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
>> >> >
>> >> >
>> >> > Although the inflate step already handles multiple suffices, the
>> >> > extract does not yet.
>> >> >
>> >
>> > Is the inflate step being used for tar archives?
>> >
>> > The reason I ask is because GNU-tar (and BSD-tar) will auto-detect
>> > the compression method from the file and un-archive it without
>> > a prior de-compression.
>> >
>>
>> Yes, the inflate step is used for tar archives.
>>
>
> My point was, it does not need to.
> The INFLATE.* methods need only be available for use by things
> other than gnu-tar.
>
> Without leaving out any of the details (except the src/dest ls check):
>
> mszick at wolf466:~$ tar --create --file=foo.tar.gz *.txt
>
> mszick at wolf466:~$ ls -l *.tar.gz
> -rw-r--r-- 1 mszick mszick 501760 2011-12-09 08:02 foo.tar.gz
>
> mszick at wolf466:~$ mkdir tmp
> mszick at wolf466:~$ cd tmp
>
> mszick at wolf466:~/tmp$ tar --extract --file=../foo.tar.gz
>
> mszick at wolf466:~/tmp$ ls
> - - - shows the expected files - - -
>
> mszick at wolf466:~/tmp$ tar --version
> tar (GNU tar) 1.20
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Which is a pretty old gnu-tar.
> I think the auto-detect went in around 1.17 or so.

Thanks for the tip. I wasn't aware of this.

As long as tar supports all compression methods buildroot supports
(currently .gz, .bz2 and .xz) then we should be able to remove the
inflate step, I guess. I'll try to make a patch for this.

It doesn't help the cpio unpacking though, so it is more or less an
independent patch.

Best regards,
Thomas