* [Buildroot] [PATCH 01/12] package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-14 21:12 ` Romain Naour
2019-12-10 21:59 ` [Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro Thomas Petazzoni
` (11 subsequent siblings)
12 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
The extractor-dependency macro returns which system-provided tools are
needed to be able to extract the archive passed as argument. The
result of this macro is added to DL_TOOLS_DEPENDENCIES so that the
logic in support/dependencies/ verifies that the necessary tools are
provided by the system.
However, we are going to add another macro, extractor-pkg-dependency,
which says which Buildroot packages are needed to extract the archive
passed as argument. Indeed, for those archive types, if the extractor
is not provided system-wide, we build it as a host Buildroot package.
To clarify the distinction between the upcoming
extractor-pkg-dependency and existing extractor-dependency, we rename
the latter to extractor-system-dependency.
We take this opportunity to extend the documentation of this macro.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-generic.mk | 2 +-
package/pkg-utils.mk | 14 +++++++++-----
2 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 6d598cfd9d..5ad8f1d894 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs)
DL_TOOLS_DEPENDENCIES += cvs
endif # SITE_METHOD
-DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
+DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE))
# Ensure all virtual targets are PHONY. Listed alphabetically.
.PHONY: $(1) \
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index d38971caf6..0f916d81d8 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -46,11 +46,15 @@ INFLATE.tar = cat
# suitable-extractor(filename): returns extractor based on suffix
suitable-extractor = $(INFLATE$(suffix $(1)))
-# extractor-dependency(filename): returns extractor for 'filename' if the
-# extractor is a dependency. If we build the extractor return nothing.
-# $(firstword) is used here because the extractor can have arguments, like
-# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'.
-extractor-dependency = $(firstword $(INFLATE$(filter-out \
+# extractor-system-dependency(filename): returns the name of the tool
+# needed to extract 'filename', and is meant to be used with
+# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
+# provided by the system Buildroot runs on.
+#
+# $(firstword) is used here because the extractor can have arguments,
+# like ZCAT="gzip -d -c", and to check for the dependency we only want
+# 'gzip'.
+extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
# check-deprecated-variable -- throw an error on deprecated variables
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 01/12] package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency
2019-12-10 21:59 ` [Buildroot] [PATCH 01/12] package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency Thomas Petazzoni
@ 2019-12-14 21:12 ` Romain Naour
0 siblings, 0 replies; 24+ messages in thread
From: Romain Naour @ 2019-12-14 21:12 UTC (permalink / raw)
To: buildroot
Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> The extractor-dependency macro returns which system-provided tools are
> needed to be able to extract the archive passed as argument. The
> result of this macro is added to DL_TOOLS_DEPENDENCIES so that the
> logic in support/dependencies/ verifies that the necessary tools are
> provided by the system.
>
> However, we are going to add another macro, extractor-pkg-dependency,
> which says which Buildroot packages are needed to extract the archive
> passed as argument. Indeed, for those archive types, if the extractor
> is not provided system-wide, we build it as a host Buildroot package.
>
> To clarify the distinction between the upcoming
> extractor-pkg-dependency and existing extractor-dependency, we rename
> the latter to extractor-system-dependency.
>
> We take this opportunity to extend the documentation of this macro.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Best regards,
Romain
> ---
> package/pkg-generic.mk | 2 +-
> package/pkg-utils.mk | 14 +++++++++-----
> 2 files changed, 10 insertions(+), 6 deletions(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 6d598cfd9d..5ad8f1d894 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs)
> DL_TOOLS_DEPENDENCIES += cvs
> endif # SITE_METHOD
>
> -DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
> +DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE))
>
> # Ensure all virtual targets are PHONY. Listed alphabetically.
> .PHONY: $(1) \
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index d38971caf6..0f916d81d8 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -46,11 +46,15 @@ INFLATE.tar = cat
> # suitable-extractor(filename): returns extractor based on suffix
> suitable-extractor = $(INFLATE$(suffix $(1)))
>
> -# extractor-dependency(filename): returns extractor for 'filename' if the
> -# extractor is a dependency. If we build the extractor return nothing.
> -# $(firstword) is used here because the extractor can have arguments, like
> -# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'.
> -extractor-dependency = $(firstword $(INFLATE$(filter-out \
> +# extractor-system-dependency(filename): returns the name of the tool
> +# needed to extract 'filename', and is meant to be used with
> +# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
> +# provided by the system Buildroot runs on.
> +#
> +# $(firstword) is used here because the extractor can have arguments,
> +# like ZCAT="gzip -d -c", and to check for the dependency we only want
> +# 'gzip'.
> +extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
> $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
>
> # check-deprecated-variable -- throw an error on deprecated variables
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 01/12] package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-14 21:20 ` Romain Naour
2019-12-10 21:59 ` [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency Thomas Petazzoni
` (10 subsequent siblings)
12 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
To extract some archive types, if the extracting tool is not available
on the system where Buildroot runs on, we build it using a Buildroot
host package.
Such dependencies are currently explicitly handled by the
inner-generic-package macro, but in fact we also need to handle them
in all places where the "suitable-extractor" macro is invoked, and
some packages invoke it directly. Otherwise, such packages may be
missing a dependency to the appropriate host Buildroot package
building the extracting tool they need. An example is gettext-tiny,
whose source code is a gzip-compressed tarball, but in addition
manually extracts a xz-compressed tarball.
This extractor-pkg-dependency macro will be used in follow-up commits
to ensure all the packages that use suitable-extractor properly add
the correct dependencies.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-utils.mk | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 0f916d81d8..35a441e0aa 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -46,6 +46,14 @@ INFLATE.tar = cat
# suitable-extractor(filename): returns extractor based on suffix
suitable-extractor = $(INFLATE$(suffix $(1)))
+EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY)
+EXTRACTOR_PKG_DEPENDENCY.xz = $(BR2_XZCAT_HOST_DEPENDENCY)
+EXTRACTOR_PKG_DEPENDENCY.lz = $(BR2_LZIP_HOST_DEPENDENCY)
+
+# extractor-pkg-dependency(filename): returns a Buildroot package
+# dependency needed to extract file based on suffix
+extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
+
# extractor-system-dependency(filename): returns the name of the tool
# needed to extract 'filename', and is meant to be used with
# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro
2019-12-10 21:59 ` [Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro Thomas Petazzoni
@ 2019-12-14 21:20 ` Romain Naour
2019-12-14 21:42 ` Thomas Petazzoni
0 siblings, 1 reply; 24+ messages in thread
From: Romain Naour @ 2019-12-14 21:20 UTC (permalink / raw)
To: buildroot
Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> To extract some archive types, if the extracting tool is not available
> on the system where Buildroot runs on, we build it using a Buildroot
> host package.
>
> Such dependencies are currently explicitly handled by the
> inner-generic-package macro, but in fact we also need to handle them
> in all places where the "suitable-extractor" macro is invoked, and
> some packages invoke it directly. Otherwise, such packages may be
> missing a dependency to the appropriate host Buildroot package
> building the extracting tool they need. An example is gettext-tiny,
> whose source code is a gzip-compressed tarball, but in addition
> manually extracts a xz-compressed tarball.
>
> This extractor-pkg-dependency macro will be used in follow-up commits
> to ensure all the packages that use suitable-extractor properly add
> the correct dependencies.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> package/pkg-utils.mk | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 0f916d81d8..35a441e0aa 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -46,6 +46,14 @@ INFLATE.tar = cat
> # suitable-extractor(filename): returns extractor based on suffix
> suitable-extractor = $(INFLATE$(suffix $(1)))
>
What about gzip ?
EXTRACTOR_PKG_DEPENDENCY.gz = $(BR2_GZIP_HOST_DEPENDENCY)
EXTRACTOR_PKG_DEPENDENCY.tgz = $(BR2_GZIP_HOST_DEPENDENCY)
Best regards,
Romain
> +EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY)
> +EXTRACTOR_PKG_DEPENDENCY.xz = $(BR2_XZCAT_HOST_DEPENDENCY)
> +EXTRACTOR_PKG_DEPENDENCY.lz = $(BR2_LZIP_HOST_DEPENDENCY)
> +
> +# extractor-pkg-dependency(filename): returns a Buildroot package
> +# dependency needed to extract file based on suffix
> +extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
> +
> # extractor-system-dependency(filename): returns the name of the tool
> # needed to extract 'filename', and is meant to be used with
> # DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro
2019-12-14 21:20 ` Romain Naour
@ 2019-12-14 21:42 ` Thomas Petazzoni
2019-12-17 20:49 ` Yann E. MORIN
0 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-14 21:42 UTC (permalink / raw)
To: buildroot
On Sat, 14 Dec 2019 22:20:21 +0100
Romain Naour <romain.naour@smile.fr> wrote:
> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index 0f916d81d8..35a441e0aa 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -46,6 +46,14 @@ INFLATE.tar = cat
> > # suitable-extractor(filename): returns extractor based on suffix
> > suitable-extractor = $(INFLATE$(suffix $(1)))
> >
>
> What about gzip ?
>
> EXTRACTOR_PKG_DEPENDENCY.gz = $(BR2_GZIP_HOST_DEPENDENCY)
> EXTRACTOR_PKG_DEPENDENCY.tgz = $(BR2_GZIP_HOST_DEPENDENCY)
Because oddly enough, we are today not taking into account the
dependency on gzip when extracting tarballs.
We do take it into account only in:
ifneq ($$(filter cvs git svn,$$($(2)_SITE_METHOD)),)
$(2)_DOWNLOAD_DEPENDENCIES += \
$(BR2_GZIP_HOST_DEPENDENCY) \
$(BR2_TAR_HOST_DEPENDENCY)
endif
i.e, when we fetch from a version control system, and create our own
tarball locally.
But if we download directly from http/ftp a .tar.gz or .tgz, we don't
rely on BR2_GZIP_HOST_DEPENDENCY, and simply assume we can use the one
from the host machine.
This should be fixed, I agree, but it's already broken today I believe.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro
2019-12-14 21:42 ` Thomas Petazzoni
@ 2019-12-17 20:49 ` Yann E. MORIN
0 siblings, 0 replies; 24+ messages in thread
From: Yann E. MORIN @ 2019-12-17 20:49 UTC (permalink / raw)
To: buildroot
Thomas, Romain, All,
On 2019-12-14 22:42 +0100, Thomas Petazzoni spake thusly:
> On Sat, 14 Dec 2019 22:20:21 +0100
> Romain Naour <romain.naour@smile.fr> wrote:
>
> > > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > > index 0f916d81d8..35a441e0aa 100644
> > > --- a/package/pkg-utils.mk
> > > +++ b/package/pkg-utils.mk
> > > @@ -46,6 +46,14 @@ INFLATE.tar = cat
> > > # suitable-extractor(filename): returns extractor based on suffix
> > > suitable-extractor = $(INFLATE$(suffix $(1)))
> > >
> >
> > What about gzip ?
> >
> > EXTRACTOR_PKG_DEPENDENCY.gz = $(BR2_GZIP_HOST_DEPENDENCY)
> > EXTRACTOR_PKG_DEPENDENCY.tgz = $(BR2_GZIP_HOST_DEPENDENCY)
>
> Because oddly enough, we are today not taking into account the
> dependency on gzip when extracting tarballs.
>
> We do take it into account only in:
>
> ifneq ($$(filter cvs git svn,$$($(2)_SITE_METHOD)),)
> $(2)_DOWNLOAD_DEPENDENCIES += \
> $(BR2_GZIP_HOST_DEPENDENCY) \
> $(BR2_TAR_HOST_DEPENDENCY)
> endif
>
> i.e, when we fetch from a version control system, and create our own
> tarball locally.
>
> But if we download directly from http/ftp a .tar.gz or .tgz, we don't
> rely on BR2_GZIP_HOST_DEPENDENCY, and simply assume we can use the one
> from the host machine.
>
> This should be fixed, I agree, but it's already broken today I believe.
In fact no. We need to build our own host-gzip only to create archives,
because some host have pigz instead of gzip, and this creates
non-reproducible archives.
However, decompressing with gzip or pigz is exactly the same; we do not
need our own host-gzip, and we can rely on the system-provided gzip,
whether that be gzip of pigz.
So, the current state of affairs is, AFAICS, correct.
Regards,
Yann E. MORIN.
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 01/12] package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 02/12] package/pkg-utils.mk: add extractor-pkg-dependency macro Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-14 21:37 ` Romain Naour
2019-12-10 21:59 ` [Buildroot] [PATCH 04/12] package/pkg-generic.mk: use extractor-pkg-dependency macro Thomas Petazzoni
` (9 subsequent siblings)
12 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
we can use them to implement extractor-system-dependency: if for a
given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
variable is empty, then it means we need the corresponding extractor
tool to be provided by the system.
Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
longer used, so we can drop it from support/dependencies/.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-utils.mk | 4 ++--
support/dependencies/check-host-lzip.mk | 1 -
support/dependencies/check-host-xzcat.mk | 1 -
3 files changed, 2 insertions(+), 4 deletions(-)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 35a441e0aa..d324934dba 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
# $(firstword) is used here because the extractor can have arguments,
# like ZCAT="gzip -d -c", and to check for the dependency we only want
# 'gzip'.
-extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
- $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
+extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
+ $(firstword $(INFLATE$(suffix $(1)))))
# check-deprecated-variable -- throw an error on deprecated variables
# example:
diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
index cdd784058c..98a4f42388 100644
--- a/support/dependencies/check-host-lzip.mk
+++ b/support/dependencies/check-host-lzip.mk
@@ -1,5 +1,4 @@
ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
BR2_LZIP_HOST_DEPENDENCY = host-lzip
-EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
LZCAT = $(HOST_DIR)/bin/lzip -d -c
endif
diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
index e5d72f95dd..9a78a8f38a 100644
--- a/support/dependencies/check-host-xzcat.mk
+++ b/support/dependencies/check-host-xzcat.mk
@@ -3,6 +3,5 @@
ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
BR2_XZCAT_HOST_DEPENDENCY = host-xz
-EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
XZCAT = $(HOST_DIR)/bin/xzcat
endif
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency
2019-12-10 21:59 ` [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency Thomas Petazzoni
@ 2019-12-14 21:37 ` Romain Naour
2019-12-14 21:43 ` Thomas Petazzoni
2019-12-14 21:45 ` Yann E. MORIN
0 siblings, 2 replies; 24+ messages in thread
From: Romain Naour @ 2019-12-14 21:37 UTC (permalink / raw)
To: buildroot
Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
> we can use them to implement extractor-system-dependency: if for a
> given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
> variable is empty, then it means we need the corresponding extractor
> tool to be provided by the system.
>
> Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
> longer used, so we can drop it from support/dependencies/.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> package/pkg-utils.mk | 4 ++--
> support/dependencies/check-host-lzip.mk | 1 -
> support/dependencies/check-host-xzcat.mk | 1 -
> 3 files changed, 2 insertions(+), 4 deletions(-)
>
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 35a441e0aa..d324934dba 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
> # $(firstword) is used here because the extractor can have arguments,
> # like ZCAT="gzip -d -c", and to check for the dependency we only want
> # 'gzip'.
> -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
> - $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
> +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
"EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"
Best regards,
Romain
> + $(firstword $(INFLATE$(suffix $(1)))))
>
> # check-deprecated-variable -- throw an error on deprecated variables
> # example:
> diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
> index cdd784058c..98a4f42388 100644
> --- a/support/dependencies/check-host-lzip.mk
> +++ b/support/dependencies/check-host-lzip.mk
> @@ -1,5 +1,4 @@
> ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
> BR2_LZIP_HOST_DEPENDENCY = host-lzip
> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
> LZCAT = $(HOST_DIR)/bin/lzip -d -c
> endif
> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> index e5d72f95dd..9a78a8f38a 100644
> --- a/support/dependencies/check-host-xzcat.mk
> +++ b/support/dependencies/check-host-xzcat.mk
> @@ -3,6 +3,5 @@
>
> ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
> BR2_XZCAT_HOST_DEPENDENCY = host-xz
> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
> XZCAT = $(HOST_DIR)/bin/xzcat
> endif
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency
2019-12-14 21:37 ` Romain Naour
@ 2019-12-14 21:43 ` Thomas Petazzoni
2019-12-14 21:45 ` Yann E. MORIN
1 sibling, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-14 21:43 UTC (permalink / raw)
To: buildroot
On Sat, 14 Dec 2019 22:37:14 +0100
Romain Naour <romain.naour@smile.fr> wrote:
> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index 35a441e0aa..d324934dba 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
> > # $(firstword) is used here because the extractor can have arguments,
> > # like ZCAT="gzip -d -c", and to check for the dependency we only want
> > # 'gzip'.
> > -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
> > - $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
> > +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
>
> I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
> "EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"
You're absolutely right. Now I wonder how it could have worked with my
code. I did quite a bit of testing, verifying the values of a number of
internal variables in various situations to make sure it was all
working well.
Thanks,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency
2019-12-14 21:37 ` Romain Naour
2019-12-14 21:43 ` Thomas Petazzoni
@ 2019-12-14 21:45 ` Yann E. MORIN
2019-12-14 21:53 ` Romain Naour
1 sibling, 1 reply; 24+ messages in thread
From: Yann E. MORIN @ 2019-12-14 21:45 UTC (permalink / raw)
To: buildroot
Romain, All,
On 2019-12-14 22:37 +0100, Romain Naour spake thusly:
> Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> > Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
> > we can use them to implement extractor-system-dependency: if for a
> > given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
> > variable is empty, then it means we need the corresponding extractor
> > tool to be provided by the system.
> >
> > Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
> > longer used, so we can drop it from support/dependencies/.
> >
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > ---
> > package/pkg-utils.mk | 4 ++--
> > support/dependencies/check-host-lzip.mk | 1 -
> > support/dependencies/check-host-xzcat.mk | 1 -
> > 3 files changed, 2 insertions(+), 4 deletions(-)
> >
> > diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> > index 35a441e0aa..d324934dba 100644
> > --- a/package/pkg-utils.mk
> > +++ b/package/pkg-utils.mk
> > @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
> > # $(firstword) is used here because the extractor can have arguments,
> > # like ZCAT="gzip -d -c", and to check for the dependency we only want
> > # 'gzip'.
> > -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
> > - $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
> > +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
>
> I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
> "EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"
In a Makefile:
FOO = a.b
$(info $(suffix $(FOO)))
would yield:
.b
so we already have the dot.
Regards,
Yann E. MORIN.
> Best regards,
> Romain
>
>
> > + $(firstword $(INFLATE$(suffix $(1)))))
> >
> > # check-deprecated-variable -- throw an error on deprecated variables
> > # example:
> > diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
> > index cdd784058c..98a4f42388 100644
> > --- a/support/dependencies/check-host-lzip.mk
> > +++ b/support/dependencies/check-host-lzip.mk
> > @@ -1,5 +1,4 @@
> > ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
> > BR2_LZIP_HOST_DEPENDENCY = host-lzip
> > -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
> > LZCAT = $(HOST_DIR)/bin/lzip -d -c
> > endif
> > diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
> > index e5d72f95dd..9a78a8f38a 100644
> > --- a/support/dependencies/check-host-xzcat.mk
> > +++ b/support/dependencies/check-host-xzcat.mk
> > @@ -3,6 +3,5 @@
> >
> > ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
> > BR2_XZCAT_HOST_DEPENDENCY = host-xz
> > -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
> > XZCAT = $(HOST_DIR)/bin/xzcat
> > endif
> >
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency
2019-12-14 21:45 ` Yann E. MORIN
@ 2019-12-14 21:53 ` Romain Naour
0 siblings, 0 replies; 24+ messages in thread
From: Romain Naour @ 2019-12-14 21:53 UTC (permalink / raw)
To: buildroot
Yann, Thomas,
Le 14/12/2019 ? 22:45, Yann E. MORIN a ?crit?:
> Romain, All,
>
> On 2019-12-14 22:37 +0100, Romain Naour spake thusly:
>> Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
>>> Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available,
>>> we can use them to implement extractor-system-dependency: if for a
>>> given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type>
>>> variable is empty, then it means we need the corresponding extractor
>>> tool to be provided by the system.
>>>
>>> Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no
>>> longer used, so we can drop it from support/dependencies/.
>>>
>>> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>>> ---
>>> package/pkg-utils.mk | 4 ++--
>>> support/dependencies/check-host-lzip.mk | 1 -
>>> support/dependencies/check-host-xzcat.mk | 1 -
>>> 3 files changed, 2 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
>>> index 35a441e0aa..d324934dba 100644
>>> --- a/package/pkg-utils.mk
>>> +++ b/package/pkg-utils.mk
>>> @@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
>>> # $(firstword) is used here because the extractor can have arguments,
>>> # like ZCAT="gzip -d -c", and to check for the dependency we only want
>>> # 'gzip'.
>>> -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
>>> - $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
>>> +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
>>
>> I would add a "." after EXTRACTOR_PKG_DEPENDENCY because we want to use
>> "EXTRACTOR_PKG_DEPENDENCY.$(suffix $(1))"
>
> In a Makefile:
> FOO = a.b
> $(info $(suffix $(FOO)))
>
> would yield:
> .b
>
> so we already have the dot.
Thanks, Indeed we already have the dot.
Sorry for the noise.
Best regards,
Romain
>
> Regards,
> Yann E. MORIN.
>
>> Best regards,
>> Romain
>>
>>
>>> + $(firstword $(INFLATE$(suffix $(1)))))
>>>
>>> # check-deprecated-variable -- throw an error on deprecated variables
>>> # example:
>>> diff --git a/support/dependencies/check-host-lzip.mk b/support/dependencies/check-host-lzip.mk
>>> index cdd784058c..98a4f42388 100644
>>> --- a/support/dependencies/check-host-lzip.mk
>>> +++ b/support/dependencies/check-host-lzip.mk
>>> @@ -1,5 +1,4 @@
>>> ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
>>> BR2_LZIP_HOST_DEPENDENCY = host-lzip
>>> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
>>> LZCAT = $(HOST_DIR)/bin/lzip -d -c
>>> endif
>>> diff --git a/support/dependencies/check-host-xzcat.mk b/support/dependencies/check-host-xzcat.mk
>>> index e5d72f95dd..9a78a8f38a 100644
>>> --- a/support/dependencies/check-host-xzcat.mk
>>> +++ b/support/dependencies/check-host-xzcat.mk
>>> @@ -3,6 +3,5 @@
>>>
>>> ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
>>> BR2_XZCAT_HOST_DEPENDENCY = host-xz
>>> -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
>>> XZCAT = $(HOST_DIR)/bin/xzcat
>>> endif
>>>
>>
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 04/12] package/pkg-generic.mk: use extractor-pkg-dependency macro
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (2 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 03/12] package/pkg-utils.mk: rework implementation of extractor-system-dependency Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-14 21:55 ` Romain Naour
2019-12-10 21:59 ` [Buildroot] [PATCH 05/12] arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract Thomas Petazzoni
` (8 subsequent siblings)
12 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
Instead of manually calculating the EXTRACT_DEPENDENCIES value based
on the archive extension, let's use the newly introduced
extractor-pkg-dependency macro.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/pkg-generic.mk | 10 +---------
1 file changed, 1 insertion(+), 9 deletions(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5ad8f1d894..45a9a133bb 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -644,15 +644,7 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY)
endif
ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
-ifneq ($$(filter .xz .lzma,$$(suffix $$($(2)_SOURCE))),)
-$(2)_EXTRACT_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)
-endif
-endif
-
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
-ifneq ($$(filter .lz,$$(suffix $$($(2)_SOURCE))),)
-$(2)_EXTRACT_DEPENDENCIES += $$(BR2_LZIP_HOST_DEPENDENCY)
-endif
+$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE))
endif
ifeq ($$(BR2_CCACHE),y)
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 04/12] package/pkg-generic.mk: use extractor-pkg-dependency macro
2019-12-10 21:59 ` [Buildroot] [PATCH 04/12] package/pkg-generic.mk: use extractor-pkg-dependency macro Thomas Petazzoni
@ 2019-12-14 21:55 ` Romain Naour
0 siblings, 0 replies; 24+ messages in thread
From: Romain Naour @ 2019-12-14 21:55 UTC (permalink / raw)
To: buildroot
Le 10/12/2019 ? 22:59, Thomas Petazzoni a ?crit?:
> Instead of manually calculating the EXTRACT_DEPENDENCIES value based
> on the archive extension, let's use the newly introduced
> extractor-pkg-dependency macro.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Best regards,
Romain
> ---
> package/pkg-generic.mk | 10 +---------
> 1 file changed, 1 insertion(+), 9 deletions(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 5ad8f1d894..45a9a133bb 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -644,15 +644,7 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY)
> endif
>
> ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
> -ifneq ($$(filter .xz .lzma,$$(suffix $$($(2)_SOURCE))),)
> -$(2)_EXTRACT_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)
> -endif
> -endif
> -
> -ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
> -ifneq ($$(filter .lz,$$(suffix $$($(2)_SOURCE))),)
> -$(2)_EXTRACT_DEPENDENCIES += $$(BR2_LZIP_HOST_DEPENDENCY)
> -endif
> +$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE))
> endif
>
> ifeq ($$(BR2_CCACHE),y)
>
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 05/12] arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (3 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 04/12] package/pkg-generic.mk: use extractor-pkg-dependency macro Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-10 22:09 ` Max Filippov
2019-12-10 21:59 ` [Buildroot] [PATCH 06/12] package/android-tools: add missing extract dependencies Thomas Petazzoni
` (7 subsequent siblings)
12 siblings, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
arch-xtensa-overlay-extract was used in gcc, linux, uboot, binutils
and gdb without properly accouting for the package dependencies that
may be needed to extract the overlay tarball.
This commit fixes that by introducing an
ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable in arch.mk.xtensa,
and then use it where appropriate in gcc, linux, uboot, binutils and
gdb.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
Note: if this is preferred, we could avoid adding the
ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable, and directly call
extractor-pkg-dependency from each of gcc, linux, uboot, binutils and
gdb. Having an intermediate variable felt a bit simpler/cleaner, but I
wouldn't mind changing it.
---
arch/arch.mk.xtensa | 2 ++
boot/uboot/uboot.mk | 1 +
linux/linux.mk | 1 +
package/binutils/binutils.mk | 2 ++
package/gcc/gcc-final/gcc-final.mk | 1 +
package/gcc/gcc-initial/gcc-initial.mk | 1 +
package/gdb/gdb.mk | 2 ++
7 files changed, 10 insertions(+)
diff --git a/arch/arch.mk.xtensa b/arch/arch.mk.xtensa
index fd410f6bfa..c0f83c27d2 100644
--- a/arch/arch.mk.xtensa
+++ b/arch/arch.mk.xtensa
@@ -19,6 +19,8 @@ else
ARCH_XTENSA_OVERLAY_FILE = $(BR_ARCH_XTENSA_OVERLAY_FILE)
endif
+ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(ARCH_XTENSA_OVERLAY_FILE))
+
################################################################################
# arch-xtensa-overlay-extract -- extract an extensa overlay
#
diff --git a/boot/uboot/uboot.mk b/boot/uboot/uboot.mk
index 0ebda6588c..eaf8d4a30a 100644
--- a/boot/uboot/uboot.mk
+++ b/boot/uboot/uboot.mk
@@ -185,6 +185,7 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
define UBOOT_XTENSA_OVERLAY_EXTRACT
$(call arch-xtensa-overlay-extract,$(@D),u-boot)
endef
+UBOOT_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
UBOOT_POST_EXTRACT_HOOKS += UBOOT_XTENSA_OVERLAY_EXTRACT
UBOOT_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
endif
diff --git a/linux/linux.mk b/linux/linux.mk
index 762f8868b1..325a15f74c 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -120,6 +120,7 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
define LINUX_XTENSA_OVERLAY_EXTRACT
$(call arch-xtensa-overlay-extract,$(@D),linux)
endef
+LINUX_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
LINUX_POST_EXTRACT_HOOKS += LINUX_XTENSA_OVERLAY_EXTRACT
LINUX_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
endif
diff --git a/package/binutils/binutils.mk b/package/binutils/binutils.mk
index a19d6940f7..0af83e92d0 100644
--- a/package/binutils/binutils.mk
+++ b/package/binutils/binutils.mk
@@ -120,8 +120,10 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
define BINUTILS_XTENSA_OVERLAY_EXTRACT
$(call arch-xtensa-overlay-extract,$(@D),binutils)
endef
+BINUTILS_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
BINUTILS_POST_EXTRACT_HOOKS += BINUTILS_XTENSA_OVERLAY_EXTRACT
BINUTILS_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
+HOST_BINUTILS_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
HOST_BINUTILS_POST_EXTRACT_HOOKS += BINUTILS_XTENSA_OVERLAY_EXTRACT
HOST_BINUTILS_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
endif
diff --git a/package/gcc/gcc-final/gcc-final.mk b/package/gcc/gcc-final/gcc-final.mk
index 24d034b720..57a7e6d8a2 100644
--- a/package/gcc/gcc-final/gcc-final.mk
+++ b/package/gcc/gcc-final/gcc-final.mk
@@ -17,6 +17,7 @@ HOST_GCC_FINAL_DEPENDENCIES = \
HOST_GCC_FINAL_EXCLUDES = $(HOST_GCC_EXCLUDES)
ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
+HOST_GCC_FINAL_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
HOST_GCC_FINAL_POST_EXTRACT_HOOKS += HOST_GCC_XTENSA_OVERLAY_EXTRACT
HOST_GCC_FINAL_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
endif
diff --git a/package/gcc/gcc-initial/gcc-initial.mk b/package/gcc/gcc-initial/gcc-initial.mk
index f5bc6c5b77..36460d3712 100644
--- a/package/gcc/gcc-initial/gcc-initial.mk
+++ b/package/gcc/gcc-initial/gcc-initial.mk
@@ -18,6 +18,7 @@ HOST_GCC_INITIAL_DEPENDENCIES = $(HOST_GCC_COMMON_DEPENDENCIES)
HOST_GCC_INITIAL_EXCLUDES = $(HOST_GCC_EXCLUDES)
ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
+HOST_GCC_INITIAL_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
HOST_GCC_INITIAL_POST_EXTRACT_HOOKS += HOST_GCC_XTENSA_OVERLAY_EXTRACT
HOST_GCC_INITIAL_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
endif
diff --git a/package/gdb/gdb.mk b/package/gdb/gdb.mk
index 46d745a897..b1bfca1535 100644
--- a/package/gdb/gdb.mk
+++ b/package/gdb/gdb.mk
@@ -48,8 +48,10 @@ ifneq ($(ARCH_XTENSA_OVERLAY_FILE),)
define GDB_XTENSA_OVERLAY_EXTRACT
$(call arch-xtensa-overlay-extract,$(@D),gdb)
endef
+GDB_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
GDB_POST_EXTRACT_HOOKS += GDB_XTENSA_OVERLAY_EXTRACT
GDB_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
+HOST_GDB_EXTRACT_DEPENDENCIES += $(ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES)
HOST_GDB_POST_EXTRACT_HOOKS += GDB_XTENSA_OVERLAY_EXTRACT
HOST_GDB_EXTRA_DOWNLOADS += $(ARCH_XTENSA_OVERLAY_URL)
endif
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 05/12] arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract
2019-12-10 21:59 ` [Buildroot] [PATCH 05/12] arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract Thomas Petazzoni
@ 2019-12-10 22:09 ` Max Filippov
0 siblings, 0 replies; 24+ messages in thread
From: Max Filippov @ 2019-12-10 22:09 UTC (permalink / raw)
To: buildroot
On Tue, Dec 10, 2019 at 2:00 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> arch-xtensa-overlay-extract was used in gcc, linux, uboot, binutils
> and gdb without properly accouting for the package dependencies that
> may be needed to extract the overlay tarball.
>
> This commit fixes that by introducing an
> ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable in arch.mk.xtensa,
> and then use it where appropriate in gcc, linux, uboot, binutils and
> gdb.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
> Note: if this is preferred, we could avoid adding the
> ARCH_XTENSA_OVERLAY_EXTRACT_DEPENDENCIES variable, and directly call
> extractor-pkg-dependency from each of gcc, linux, uboot, binutils and
> gdb. Having an intermediate variable felt a bit simpler/cleaner, but I
> wouldn't mind changing it.
> ---
> arch/arch.mk.xtensa | 2 ++
> boot/uboot/uboot.mk | 1 +
> linux/linux.mk | 1 +
> package/binutils/binutils.mk | 2 ++
> package/gcc/gcc-final/gcc-final.mk | 1 +
> package/gcc/gcc-initial/gcc-initial.mk | 1 +
> package/gdb/gdb.mk | 2 ++
> 7 files changed, 10 insertions(+)
Reviewed-by: Max Filippov <jcmvbkbc@gmail.com>
--
Thanks.
-- Max
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH 06/12] package/android-tools: add missing extract dependencies
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (4 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 05/12] arch/arch.mk.xtensa: fix usage of arch-xtensa-overlay-extract Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 07/12] package/cargo: " Thomas Petazzoni
` (6 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
android-tools uses suitable-extractor to manually extract a tarball,
so it should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/android-tools/android-tools.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/package/android-tools/android-tools.mk b/package/android-tools/android-tools.mk
index 3d5c76a322..fda58e13bb 100644
--- a/package/android-tools/android-tools.mk
+++ b/package/android-tools/android-tools.mk
@@ -22,7 +22,9 @@ define ANDROID_TOOLS_DEBIAN_EXTRACT
endef
HOST_ANDROID_TOOLS_POST_EXTRACT_HOOKS += ANDROID_TOOLS_DEBIAN_EXTRACT
+HOST_ANDROID_TOOLS_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(ANDROID_TOOLS_EXTRA_DOWNLOADS))
ANDROID_TOOLS_POST_EXTRACT_HOOKS += ANDROID_TOOLS_DEBIAN_EXTRACT
+ANDROID_TOOLS_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(ANDROID_TOOLS_EXTRA_DOWNLOADS))
# Apply the Debian patches before applying the Buildroot patches
define ANDROID_TOOLS_DEBIAN_PATCH
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 07/12] package/cargo: add missing extract dependencies
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (5 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 06/12] package/android-tools: add missing extract dependencies Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 08/12] package/gettext-tiny: " Thomas Petazzoni
` (5 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
cargo uses suitable-extractor to manually extract a tarball, so it
should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/cargo/cargo.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/package/cargo/cargo.mk b/package/cargo/cargo.mk
index a387281b4c..adb7bc282e 100644
--- a/package/cargo/cargo.mk
+++ b/package/cargo/cargo.mk
@@ -42,6 +42,7 @@ define HOST_CARGO_EXTRACT_DEPS
endef
HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_DEPS
+HOST_CARGO_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(CARGO_DEPS_SOURCE))
define HOST_CARGO_EXTRACT_INSTALLER
@mkdir -p $(@D)/src/rust-installer
@@ -51,6 +52,7 @@ define HOST_CARGO_EXTRACT_INSTALLER
endef
HOST_CARGO_POST_EXTRACT_HOOKS += HOST_CARGO_EXTRACT_INSTALLER
+HOST_CARGO_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(CARGO_INSTALLER_SOURCE))
define HOST_CARGO_SETUP_DEPS
mkdir -p $(@D)/.cargo
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 08/12] package/gettext-tiny: add missing extract dependencies
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (6 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 07/12] package/cargo: " Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 09/12] package/murata-cyw-fw: " Thomas Petazzoni
` (4 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
gettext-tiny uses suitable-extractor to manually extract a tarball, so
it should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.
Fixes:
http://autobuild.buildroot.net/results/83c6d47c06334bef27791a59bdd491b1de124c49/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/gettext-tiny/gettext-tiny.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/gettext-tiny/gettext-tiny.mk b/package/gettext-tiny/gettext-tiny.mk
index 6ee6269e09..403f974b9d 100644
--- a/package/gettext-tiny/gettext-tiny.mk
+++ b/package/gettext-tiny/gettext-tiny.mk
@@ -39,6 +39,7 @@ define HOST_GETTEXT_TINY_EXTRACT_GNU_GETTEXT
$(TAR) --strip-components=1 -C $(@D)/gettext-gnu $(TAR_OPTIONS) -
endef
HOST_GETTEXT_TINY_POST_EXTRACT_HOOKS += HOST_GETTEXT_TINY_EXTRACT_GNU_GETTEXT
+HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(GETTEXT_GNU_SOURCE))
define HOST_GETTEXT_TINY_COPY_EXTRA_FILES
$(foreach f,$(GETTEXT_TINY_EXTRA_GETTEXT_FILES),\
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 09/12] package/murata-cyw-fw: add missing extract dependencies
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (7 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 08/12] package/gettext-tiny: " Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 10/12] package/nvidia-tegra23/nvidia-tegra32-binaries: " Thomas Petazzoni
` (3 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
murata-cyw-fw uses suitable-extractor to manually extract a tarball,
so it should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/murata-cyw-fw/murata-cyw-fw.mk | 3 +++
1 file changed, 3 insertions(+)
diff --git a/package/murata-cyw-fw/murata-cyw-fw.mk b/package/murata-cyw-fw/murata-cyw-fw.mk
index b99168b875..ea4f3fcc34 100644
--- a/package/murata-cyw-fw/murata-cyw-fw.mk
+++ b/package/murata-cyw-fw/murata-cyw-fw.mk
@@ -22,6 +22,9 @@ define MURATA_CYW_FW_EXTRACT_NVRAM_PATCH
)
endef
MURATA_CYW_FW_POST_EXTRACT_HOOKS += MURATA_CYW_FW_EXTRACT_NVRAM_PATCH
+MURATA_CYW_FW_EXTRACT_DEPENDENCIES += \
+ $(foreach tar,$(notdir $(MURATA_CYW_FW_EXTRA_DOWNLOADS)),\
+ $(call extractor-pkg-dependency,$(tar)))
MURATA_CYW_FW_FILES_$(BR2_PACKAGE_MURATA_CYW_FW_CYW43012) += \
brcmfmac43012-sdio.bin \
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 10/12] package/nvidia-tegra23/nvidia-tegra32-binaries: add missing extract dependencies
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (8 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 09/12] package/murata-cyw-fw: " Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 11/12] package/perl: " Thomas Petazzoni
` (2 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
nvidia-tegra23-binaries uses suitable-extractor to manually extract a
tarball, so it should have the appropriate package dependencies added
to <pkg>_EXTRACT_DEPENDENCIES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
.../nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk b/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk
index 9b153b0250..1785856847 100644
--- a/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk
+++ b/package/nvidia-tegra23/nvidia-tegra23-binaries/nvidia-tegra23-binaries.mk
@@ -44,6 +44,7 @@ define NVIDIA_TEGRA23_BINARIES_EXTRACT_FURTHER
$(TAR) --strip-components=0 -C $(@D)/nv_tegra/nv_sample_apps/nvgstapps/ $(TAR_OPTIONS) -
endef
NVIDIA_TEGRA23_BINARIES_POST_EXTRACT_HOOKS += NVIDIA_TEGRA23_BINARIES_EXTRACT_FURTHER
+NVIDIA_TEGRA23_BINARIES_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,nvidia_drivers.tbz2)
define NVIDIA_TEGRA23_BINARIES_INSTALL_LIBS
mkdir -p $(1)/usr/lib
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 11/12] package/perl: add missing extract dependencies
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (9 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 10/12] package/nvidia-tegra23/nvidia-tegra32-binaries: " Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-10 21:59 ` [Buildroot] [PATCH 12/12] package/rust-bin: " Thomas Petazzoni
2019-12-14 22:10 ` [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Yann E. MORIN
12 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
perl uses suitable-extractor to manually extract a tarball, so it
should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/perl/perl.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/perl/perl.mk b/package/perl/perl.mk
index 0ab81ddb34..3b8b9fb0c4 100644
--- a/package/perl/perl.mk
+++ b/package/perl/perl.mk
@@ -27,6 +27,7 @@ define PERL_CROSS_EXTRACT
$(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(PERL_DL_DIR)/$(PERL_CROSS_SOURCE) | \
$(TAR) --strip-components=1 -C $(@D) $(TAR_OPTIONS) -
endef
+PERL_EXTRACT_DEPENDENCIES += $(call extractor-pkg-dependency,$(PERL_CROSS_SOURCE))
PERL_POST_EXTRACT_HOOKS += PERL_CROSS_EXTRACT
# Even though perl is not an autotools-package, it uses config.sub and
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 12/12] package/rust-bin: add missing extract dependencies
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (10 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 11/12] package/perl: " Thomas Petazzoni
@ 2019-12-10 21:59 ` Thomas Petazzoni
2019-12-14 22:10 ` [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Yann E. MORIN
12 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2019-12-10 21:59 UTC (permalink / raw)
To: buildroot
rust-bin uses suitable-extractor to manually extract a tarball, so it
should have the appropriate package dependencies added to
<pkg>_EXTRACT_DEPENDENCIES.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
package/rust-bin/rust-bin.mk | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/package/rust-bin/rust-bin.mk b/package/rust-bin/rust-bin.mk
index a374215e20..526c45a45e 100644
--- a/package/rust-bin/rust-bin.mk
+++ b/package/rust-bin/rust-bin.mk
@@ -32,7 +32,9 @@ define HOST_RUST_BIN_LIBSTD_EXTRACT
cd $(@D)/rustc/lib/rustlib/$(RUSTC_HOST_NAME)/lib; \
ln -sf ../../../../../std/$(HOST_RUST_BIN_LIBSTD_HOST_PREFIX)/lib/rustlib/$(RUSTC_HOST_NAME)/lib/* .
endef
-
+HOST_RUST_BIN_EXTRACT_DEPENDENCIES += \
+ $(foreach f,$(HOST_RUST_BIN_EXTRA_DOWNLOADS),\
+ $(call extractor-pkg-dependency,$(f)))
HOST_RUST_BIN_POST_EXTRACT_HOOKS += HOST_RUST_BIN_LIBSTD_EXTRACT
HOST_RUST_BIN_INSTALL_OPTS = \
--
2.23.0
^ permalink raw reply related [flat|nested] 24+ messages in thread* [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account
2019-12-10 21:59 [Buildroot] [PATCH 00/12] Ensure extractor dependencies are taken into account Thomas Petazzoni
` (11 preceding siblings ...)
2019-12-10 21:59 ` [Buildroot] [PATCH 12/12] package/rust-bin: " Thomas Petazzoni
@ 2019-12-14 22:10 ` Yann E. MORIN
12 siblings, 0 replies; 24+ messages in thread
From: Yann E. MORIN @ 2019-12-14 22:10 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2019-12-10 22:59 +0100, Thomas Petazzoni spake thusly:
> This series solves that by introducing an extractor-pkg-dependency
> that returns the list of packages that must be added to
> <pkg>_EXTRACT_DEPENDENCIES to extract the archive passed as
> argument. It is then used in package/pkg-generic.mk to replace the
> existing logic to calculate <pkg>_EXTRACT_DEPENDENCIES, and then in
> the several places where suitable-extractor was used to manually
> extract a tarball.
Here is a summarry of our little discussion on IRC:
- the extract dependencies for the main _SOURCE are automatically
computed by Buildroot, but not for the _EXTRA_DOWNLOADS, which is
not very systematic;
- practically 99% of EXTRA_DOWNLOADS that are compressed, will be
extracted at build time; the cases for (e.g.) an .xz to be stored
as-is on the target is expected to be very rare;
- so we should change the logic in this way (basically):
$(2)_EXTRACT_DEPENDENCIES = $(foreach URI,$($(2)_SOURCE) $($(2)_EXTRA_DOWNLOADS),$(call extractor-pkg-dependency,$(notdir $(URI))))
Regards,
Yann E. MORIN.
> We also rename extractor-dependency to extractor-system-dependency to
> clarify its difference from extractor-pkg-dependency, and we rework
> the implementation of extractor-system-dependency.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread