* [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature
@ 2017-10-25 20:09 Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check Peter Seiderer
` (12 more replies)
0 siblings, 13 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
On Tue, 4 Jul 2017 18:21:58 +0200, Maxime Hadjinlian <maxime.hadjinlian@gmail.com> wrote:
> Hi all,
>
> This series will progressively reorganize the DL_DIR folder, in order to
> hve a subdirectory for each packages.
> This way, we store patches into subdirectory called by the package name
> instead of flattly into DL_DIR.
> We also avoid possible conflicts between packages.
>
> Finally, this series introduce the 'git cache' feature. What it does is
> that it keeps a copy of the Git clone in the subdirectory of the
> package.
> This way, when you change the version of said package, instead of
> cloning from scratch, we only fetch the missing objects and create the
> tarball from that git clone.
>
Updated version from https://github.com/maximeh/buildroot/commits/git_cache
with added change log for v2.
Maxime Hadjinlian (12):
pkg-{download,generic}: remove source-check
download: put most of the infra in dl-wrapper
packages: use new $($PKG)_DL_DIR) variable
pkg-{download,generic}: use new $($(PKG)_DL_DIR)
support/download: make sure the download folder is created
pkg-generic: add a subdirectory to the DL_DIR
pkg-download: support new subdir for mirrors
pkg-generic: introduce _SAME_SOURCE_AS
help/manual: update help about the new $(LIBFOO_DL_DIR)
download: add flock call before dl-wrapper
download: git: introduce cache feature
WIP: support package with '-'
Yann E. MORIN (1):
core/pkg-download: change all helpers to use common options
Config.in | 3 +
Makefile | 7 +-
docs/manual/adding-packages-generic.txt | 6 +-
package/amd-catalyst/amd-catalyst.mk | 2 +-
package/android-tools/android-tools.mk | 2 +-
package/angularjs/angularjs.mk | 2 +-
package/bootstrap/bootstrap.mk | 2 +-
package/cache-calibrator/cache-calibrator.mk | 2 +-
package/cracklib/cracklib.mk | 2 +-
package/cryptopp/cryptopp.mk | 2 +-
package/devmem2/devmem2.mk | 2 +-
package/dhrystone/dhrystone.mk | 2 +-
package/doom-wad/doom-wad.mk | 2 +-
package/espeak/espeak.mk | 2 +-
package/fan-ctrl/fan-ctrl.mk | 2 +-
package/gcc/gcc.mk | 2 +-
package/irrlicht/irrlicht.mk | 2 +-
package/jquery-mobile/jquery-mobile.mk | 2 +-
package/jquery-sparkline/jquery-sparkline.mk | 2 +-
package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
package/jquery-ui/jquery-ui.mk | 2 +-
package/jquery-validation/jquery-validation.mk | 2 +-
package/jquery/jquery.mk | 2 +-
package/libfreeimage/libfreeimage.mk | 2 +-
package/libjson/libjson.mk | 2 +-
package/libsoil/libsoil.mk | 2 +-
package/lsof/lsof.mk | 2 +-
package/musl-compat-headers/musl-compat-headers.mk | 2 +-
package/nmon/nmon.mk | 2 +-
package/nvidia-driver/nvidia-driver.mk | 2 +-
package/openobex/openobex.mk | 6 +
package/opentyrian-data/opentyrian-data.mk | 2 +-
package/perl/perl.mk | 2 +-
package/pkg-download.mk | 216 ++++-----------------
package/pkg-generic.mk | 20 +-
package/python-keyring/python-keyring.mk | 2 +-
.../python-simplegeneric/python-simplegeneric.mk | 2 +-
package/rapidxml/rapidxml.mk | 2 +-
package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
package/sam-ba/sam-ba.mk | 2 +-
package/spidev_test/spidev_test.mk | 2 +-
package/tar/tar.mk | 2 +-
package/tesseract-ocr/tesseract-ocr.mk | 2 +-
package/ti-cgt-pru/ti-cgt-pru.mk | 4 +-
package/ti-gfx/ti-gfx.mk | 4 +-
package/ts4900-fpga/ts4900-fpga.mk | 2 +-
package/unscd/unscd.mk | 2 +-
package/urg/urg.mk | 4 +-
package/waf/waf.mk | 2 +-
package/whetstone/whetstone.mk | 2 +-
package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
support/download/bzr | 28 +--
support/download/cvs | 37 ++--
support/download/dl-wrapper | 110 +++++++----
support/download/{cp => file} | 22 ++-
support/download/git | 99 ++++++----
support/download/hg | 28 +--
support/download/scp | 20 +-
support/download/svn | 28 +--
support/download/wget | 28 ++-
61 files changed, 345 insertions(+), 411 deletions(-)
rename support/download/{cp => file} (57%)
--
2.14.2
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2018-01-08 20:22 ` Yann E. MORIN
2018-01-08 20:28 ` Thomas Petazzoni
2017-10-25 20:09 ` [Buildroot] [PATCH v2 02/13] core/pkg-download: change all helpers to use common options Peter Seiderer
` (11 subsequent siblings)
12 siblings, 2 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
This feature is not used by anyone in the core developpers and makes a
drastic simplification of the pkg-download infrastructure harder.
The future patch will move much of what's in the current pkg-download.mk
file into the dl-wrapper which is a shell script.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/a4d0e83729a1ab534ff99d6f258058a1be1ed949.patch
- remove more references to source-check (suggested by Thomas Petazzoni)
---
Makefile | 7 +------
package/pkg-download.mk | 44 --------------------------------------------
package/pkg-generic.mk | 14 +-------------
3 files changed, 2 insertions(+), 63 deletions(-)
diff --git a/Makefile b/Makefile
index 79db7fe48a..45047f7e9a 100644
--- a/Makefile
+++ b/Makefile
@@ -141,7 +141,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf
# We're building in two situations: when MAKECMDGOALS is empty
# (default target is to build), or when MAKECMDGOALS contains
# something else than one of the nobuild_targets.
-nobuild_targets := source %-source source-check \
+nobuild_targets := source %-source \
legal-info %-legal-info external-deps _external-deps \
clean distclean help show-targets graph-depends \
%-graph-depends %-show-depends %-show-version \
@@ -754,10 +754,6 @@ _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps)
external-deps:
@$(MAKE1) -Bs $(EXTRAMAKEARGS) _external-deps | sort -u
-# check if download URLs are outdated
-.PHONY: source-check
-source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check)
-
.PHONY: legal-info-clean
legal-info-clean:
@rm -fr $(LEGAL_INFO_DIR)
@@ -1044,7 +1040,6 @@ help:
@echo
@echo 'Miscellaneous:'
@echo ' source - download all sources needed for offline-build'
- @echo ' source-check - check selected packages for valid download URLs'
@echo ' external-deps - list external packages used'
@echo ' legal-info - generate info about license compliance'
@echo ' printvars - dump all the internal variables'
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index dc4ff1c8c7..2b845e77a2 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -68,9 +68,6 @@ export BR_NO_CHECK_HASH_FOR =
# ssh authentication. DOWNLOAD_WGET is the normal wget-based download
# mechanism.
#
-# The SOURCE_CHECK_* helpers are in charge of simply checking that the source
-# is available for download. This can be used to make sure one will be able
-# to get all the sources needed for one's build configuration.
################################################################################
define DOWNLOAD_GIT
@@ -86,12 +83,6 @@ define DOWNLOAD_GIT
$($(PKG)_DL_OPTS)
endef
-# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
-# repository
-define SOURCE_CHECK_GIT
- $(GIT) ls-remote --heads $($(PKG)_SITE) > /dev/null
-endef
-
define DOWNLOAD_BZR
$(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -103,10 +94,6 @@ define DOWNLOAD_BZR
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_BZR
- $(BZR) ls --quiet $($(PKG)_SITE) > /dev/null
-endef
-
define DOWNLOAD_CVS
$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -119,11 +106,6 @@ define DOWNLOAD_CVS
$($(PKG)_DL_OPTS)
endef
-# Not all CVS servers support ls/rls, use login to see if we can connect
-define SOURCE_CHECK_CVS
- $(CVS) -d:pserver:anonymous:@$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) login
-endef
-
define DOWNLOAD_SVN
$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -135,10 +117,6 @@ define DOWNLOAD_SVN
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_SVN
- $(SVN) ls $($(PKG)_SITE)@$($(PKG)_DL_VERSION) > /dev/null
-endef
-
# SCP URIs should be of the form scp://[user@]host:filepath
# Note that filepath is relative to the user's home directory, so you may want
# to prepend the path with a slash: scp://[user@]host:/absolutepath
@@ -152,10 +130,6 @@ define DOWNLOAD_SCP
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_SCP
- $(SSH) $(call domain,$(1),:) ls '$(call notdomain,$(1),:)' > /dev/null
-endef
-
define DOWNLOAD_HG
$(EXTRA_ENV) $(DL_WRAPPER) -b hg \
-o $(DL_DIR)/$($(PKG)_SOURCE) \
@@ -167,12 +141,6 @@ define DOWNLOAD_HG
$($(PKG)_DL_OPTS)
endef
-# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
-# repository
-define SOURCE_CHECK_HG
- $(HG) incoming --force -l1 $($(PKG)_SITE) > /dev/null
-endef
-
define DOWNLOAD_WGET
$(EXTRA_ENV) $(DL_WRAPPER) -b wget \
-o $(DL_DIR)/$(2) \
@@ -183,10 +151,6 @@ define DOWNLOAD_WGET
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_WGET
- $(WGET) --spider '$(call qstrip,$(1))'
-endef
-
define DOWNLOAD_LOCALFILES
$(EXTRA_ENV) $(DL_WRAPPER) -b cp \
-o $(DL_DIR)/$(2) \
@@ -197,10 +161,6 @@ define DOWNLOAD_LOCALFILES
$($(PKG)_DL_OPTS)
endef
-define SOURCE_CHECK_LOCALFILES
- test -e $(call stripurischeme,$(call qstrip,$(1)))
-endef
-
################################################################################
# DOWNLOAD -- Download helper. Will try to download source from:
# 1) BR2_PRIMARY_SITE if enabled
@@ -221,10 +181,6 @@ define DOWNLOAD
$(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
endef
-define SOURCE_CHECK
- $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),SOURCE_CHECK)
-endef
-
define DOWNLOAD_INNER
$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index cca94ba338..5989417902 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -706,10 +706,6 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE)
endif # actual sources != sources
endif # actual sources != ""
-$(1)-source-check: PKG=$(2)
-$(1)-source-check:
- $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep))
-
$(1)-external-deps:
@for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \
echo `basename $$$$p` ; \
@@ -734,9 +730,6 @@ $(1)-rsync: $$($(2)_TARGET_RSYNC)
$(1)-source:
$(1)-legal-source:
-$(1)-source-check:
- test -d $$($(2)_OVERRIDE_SRCDIR)
-
$(1)-external-deps:
@echo "file://$$($(2)_OVERRIDE_SRCDIR)"
endif
@@ -762,9 +755,6 @@ $(1)-graph-rdepends: graph-depends-requirements
$(1)-all-source: $(1)-source
$(1)-all-source: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source)
-$(1)-all-source-check: $(1)-source-check
-$(1)-all-source-check: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check)
-
$(1)-all-external-deps: $(1)-external-deps
$(1)-all-external-deps: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)
@@ -964,7 +954,6 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
$(1)-all-external-deps \
$(1)-all-legal-info \
$(1)-all-source \
- $(1)-all-source-check \
$(1)-build \
$(1)-clean-for-rebuild \
$(1)-clean-for-reconfigure \
@@ -989,8 +978,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
$(1)-rsync \
$(1)-show-depends \
$(1)-show-version \
- $(1)-source \
- $(1)-source-check
+ $(1)-source
ifneq ($$($(2)_SOURCE),)
ifeq ($$($(2)_SITE),)
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 02/13] core/pkg-download: change all helpers to use common options
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2018-02-05 15:34 ` Luca Ceresoli
2017-10-25 20:09 ` [Buildroot] [PATCH v2 03/13] download: put most of the infra in dl-wrapper Peter Seiderer
` (10 subsequent siblings)
12 siblings, 1 reply; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: "Yann E. MORIN" <yann.morin.1998@free.fr>
Currently all download helpers accepts the local output file, the remote
locations, the changesets and so on... as positional arguments.
This was well and nice when that's was all we needed.
But then we added an option to quiesce their verbosity, and that was
shoehorned with a trivial getopts, still keeping all the existing
positional arguments as... positional arguments.
Adding yet more options while keeping positional arguments will not be
very easy, even if we do not envision any new option in the foreseeable
future (but 640K ought to be enough for everyone, remember? ;-) ).
Change all helpers to accept a set of generic options (-q for quiet and
-o for the output file) as well as helper-specific options (like -r for
the repository, -c for a changeset...).
Maxime:
Changed -R to -r for recurse (only for the git backend)
Changed -r to -u for URI (for all backend)
Change -R to -c for cset (for CVS and SVN backend)
Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
can use the same option easily
Now all the backends use the same common options.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/a1facaac906794d027a0ed5837fb617f93cad662.patch
- no more change to support/download/check-hash
---
package/pkg-download.mk | 38 +++++++++++++++++++-------------------
support/download/bzr | 25 ++++++++++++++-----------
support/download/cp | 17 +++++++++--------
support/download/cvs | 34 +++++++++++++++++++---------------
support/download/dl-wrapper | 7 ++++++-
support/download/git | 33 +++++++++++++++++----------------
support/download/hg | 25 ++++++++++++++-----------
support/download/scp | 19 ++++++++++---------
support/download/svn | 25 ++++++++++++++-----------
support/download/wget | 17 +++++++++--------
10 files changed, 131 insertions(+), 109 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 2b845e77a2..4d724e7494 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -77,9 +77,9 @@ define DOWNLOAD_GIT
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -88,9 +88,9 @@ define DOWNLOAD_BZR
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -99,10 +99,10 @@ define DOWNLOAD_CVS
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAWNAME) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
+ -c $($(PKG)_DL_VERSION) \
+ -N $($(PKG)_RAWNAME) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -111,9 +111,9 @@ define DOWNLOAD_SVN
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -126,7 +126,7 @@ define DOWNLOAD_SCP
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- '$(call stripurischeme,$(call qstrip,$(1)))' \
+ -u '$(call stripurischeme,$(call qstrip,$(1)))' \
$($(PKG)_DL_OPTS)
endef
@@ -135,9 +135,9 @@ define DOWNLOAD_HG
-o $(DL_DIR)/$($(PKG)_SOURCE) \
$(QUIET) \
-- \
- $($(PKG)_SITE) \
- $($(PKG)_DL_VERSION) \
- $($(PKG)_RAW_BASE_NAME) \
+ -u $($(PKG)_SITE) \
+ -c $($(PKG)_DL_VERSION) \
+ -n $($(PKG)_RAW_BASE_NAME) \
$($(PKG)_DL_OPTS)
endef
@@ -147,7 +147,7 @@ define DOWNLOAD_WGET
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- '$(call qstrip,$(1))' \
+ -u '$(call qstrip,$(1))' \
$($(PKG)_DL_OPTS)
endef
@@ -157,7 +157,7 @@ define DOWNLOAD_LOCALFILES
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
$(QUIET) \
-- \
- $(call stripurischeme,$(call qstrip,$(1))) \
+ -u $(call stripurischeme,$(call qstrip,$(1))) \
$($(PKG)_DL_OPTS)
endef
diff --git a/support/download/bzr b/support/download/bzr
index 75b7b415c1..5289a421cd 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -5,28 +5,31 @@ set -e
# Download helper for bzr, to be called from the download wrapper script
#
-# Call it as:
-# .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
+# Options:
+# -q Be quiet
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset (or revision) CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# BZR : the bzr command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -51,5 +54,5 @@ if [ ${bzr_version} -ge ${bzr_min_version} ]; then
fi
_bzr export ${verbose} --root="'${basename}/'" --format=tgz \
- ${timestamp_opt} - "${@}" "'${repo}'" -r "'${rev}'" \
+ ${timestamp_opt} - "${@}" "'${uri}'" -r "'${cset}'" \
>"${output}"
diff --git a/support/download/cp b/support/download/cp
index 0ee1f3ba82..52fe2de83d 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -5,8 +5,10 @@ set -e
# Download helper for cp, to be called from the download wrapper script
#
-# Call it as:
-# .../cp [-q] OUT_FILE SRC_FILE
+# Options:
+# -q Be quiet.
+# -o FILE Copy to file FILE.
+# -u FILE Copy from file FILE.
#
# Environment:
# LOCALFILES: the cp command to call
@@ -17,18 +19,17 @@ set -e
# Make 'cp' verbose by default, so it behaves a bit like the others.
verbose=-v
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=;;
+ o) output="${OPTARG}";;
+ u) source="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-source="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
diff --git a/support/download/cvs b/support/download/cvs
index 50050ab1c9..69d5c71f28 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -5,28 +5,32 @@ set -e
# Download helper for cvs, to be called from the download wrapper script
#
-# Call it as:
-# .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
+# Options:
+# -q Be quiet
+# -o FILE Generate archive in FILE.
+# -u URI Checkout from repository at URI.
+# -c REV Use revision REV.
+# -N RAWNAME Use rawname (aka module name) RAWNAME.
+# -n NAME Use basename NAME.
#
# Environment:
# CVS : the cvs command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-Q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) rev="${OPTARG}";;
+ N) rawname="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-rawname="${4}"
-basename="${5}"
-
-shift 5 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -42,14 +46,14 @@ else
select="-r"
fi
-# The absence of an initial : on ${repo} means access method undefined
-if [[ ! "${repo}" =~ ^: ]]; then
+# The absence of an initial : on ${uri} means access method undefined
+if [[ ! "${uri}" =~ ^: ]]; then
# defaults to anonymous pserver
- repo=":pserver:anonymous@${repo}"
+ uri=":pserver:anonymous@${uri}"
fi
export TZ=UTC
-_cvs ${verbose} -z3 -d"'${repo}'" \
+_cvs ${verbose} -z3 -d"'${uri}'" \
co "${@}" -d "'${basename}'" ${select} "'${rev}'" -P "'${rawname}'"
tar czf "${output}" "${basename}"
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index f944b71db5..510e7ef852 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,6 +19,8 @@
# We want to catch any unexpected failure, and exit immediately.
set -e
+export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:q"
+
main() {
local OPT OPTARG
local backend output hfile recurse quiet
@@ -83,7 +85,10 @@ main() {
# If the backend fails, we can just remove the temporary directory to
# remove all the cruft it may have left behind. Then we just exit in
# error too.
- if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
+ if ! "${OLDPWD}/support/download/${backend}" \
+ ${quiet} ${recurse} \
+ -o "${tmpf}" "${@}"
+ then
rm -rf "${tmpd}"
exit 1
fi
diff --git a/support/download/git b/support/download/git
index f590ff6494..58a2c6ad9d 100755
--- a/support/download/git
+++ b/support/download/git
@@ -5,32 +5,33 @@ set -e
# Download helper for git, to be called from the download wrapper script
#
-# Call it as:
-# .../git [-q] [-r] OUT_FILE REPO_URL CSET BASENAME
-#
-# -q Be quiet.
-# -r Clone and archive sub-modules.
+# Options:
+# -q Be quiet.
+# -r Clone and archive sub-modules.
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# GIT : the git command to call
verbose=
recurse=0
-while getopts :qr OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q; exec >/dev/null;;
r) recurse=1;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-
-output="${1}"
-repo="${2}"
-cset="${3}"
-basename="${4}"
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -46,9 +47,9 @@ _git() {
# Messages for the type of clone used are provided to ease debugging in case of
# problems
git_done=0
-if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
+if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
printf "Doing shallow clone\n"
- if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${repo}'" "'${basename}'"; then
+ if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
git_done=1
else
printf "Shallow clone failed, falling back to doing a full clone\n"
@@ -56,7 +57,7 @@ if [ -n "$(_git ls-remote "'${repo}'" "'${cset}'" 2>&1)" ]; then
fi
if [ ${git_done} -eq 0 ]; then
printf "Doing full clone\n"
- _git clone ${verbose} "${@}" "'${repo}'" "'${basename}'"
+ _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
fi
pushd "${basename}" >/dev/null
diff --git a/support/download/hg b/support/download/hg
index 3af01690b3..efb515fca5 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -5,27 +5,30 @@ set -e
# Download helper for hg, to be called from the download wrapper script
#
-# Call it as:
-# .../hg [-q] OUT_FILE REPO_URL CSET BASENAME
+# Options:
+# -q Be quiet.
+# -o FILE Generate archive in FILE.
+# -u URI Clone from repository at URI.
+# -c CSET Use changeset (or revision) CSET.
+# -n NAME Use basename NAME.
#
# Environment:
# HG : the hg command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) cset="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-cset="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -33,7 +36,7 @@ _hg() {
eval ${HG} "${@}"
}
-_hg clone ${verbose} "${@}" --noupdate "'${repo}'" "'${basename}'"
+_hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
_hg archive ${verbose} --repository "'${basename}'" --type tgz \
--prefix "'${basename}'" --rev "'${cset}'" \
diff --git a/support/download/scp b/support/download/scp
index 825fd41c64..8ecf2f4b22 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -5,25 +5,26 @@ set -e
# Download helper for scp, to be called from the download wrapper script
#
-# Call it as:
-# .../scp [-q] OUT_FILE SRC_URL
+# Options:
+# -q Be quiet.
+# -o FILE Copy to local file FILE.
+# -u FILE Copy from remote file FILE.
#
# Environment:
# SCP : the scp command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-url="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -31,4 +32,4 @@ _scp() {
eval ${SCP} "${@}"
}
-_scp ${verbose} "${@}" "'${url}'" "'${output}'"
+_scp ${verbose} "${@}" "'${uri}'" "'${output}'"
diff --git a/support/download/svn b/support/download/svn
index 77abf3d02d..542b25c0a2 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -5,27 +5,30 @@ set -e
# Download helper for svn, to be called from the download wrapper script
#
-# Call it as:
-# .../svn [-q] OUT_FILE REPO_URL REV BASNAME
+# Options:
+# -q Be quiet.
+# -o FILE Generate archive in FILE.
+# -u URI Checkout from repository at URI.
+# -c REV Use revision REV.
+# -n NAME Use basename NAME.
#
# Environment:
# SVN : the svn command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) uri="${OPTARG}";;
+ c) rev="${OPTARG}";;
+ n) basename="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-repo="${2}"
-rev="${3}"
-basename="${4}"
-
-shift 4 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
@@ -33,6 +36,6 @@ _svn() {
eval ${SVN} "${@}"
}
-_svn export ${verbose} "${@}" "'${repo}@${rev}'" "'${basename}'"
+_svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
tar czf "${output}" "${basename}"
diff --git a/support/download/wget b/support/download/wget
index 768de904c3..fece6663ca 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -5,25 +5,26 @@ set -e
# Download helper for wget, to be called from the download wrapper script
#
-# Call it as:
-# .../wget [-q] OUT_FILE URL
+# Options:
+# -q Be quiet.
+# -o FILE Save into file FILE.
+# -u URL Download file at URL.
#
# Environment:
# WGET : the wget command to call
verbose=
-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ o) output="${OPTARG}";;
+ u) url="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
done
-shift $((OPTIND-1))
-output="${1}"
-url="${2}"
-
-shift 2 # Get rid of our options
+shift $((OPTIND-1)) # Get rid of our options
# Caller needs to single-quote its arguments to prevent them from
# being expanded a second time (in case there are spaces in them)
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 03/13] download: put most of the infra in dl-wrapper
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 02/13] core/pkg-download: change all helpers to use common options Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2018-02-05 15:34 ` Luca Ceresoli
2017-10-25 20:09 ` [Buildroot] [PATCH v2 04/13] packages: use new $($PKG)_DL_DIR) variable Peter Seiderer
` (9 subsequent siblings)
12 siblings, 1 reply; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
The goal here is to simplify the infrastructure by putting most of the
code in the dl-wrapper as it's easier to implement and to read.
Most of the function where common already, this patch finalizes it by
making the pkg-download.mk pass all the parameters needed to the
dl-wrapper which in turns will pass everything to every backend.
The backend will then cherry-pick what it needs from these arguments
and act accordingly.
It eases the transition to the addition of a sub directory per package
in the DL_DIR, and later on, a git cache.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/5ade24239ae84259711170c0ec9fe7384baaddd4.patch
- change getschemeplusuri from git+http://example.com to git|paremetr+http://example.com
- use urlencode
- rename support/download/cp to support/download/file
- fixes some typos
---
package/pkg-download.mk | 166 ++++++++----------------------------------
support/download/cvs | 2 +-
support/download/dl-wrapper | 108 ++++++++++++++++++---------
support/download/{cp => file} | 4 +-
support/download/wget | 10 ++-
5 files changed, 113 insertions(+), 177 deletions(-)
rename support/download/{cp => file} (90%)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 4d724e7494..233a9fafc7 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -42,6 +42,8 @@ DL_DIR := $(shell mkdir -p $(DL_DIR) && cd $(DL_DIR) >/dev/null && pwd)
#
# geturischeme: http
geturischeme = $(firstword $(subst ://, ,$(call qstrip,$(1))))
+# getschemeplusuri: git|paremetr+http://example.com
+getschemeplusuri = $(call geturischeme,$(1))$(if $(2),\|$(2))+$(1)
# stripurischeme: www.example.com/dir/file
stripurischeme = $(lastword $(subst ://, ,$(call qstrip,$(1))))
# domain: www.example.com
@@ -61,152 +63,42 @@ github = https://github.com/$(1)/$(2)/archive/$(or $(3),$($(call UPPERCASE,$(pkg
export BR_NO_CHECK_HASH_FOR =
################################################################################
-# The DOWNLOAD_* helpers are in charge of getting a working copy
-# of the source repository for their corresponding SCM,
-# checking out the requested version / commit / tag, and create an
-# archive out of it. DOWNLOAD_SCP uses scp to obtain a remote file with
-# ssh authentication. DOWNLOAD_WGET is the normal wget-based download
-# mechanism.
+# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
+# source from:
+# 1) BR2_PRIMARY_SITE if enabled
+# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
+# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
+#
+# Argument 1 is the source location
#
################################################################################
-define DOWNLOAD_GIT
- $(EXTRA_ENV) $(DL_WRAPPER) -b git \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(if $($(PKG)_GIT_SUBMODULES),-r) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_BZR
- $(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
+ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
+DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE),urlencode)
+endif
-define DOWNLOAD_CVS
- $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
- -c $($(PKG)_DL_VERSION) \
- -N $($(PKG)_RAWNAME) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
+ifeq ($(BR2_PRIMARY_SITE_ONLY),)
+DOWNLOAD_URIS += \
+ -u $($(PKG)_SITE_METHOD)+$(dir $(1))
+ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
+DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_BACKUP_SITE),urlencode)
+endif
+endif
-define DOWNLOAD_SVN
- $(EXTRA_ENV) $(DL_WRAPPER) -b svn \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
+define DOWNLOAD
+ $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
+ $(EXTRA_ENV) $(DL_WRAPPER) \
-c $($(PKG)_DL_VERSION) \
- -n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
-
-# SCP URIs should be of the form scp://[user@]host:filepath
-# Note that filepath is relative to the user's home directory, so you may want
-# to prepend the path with a slash: scp://[user@]host:/absolutepath
-define DOWNLOAD_SCP
- $(EXTRA_ENV) $(DL_WRAPPER) -b scp \
- -o $(DL_DIR)/$(2) \
+ -f $(notdir $(1)) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u '$(call stripurischeme,$(call qstrip,$(1)))' \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_HG
- $(EXTRA_ENV) $(DL_WRAPPER) -b hg \
- -o $(DL_DIR)/$($(PKG)_SOURCE) \
- $(QUIET) \
- -- \
- -u $($(PKG)_SITE) \
- -c $($(PKG)_DL_VERSION) \
-n $($(PKG)_RAW_BASE_NAME) \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_WGET
- $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
- -o $(DL_DIR)/$(2) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
- $(QUIET) \
- -- \
- -u '$(call qstrip,$(1))' \
- $($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_LOCALFILES
- $(EXTRA_ENV) $(DL_WRAPPER) -b cp \
- -o $(DL_DIR)/$(2) \
- -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+ -N $($(PKG)_RAWNAME) \
+ -o $(DL_DIR)/$(notdir $(1)) \
+ $(if $($(PKG)_GIT_SUBMODULES),-r) \
+ $(DOWNLOAD_URIS) \
$(QUIET) \
-- \
- -u $(call stripurischeme,$(call qstrip,$(1))) \
$($(PKG)_DL_OPTS)
endef
-
-################################################################################
-# DOWNLOAD -- Download helper. Will try to download source from:
-# 1) BR2_PRIMARY_SITE if enabled
-# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
-# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
-#
-# Argument 1 is the source location
-#
-# E.G. use like this:
-# $(call DOWNLOAD,$(FOO_SITE))
-#
-# For PRIMARY and BACKUP site, any ? in the URL is replaced by %3F. A ? in
-# the URL is used to separate query arguments, but the PRIMARY and BACKUP
-# sites serve just plain files.
-################################################################################
-
-define DOWNLOAD
- $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
-endef
-
-define DOWNLOAD_INNER
- $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
- if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
- case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
- file) $(call $(3)_LOCALFILES,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
- scp) $(call $(3)_SCP,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
- *) $(call $(3)_WGET,$(BR2_PRIMARY_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ;; \
- esac ; \
- fi ; \
- if test "$(BR2_PRIMARY_SITE_ONLY)" = "y" ; then \
- exit 1 ; \
- fi ; \
- if test -n "$(1)" ; then \
- case "$($(PKG)_SITE_METHOD)" in \
- git) $($(3)_GIT) && exit ;; \
- svn) $($(3)_SVN) && exit ;; \
- cvs) $($(3)_CVS) && exit ;; \
- bzr) $($(3)_BZR) && exit ;; \
- file) $($(3)_LOCALFILES) && exit ;; \
- scp) $($(3)_SCP) && exit ;; \
- hg) $($(3)_HG) && exit ;; \
- *) $(call $(3)_WGET,$(1),$(2)) && exit ;; \
- esac ; \
- fi ; \
- if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \
- $(call $(3)_WGET,$(BR2_BACKUP_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ; \
- fi ; \
- exit 1
-endef
diff --git a/support/download/cvs b/support/download/cvs
index 69d5c71f28..3f77b849e4 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -21,7 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-Q;;
o) output="${OPTARG}";;
- u) uri="${OPTARG}";;
+ u) uri="${OPTARG#*://}";;
c) rev="${OPTARG}";;
N) rawname="${OPTARG}";;
n) basename="${OPTARG}";;
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 510e7ef852..67e9742767 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,31 +19,34 @@
# We want to catch any unexpected failure, and exit immediately.
set -e
-export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:q"
+export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:qf:e"
main() {
local OPT OPTARG
local backend output hfile recurse quiet
+ local -a uris
# Parse our options; anything after '--' is for the backend
- while getopts :hb:o:H:rq OPT; do
+ while getopts ":hc:o:n:N:H:rf:u:q" OPT; do
case "${OPT}" in
h) help; exit 0;;
- b) backend="${OPTARG}";;
+ c) cset="${OPTARG}";;
o) output="${OPTARG}";;
+ n) raw_base_name="${OPTARG}";;
+ N) base_name="${OPTARG}";;
H) hfile="${OPTARG}";;
r) recurse="-r";;
+ f) filename="${OPTARG}";;
+ u) uris+=( "${OPTARG}" );;
q) quiet="-q";;
:) error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
\?) error "unknown option '%s'\n" "${OPTARG}";;
esac
done
+
# Forget our options, and keep only those for the backend
shift $((OPTIND-1))
- if [ -z "${backend}" ]; then
- error "no backend specified, use -b\n"
- fi
if [ -z "${output}" ]; then
error "no output specified, use -o\n"
fi
@@ -77,28 +80,64 @@ main() {
tmpd="$(mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX")"
tmpf="${tmpd}/output"
- # Helpers expect to run in a directory that is *really* trashable, so
- # they are free to create whatever files and/or sub-dirs they might need.
- # Doing the 'cd' here rather than in all backends is easier.
- cd "${tmpd}"
-
- # If the backend fails, we can just remove the temporary directory to
- # remove all the cruft it may have left behind. Then we just exit in
- # error too.
- if ! "${OLDPWD}/support/download/${backend}" \
- ${quiet} ${recurse} \
- -o "${tmpf}" "${@}"
- then
- rm -rf "${tmpd}"
- exit 1
- fi
+ # Look through all the uris that we were given to downoad the package
+ # source
+ download_and_check=0
+ for uri in "${uris[@]}"; do
+ backend=${uri%+*}
+ case "${backend}" in
+ git|svn|cvs|bzr|file|scp|hg) ;;
+ *) backend="wget" ;;
+ esac
+ uri=${uri#*+}
+
+ urlencode=${backend#*|}
+ # urlencode must be "urlencode"
+ [ "${urlencode}" != "urlencode" ] && urlencode=""
+
+ # Helpers expect to run in a directory that is *really* trashable, so
+ # they are free to create whatever files and/or sub-dirs they might need.
+ # Doing the 'cd' here rather than in all backends is easier.
+ cd "${tmpd}"
+
+ # If the backend fails, we can just remove the content of the temporary
+ # directory to remove all the cruft it may have left behind, and tries
+ # the next URI until it succeeds. Once out of URI to tries, we need to
+ # cleanup and exit.
+ if ! "${OLDPWD}/support/download/${backend}" \
+ $([ -n "${urlencode}" ] && printf %s '-e') \
+ -c "${cset}" \
+ -n "${raw_base_name}" \
+ -N "${raw_name}" \
+ -f "${filename}" \
+ -u "${uri}" \
+ -o "${tmpf}" \
+ ${quiet} ${recurse} "${@}"
+ then
+ rm -rf "${tmpd:?}/*"
+ # cd back to keep path coherence
+ cd "${OLDPWD}"
+ continue
+ fi
- # cd back to free the temp-dir, so we can remove it later
- cd "${OLDPWD}"
+ # cd back to free the temp-dir, so we can remove it later
+ cd "${OLDPWD}"
- # Check if the downloaded file is sane, and matches the stored hashes
- # for that file
- if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
+ # Check if the downloaded file is sane, and matches the stored hashes
+ # for that file
+ if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
+ rm -rf "${tmpd:?}/*"
+ # cd back to keep path coherence
+ cd "${OLDPWD}"
+ continue
+ fi
+ download_and_check=1
+ break
+ done
+
+ # We tried every URI possible, none seems to work or to check against the
+ # available hash. *ABORT MISSION*
+ if [ "${download_and_check}" -eq 0 ]; then
rm -rf "${tmpd}"
exit 1
fi
@@ -164,16 +203,13 @@ DESCRIPTION
-h This help text.
- -b BACKEND
- Wrap the specified BACKEND. Known backends are:
- bzr Bazaar
- cp Local files
- cvs Concurrent Versions System
- git Git
- hg Mercurial
- scp Secure copy
- svn Subversion
- wget HTTP download
+ -u URIs
+ The URI to get the file from, the URI must respect the format given in
+ the example.
+ You may give as many '-u URI' as you want, the script will stop at the
+ frist successful download.
+
+ Example: backend+URI; git+http://example.com or http+http://example.com
-o FILE
Store the downloaded archive in FILE.
diff --git a/support/download/cp b/support/download/file
similarity index 90%
rename from support/download/cp
rename to support/download/file
index 52fe2de83d..a3e616a181 100755
--- a/support/download/cp
+++ b/support/download/file
@@ -3,7 +3,7 @@
# We want to catch any unexpected failure, and exit immediately
set -e
-# Download helper for cp, to be called from the download wrapper script
+# Download helper for file, to be called from the download wrapper script
#
# Options:
# -q Be quiet.
@@ -23,7 +23,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=;;
o) output="${OPTARG}";;
- u) source="${OPTARG}";;
+ u) source="${OPTARG#*://}";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
diff --git a/support/download/wget b/support/download/wget
index fece6663ca..c69e6071aa 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -8,7 +8,9 @@ set -e
# Options:
# -q Be quiet.
# -o FILE Save into file FILE.
+# -f FILENAME The filename of the tarball to get at URL
# -u URL Download file at URL.
+# -e ENCODE Tell wget to urlencode the filename passed to it
#
# Environment:
# WGET : the wget command to call
@@ -18,7 +20,9 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
o) output="${OPTARG}";;
+ f) filename="${OPTARG}";;
u) url="${OPTARG}";;
+ e) encode="-e";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
\?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
esac
@@ -32,4 +36,8 @@ _wget() {
eval ${WGET} "${@}"
}
-_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
+# Replace every '?' with '%3F' in the filename; only for the PRIMARY and BACKUP
+# mirror
+[ -n "${encode}" ] && filename=${filename//\?/%3F}
+
+_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 04/13] packages: use new $($PKG)_DL_DIR) variable
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (2 preceding siblings ...)
2017-10-25 20:09 ` [Buildroot] [PATCH v2 03/13] download: put most of the infra in dl-wrapper Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Peter Seiderer
` (8 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the
transition into a new directory structure for DL_DIR.
This commit has been generated with the following scripts:
for i in package/*/*.mk; do
if ! grep -q "\$(DL_DIR)" ${i}; then
continue
fi
pkg_name="$(basename $(dirname ${i}))"
raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
pkg_dl_dir="${raw_pkg_name}_DL_DIR"
sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}
done
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/4de06540893ebdfbfb83716db2a34cfbda1f0c43.patch
---
package/amd-catalyst/amd-catalyst.mk | 2 +-
package/android-tools/android-tools.mk | 2 +-
package/angularjs/angularjs.mk | 2 +-
package/bootstrap/bootstrap.mk | 2 +-
package/cache-calibrator/cache-calibrator.mk | 2 +-
package/cracklib/cracklib.mk | 2 +-
package/cryptopp/cryptopp.mk | 2 +-
package/devmem2/devmem2.mk | 2 +-
package/dhrystone/dhrystone.mk | 2 +-
package/doom-wad/doom-wad.mk | 2 +-
package/espeak/espeak.mk | 2 +-
package/fan-ctrl/fan-ctrl.mk | 2 +-
package/gcc/gcc.mk | 2 +-
package/irrlicht/irrlicht.mk | 2 +-
package/jquery-mobile/jquery-mobile.mk | 2 +-
package/jquery-sparkline/jquery-sparkline.mk | 2 +-
package/jquery-ui-themes/jquery-ui-themes.mk | 2 +-
package/jquery-ui/jquery-ui.mk | 2 +-
package/jquery-validation/jquery-validation.mk | 2 +-
package/jquery/jquery.mk | 2 +-
package/libfreeimage/libfreeimage.mk | 2 +-
package/libjson/libjson.mk | 2 +-
package/libsoil/libsoil.mk | 2 +-
package/lsof/lsof.mk | 2 +-
| 2 +-
package/nmon/nmon.mk | 2 +-
package/nvidia-driver/nvidia-driver.mk | 2 +-
package/openobex/openobex.mk | 6 ++++++
package/opentyrian-data/opentyrian-data.mk | 2 +-
package/perl/perl.mk | 2 +-
package/python-keyring/python-keyring.mk | 2 +-
package/python-simplegeneric/python-simplegeneric.mk | 2 +-
package/rapidxml/rapidxml.mk | 2 +-
package/rpi-wifi-firmware/rpi-wifi-firmware.mk | 2 +-
package/sam-ba/sam-ba.mk | 2 +-
package/spidev_test/spidev_test.mk | 2 +-
package/tar/tar.mk | 2 +-
package/tesseract-ocr/tesseract-ocr.mk | 2 +-
package/ti-cgt-pru/ti-cgt-pru.mk | 4 ++--
package/ti-gfx/ti-gfx.mk | 4 ++--
package/ts4900-fpga/ts4900-fpga.mk | 2 +-
package/unscd/unscd.mk | 2 +-
package/urg/urg.mk | 4 +++-
package/waf/waf.mk | 2 +-
package/whetstone/whetstone.mk | 2 +-
package/wilc1000-firmware/wilc1000-firmware.mk | 2 +-
package/zynq-boot-bin/zynq-boot-bin.mk | 2 +-
47 files changed, 56 insertions(+), 48 deletions(-)
diff --git a/package/amd-catalyst/amd-catalyst.mk b/package/amd-catalyst/amd-catalyst.mk
index b9396e11fa..d864095c31 100644
--- a/package/amd-catalyst/amd-catalyst.mk
+++ b/package/amd-catalyst/amd-catalyst.mk
@@ -17,7 +17,7 @@ AMD_CATALYST_ARCH_DIR = $(@D)/arch/x86$(AMD_CATALYST_SUFFIX)
AMD_CATALYST_LIB_SUFFIX = $(if $(BR2_x86_64),64)
define AMD_CATALYST_EXTRACT_CMDS
- unzip -q $(DL_DIR)/$(AMD_CATALYST_SOURCE) -d $(@D)
+ unzip -q $(AMD_CATALYST_DL_DIR)/$(AMD_CATALYST_SOURCE) -d $(@D)
$(SHELL) $(@D)/AMD-Catalyst-$(AMD_CATALYST_VERSION)-Linux-installer-$(AMD_CATALYST_VERBOSE_VER)-x86.x86_64.run --extract $(@D)
endef
diff --git a/package/android-tools/android-tools.mk b/package/android-tools/android-tools.mk
index f6c6913a68..11e0a15a7f 100644
--- a/package/android-tools/android-tools.mk
+++ b/package/android-tools/android-tools.mk
@@ -15,7 +15,7 @@ ANDROID_TOOLS_LICENSE_FILES = debian/copyright
# Extract the Debian tarball inside the sources
define ANDROID_TOOLS_DEBIAN_EXTRACT
$(call suitable-extractor,$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS))) \
- $(DL_DIR)/$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS)) | \
+ $(ANDROID_TOOLS_DL_DIR)/$(notdir $(ANDROID_TOOLS_EXTRA_DOWNLOADS)) | \
$(TAR) -C $(@D) $(TAR_OPTIONS) -
endef
diff --git a/package/angularjs/angularjs.mk b/package/angularjs/angularjs.mk
index 08a4b60845..8d6161125b 100644
--- a/package/angularjs/angularjs.mk
+++ b/package/angularjs/angularjs.mk
@@ -12,7 +12,7 @@ ANGULARJS_LICENSE = MIT
ANGULARJS_LICENSE_FILES = angular.js
define ANGULARJS_EXTRACT_CMDS
- unzip $(DL_DIR)/$(ANGULARJS_SOURCE) -d $(@D)
+ unzip $(ANGULARJS_DL_DIR)/$(ANGULARJS_SOURCE) -d $(@D)
mv $(@D)/angular-$(ANGULARJS_VERSION)/* $(@D)
rmdir $(@D)/angular-$(ANGULARJS_VERSION)
endef
diff --git a/package/bootstrap/bootstrap.mk b/package/bootstrap/bootstrap.mk
index 33161f53ae..d9aeeabe62 100644
--- a/package/bootstrap/bootstrap.mk
+++ b/package/bootstrap/bootstrap.mk
@@ -10,7 +10,7 @@ BOOTSTRAP_SOURCE = bootstrap-$(BOOTSTRAP_VERSION)-dist.zip
BOOTSTRAP_LICENSE = MIT
define BOOTSTRAP_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
+ $(UNZIP) $(BOOTSTRAP_DL_DIR)/$(BOOTSTRAP_SOURCE) -d $(@D)
mv $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist/* $(@D)/
rmdir $(@D)/bootstrap-$(BOOTSTRAP_VERSION)-dist
endef
diff --git a/package/cache-calibrator/cache-calibrator.mk b/package/cache-calibrator/cache-calibrator.mk
index 6f12d119ef..203732e4f7 100644
--- a/package/cache-calibrator/cache-calibrator.mk
+++ b/package/cache-calibrator/cache-calibrator.mk
@@ -10,7 +10,7 @@ CACHE_CALIBRATOR_LICENSE = Cache calibrator license
CACHE_CALIBRATOR_LICENSE_FILES = calibrator.c.license
define CACHE_CALIBRATOR_EXTRACT_CMDS
- cp $(DL_DIR)/$(CACHE_CALIBRATOR_SOURCE) $(@D)
+ cp $(CACHE_CALIBRATOR_DL_DIR)/$(CACHE_CALIBRATOR_SOURCE) $(@D)
endef
define CACHE_CALIBRATOR_EXTRACT_LICENSE
diff --git a/package/cracklib/cracklib.mk b/package/cracklib/cracklib.mk
index 2fa3d3b4ea..aeee60d1e7 100644
--- a/package/cracklib/cracklib.mk
+++ b/package/cracklib/cracklib.mk
@@ -29,7 +29,7 @@ HOST_CRACKLIB_CONF_OPTS += --without-python --without-zlib
ifeq ($(BR2_PACKAGE_CRACKLIB_FULL_DICT),y)
CRACKLIB_EXTRA_DOWNLOADS = cracklib-words-$(CRACKLIB_VERSION).gz
-CRACKLIB_DICT_SOURCE = $(DL_DIR)/cracklib-words-$(CRACKLIB_VERSION).gz
+CRACKLIB_DICT_SOURCE = $(CRACKLIB_DL_DIR)/cracklib-words-$(CRACKLIB_VERSION).gz
else
CRACKLIB_DICT_SOURCE = $(@D)/dicts/cracklib-small
endif
diff --git a/package/cryptopp/cryptopp.mk b/package/cryptopp/cryptopp.mk
index 4b121aef2d..63ab600e01 100644
--- a/package/cryptopp/cryptopp.mk
+++ b/package/cryptopp/cryptopp.mk
@@ -12,7 +12,7 @@ CRYPTOPP_LICENSE_FILES = License.txt
CRYPTOPP_INSTALL_STAGING = YES
define HOST_CRYPTOPP_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(CRYPTOPP_SOURCE) -d $(@D)
+ $(UNZIP) $(CRYPTOPP_DL_DIR)/$(CRYPTOPP_SOURCE) -d $(@D)
endef
HOST_CRYPTOPP_MAKE_OPTS = \
diff --git a/package/devmem2/devmem2.mk b/package/devmem2/devmem2.mk
index eb92db6098..044e6fa596 100644
--- a/package/devmem2/devmem2.mk
+++ b/package/devmem2/devmem2.mk
@@ -11,7 +11,7 @@ DEVMEM2_LICENSE = GPL-2.0+
DEVMEM2_LICENSE_FILES = devmem2.c.license
define DEVMEM2_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(DEVMEM2_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define DEVMEM2_EXTRACT_LICENSE
diff --git a/package/dhrystone/dhrystone.mk b/package/dhrystone/dhrystone.mk
index c0bca9895d..d54c8d94e8 100644
--- a/package/dhrystone/dhrystone.mk
+++ b/package/dhrystone/dhrystone.mk
@@ -9,7 +9,7 @@ DHRYSTONE_SOURCE = dhry-c
DHRYSTONE_SITE = http://www.netlib.org/benchmark
define DHRYSTONE_EXTRACT_CMDS
- (cd $(@D) && $(SHELL) $(DL_DIR)/$($(PKG)_SOURCE))
+ (cd $(@D) && $(SHELL) $(DHRYSTONE_DL_DIR)/$($(PKG)_SOURCE))
$(Q)cp package/dhrystone/Makefile $(@D)/
endef
diff --git a/package/doom-wad/doom-wad.mk b/package/doom-wad/doom-wad.mk
index d3ac731bd5..f348fc2207 100644
--- a/package/doom-wad/doom-wad.mk
+++ b/package/doom-wad/doom-wad.mk
@@ -11,7 +11,7 @@ DOOM_WAD_SOURCE = doom$(subst .,,$(DOOM_WAD_VERSION))s.zip
DOOM_WAD_SITE = http://www.jbserver.com/downloads/games/doom/misc/shareware
define DOOM_WAD_EXTRACT_CMDS
- $(UNZIP) -p $(DL_DIR)/$($(PKG)_SOURCE) 'DOOMS_19.[12]' > \
+ $(UNZIP) -p $(DOOM_WAD_DL_DIR)/$($(PKG)_SOURCE) 'DOOMS_19.[12]' > \
$(@D)/doom-$(DOOM_WAD_VERSION).zip
$(UNZIP) -d $(@D) $(@D)/doom-$(DOOM_WAD_VERSION).zip DOOM1.WAD
endef
diff --git a/package/espeak/espeak.mk b/package/espeak/espeak.mk
index 226f17fc05..aa0b5b0716 100644
--- a/package/espeak/espeak.mk
+++ b/package/espeak/espeak.mk
@@ -21,7 +21,7 @@ ESPEAK_DEPENDENCIES = pulseaudio
endif
define ESPEAK_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(ESPEAK_SOURCE)
+ $(UNZIP) -d $(@D) $(ESPEAK_DL_DIR)/$(ESPEAK_SOURCE)
mv $(@D)/espeak-$(ESPEAK_VERSION)-source/* $(@D)
$(RM) -r $(@D)/espeak-$(ESPEAK_VERSION)-source
endef
diff --git a/package/fan-ctrl/fan-ctrl.mk b/package/fan-ctrl/fan-ctrl.mk
index 53533a8a17..81a52f1e9f 100644
--- a/package/fan-ctrl/fan-ctrl.mk
+++ b/package/fan-ctrl/fan-ctrl.mk
@@ -12,7 +12,7 @@ FAN_CTRL_LICENSE = GPL-2.0+
FAN_CTRL_LICENSE_FILES = fan-ctrl.c
define FAN_CTRL_EXTRACT_CMDS
- cp $(DL_DIR)/$(FAN_CTRL_SOURCE) $(@D)/fan-ctrl.c
+ cp $(FAN_CTRL_DL_DIR)/$(FAN_CTRL_SOURCE) $(@D)/fan-ctrl.c
endef
define FAN_CTRL_BUILD_CMDS
diff --git a/package/gcc/gcc.mk b/package/gcc/gcc.mk
index 6bdd6c88e3..82eb21a039 100644
--- a/package/gcc/gcc.mk
+++ b/package/gcc/gcc.mk
@@ -314,7 +314,7 @@ HOST_GCC_COMMON_MAKE_OPTS = \
gcc_cv_libc_provides_ssp=$(if $(BR2_TOOLCHAIN_HAS_SSP),yes,no)
ifeq ($(BR2_CCACHE),y)
-HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE)
+HOST_GCC_COMMON_CCACHE_HASH_FILES += $(GCC_DL_DIR)/$(GCC_SOURCE)
# Cfr. PATCH_BASE_DIRS in .stamp_patched, but we catch both versioned
# and unversioned patches unconditionally. Moreover, to facilitate the
diff --git a/package/irrlicht/irrlicht.mk b/package/irrlicht/irrlicht.mk
index ccd1045ca0..a1e190d98d 100644
--- a/package/irrlicht/irrlicht.mk
+++ b/package/irrlicht/irrlicht.mk
@@ -25,7 +25,7 @@ IRRLICHT_SUBDIR = source/Irrlicht
IRRLICHT_DEPENDENCIES = libgl xlib_libXxf86vm
define IRRLICHT_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(IRRLICHT_SOURCE)
+ $(UNZIP) -d $(@D) $(IRRLICHT_DL_DIR)/$(IRRLICHT_SOURCE)
mv $(@D)/irrlicht-$(IRRLICHT_VERSION)/* $(@D)
$(RM) -r $(@D)/irrlicht-$(IRRLICHT_VERSION)
endef
diff --git a/package/jquery-mobile/jquery-mobile.mk b/package/jquery-mobile/jquery-mobile.mk
index 1067952904..1171e60d0a 100644
--- a/package/jquery-mobile/jquery-mobile.mk
+++ b/package/jquery-mobile/jquery-mobile.mk
@@ -10,7 +10,7 @@ JQUERY_MOBILE_SOURCE = jquery.mobile-$(JQUERY_MOBILE_VERSION).zip
JQUERY_MOBILE_LICENSE = MIT
define JQUERY_MOBILE_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_MOBILE_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_MOBILE_DL_DIR)/$(JQUERY_MOBILE_SOURCE)
endef
JQUERY_MOBILE_INSTALLED_FILES = \
diff --git a/package/jquery-sparkline/jquery-sparkline.mk b/package/jquery-sparkline/jquery-sparkline.mk
index 165c2fa99c..683d84f4aa 100644
--- a/package/jquery-sparkline/jquery-sparkline.mk
+++ b/package/jquery-sparkline/jquery-sparkline.mk
@@ -10,7 +10,7 @@ JQUERY_SPARKLINE_SOURCE = jquery.sparkline.min.js
JQUERY_SPARKLINE_LICENSE = BSD-3-Clause
define JQUERY_SPARKLINE_EXTRACT_CMDS
- cp $(DL_DIR)/$(JQUERY_SPARKLINE_SOURCE) $(@D)
+ cp $(JQUERY_SPARKLINE_DL_DIR)/$(JQUERY_SPARKLINE_SOURCE) $(@D)
endef
define JQUERY_SPARKLINE_INSTALL_TARGET_CMDS
diff --git a/package/jquery-ui-themes/jquery-ui-themes.mk b/package/jquery-ui-themes/jquery-ui-themes.mk
index cd3dab6d87..5108389a77 100644
--- a/package/jquery-ui-themes/jquery-ui-themes.mk
+++ b/package/jquery-ui-themes/jquery-ui-themes.mk
@@ -12,7 +12,7 @@ JQUERY_UI_THEMES_LICENSE_FILES = MIT-LICENSE.txt
JQUERY_UI_THEMES_DEPENDENCIES = jquery-ui
define JQUERY_UI_THEMES_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_UI_THEMES_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_UI_THEMES_DL_DIR)/$(JQUERY_UI_THEMES_SOURCE)
mv $(@D)/jquery-ui-themes-$(JQUERY_UI_THEMES_VERSION)/* $(@D)
$(RM) -r $(@D)/jquery-ui-themes-$(JQUERY_UI_THEMES_VERSION)
endef
diff --git a/package/jquery-ui/jquery-ui.mk b/package/jquery-ui/jquery-ui.mk
index d829d6241b..9284e683f1 100644
--- a/package/jquery-ui/jquery-ui.mk
+++ b/package/jquery-ui/jquery-ui.mk
@@ -14,7 +14,7 @@ JQUERY_UI_LICENSE = MIT
JQUERY_UI_LICENSE_FILES = MIT-LICENSE.txt
define JQUERY_UI_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_UI_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_UI_DL_DIR)/$(JQUERY_UI_SOURCE)
mv $(@D)/jquery-ui-$(JQUERY_UI_VERSION)/* $(@D)
$(RM) -r $(@D)/jquery-ui-$(JQUERY_UI_VERSION)
endef
diff --git a/package/jquery-validation/jquery-validation.mk b/package/jquery-validation/jquery-validation.mk
index 10d2a92afc..a8c2897983 100644
--- a/package/jquery-validation/jquery-validation.mk
+++ b/package/jquery-validation/jquery-validation.mk
@@ -11,7 +11,7 @@ JQUERY_VALIDATION_LICENSE = MIT
JQUERY_VALIDATION_LICENSE_FILES = README.md
define JQUERY_VALIDATION_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(JQUERY_VALIDATION_SOURCE)
+ $(UNZIP) -d $(@D) $(JQUERY_VALIDATION_DL_DIR)/$(JQUERY_VALIDATION_SOURCE)
endef
define JQUERY_VALIDATION_INSTALL_TARGET_CMDS
diff --git a/package/jquery/jquery.mk b/package/jquery/jquery.mk
index 363ee09633..f75eded8b3 100644
--- a/package/jquery/jquery.mk
+++ b/package/jquery/jquery.mk
@@ -10,7 +10,7 @@ JQUERY_SOURCE = jquery-$(JQUERY_VERSION).min.js
JQUERY_LICENSE = MIT
define JQUERY_EXTRACT_CMDS
- cp $(DL_DIR)/$(JQUERY_SOURCE) $(@D)
+ cp $(JQUERY_DL_DIR)/$(JQUERY_SOURCE) $(@D)
endef
define JQUERY_INSTALL_TARGET_CMDS
diff --git a/package/libfreeimage/libfreeimage.mk b/package/libfreeimage/libfreeimage.mk
index 0ca23933a6..e0aa1f0ae8 100644
--- a/package/libfreeimage/libfreeimage.mk
+++ b/package/libfreeimage/libfreeimage.mk
@@ -12,7 +12,7 @@ LIBFREEIMAGE_LICENSE_FILES = license-gplv2.txt license-gplv3.txt license-fi.txt
LIBFREEIMAGE_INSTALL_STAGING = YES
define LIBFREEIMAGE_EXTRACT_CMDS
- $(UNZIP) $(DL_DIR)/$(LIBFREEIMAGE_SOURCE) -d $(@D)
+ $(UNZIP) $(LIBFREEIMAGE_DL_DIR)/$(LIBFREEIMAGE_SOURCE) -d $(@D)
mv $(@D)/FreeImage/* $(@D)
rmdir $(@D)/FreeImage
endef
diff --git a/package/libjson/libjson.mk b/package/libjson/libjson.mk
index 74224ba657..d04ddc40f3 100644
--- a/package/libjson/libjson.mk
+++ b/package/libjson/libjson.mk
@@ -23,7 +23,7 @@ endif
LIBJSON_MAKE_OPTS += BUILD_TYPE= CXXFLAGS="$(LIBJSON_CXXFLAGS)"
define LIBJSON_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(LIBJSON_SOURCE)
+ $(UNZIP) -d $(@D) $(LIBJSON_DL_DIR)/$(LIBJSON_SOURCE)
mv $(@D)/libjson/* $(@D)
$(RM) -r $(@D)/libjson
$(SED) '/ldconfig/d' $(@D)/makefile
diff --git a/package/libsoil/libsoil.mk b/package/libsoil/libsoil.mk
index efa67d9eee..2945edd8dd 100644
--- a/package/libsoil/libsoil.mk
+++ b/package/libsoil/libsoil.mk
@@ -14,7 +14,7 @@ LIBSOIL_LICENSE_FILES = src/stb_image_aug.c src/image_helper.c
LIBSOIL_MAKEFILE = ../projects/makefile/alternate_Makefile.txt
define LIBSOIL_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(LIBSOIL_SOURCE)
+ $(UNZIP) -d $(@D) $(LIBSOIL_DL_DIR)/$(LIBSOIL_SOURCE)
mv $(@D)/Simple\ OpenGL\ Image\ Library/* $(@D)
endef
diff --git a/package/lsof/lsof.mk b/package/lsof/lsof.mk
index e5cd4bce6e..0dc8e2de2f 100644
--- a/package/lsof/lsof.mk
+++ b/package/lsof/lsof.mk
@@ -32,7 +32,7 @@ endif
# The .tar.bz2 contains another .tar, which contains the source code.
define LSOF_EXTRACT_CMDS
- $(call suitable-extractor,$(LSOF_SOURCE)) $(DL_DIR)/$(LSOF_SOURCE) | \
+ $(call suitable-extractor,$(LSOF_SOURCE)) $(LSOF_DL_DIR)/$(LSOF_SOURCE) | \
$(TAR) -O $(TAR_OPTIONS) - lsof_$(LSOF_VERSION)/lsof_$(LSOF_VERSION)_src.tar | \
$(TAR) --strip-components=1 -C $(LSOF_DIR) $(TAR_OPTIONS) -
endef
--git a/package/musl-compat-headers/musl-compat-headers.mk b/package/musl-compat-headers/musl-compat-headers.mk
index 91f5074ef6..1cfa894879 100644
--- a/package/musl-compat-headers/musl-compat-headers.mk
+++ b/package/musl-compat-headers/musl-compat-headers.mk
@@ -20,7 +20,7 @@ MUSL_COMPAT_HEADERS_INSTALL_STAGING = YES
# Copying both headers so legal-info finds them (they are _LICENSE_FILES)
define MUSL_COMPAT_HEADERS_EXTRACT_CMDS
- $(INSTALL) -m 0644 -D $(DL_DIR)/$(notdir $(MUSL_COMPAT_HEADERS_QUEUE_H)) $(@D)/queue.h
+ $(INSTALL) -m 0644 -D $(MUSL_COMPAT_HEADERS_DL_DIR)/$(notdir $(MUSL_COMPAT_HEADERS_QUEUE_H)) $(@D)/queue.h
$(INSTALL) -m 0644 -D $(MUSL_COMPAT_HEADERS_PKGDIR)/cdefs.h $(@D)/cdefs.h
endef
diff --git a/package/nmon/nmon.mk b/package/nmon/nmon.mk
index f561d6dcc0..f7edd66a26 100644
--- a/package/nmon/nmon.mk
+++ b/package/nmon/nmon.mk
@@ -13,7 +13,7 @@ NMON_DEPENDENCIES = ncurses
NMON_CFLAGS = $(TARGET_CFLAGS) -D JFS -D GETUSER -D LARGEMEM -D DEBIAN
define NMON_EXTRACT_CMDS
- cp $(DL_DIR)/$(NMON_SOURCE) $(@D)
+ cp $(NMON_DL_DIR)/$(NMON_SOURCE) $(@D)
endef
define NMON_BUILD_CMDS
diff --git a/package/nvidia-driver/nvidia-driver.mk b/package/nvidia-driver/nvidia-driver.mk
index e56661059f..404fa881ec 100644
--- a/package/nvidia-driver/nvidia-driver.mk
+++ b/package/nvidia-driver/nvidia-driver.mk
@@ -144,7 +144,7 @@ endif # BR2_PACKAGE_NVIDIA_DRIVER_MODULE == y
# virtually everywhere, and it is fine enough to provide useful options.
# Except it can't extract into an existing (even empty) directory.
define NVIDIA_DRIVER_EXTRACT_CMDS
- $(SHELL) $(DL_DIR)/$(NVIDIA_DRIVER_SOURCE) --extract-only --target \
+ $(SHELL) $(NVIDIA_DRIVER_DL_DIR)/$(NVIDIA_DRIVER_SOURCE) --extract-only --target \
$(@D)/tmp-extract
chmod u+w -R $(@D)
mv $(@D)/tmp-extract/* $(@D)/tmp-extract/.manifest $(@D)
diff --git a/package/openobex/openobex.mk b/package/openobex/openobex.mk
index ea41f65a80..c19780110c 100644
--- a/package/openobex/openobex.mk
+++ b/package/openobex/openobex.mk
@@ -19,6 +19,12 @@ ifeq ($(BR2_PACKAGE_BLUEZ_UTILS),y)
OPENOBEX_DEPENDENCIES += bluez_utils
endif
+define OPENOBEX_EXTRACT_CMDS
+ $(UNZIP) -d $(@D) $(OPENOBEX_DL_DIR)/$(OPENOBEX_SOURCE)
+ mv $(@D)/openobex-$(OPENOBEX_VERSION)-Source/* $(@D)
+ $(RM) -r $(@D)/openobex-$(OPENOBEX_VERSION)-Source
+endef
+
ifeq ($(BR2_PACKAGE_BLUEZ5_UTILS),y)
OPENOBEX_DEPENDENCIES += bluez5_utils
endif
diff --git a/package/opentyrian-data/opentyrian-data.mk b/package/opentyrian-data/opentyrian-data.mk
index 4b879df1c3..9e38b61bfa 100644
--- a/package/opentyrian-data/opentyrian-data.mk
+++ b/package/opentyrian-data/opentyrian-data.mk
@@ -10,7 +10,7 @@ OPENTYRIAN_DATA_SOURCE = tyrian21.zip
OPENTYRIAN_DATA_LICENSE = Freeware
define OPENTYRIAN_DATA_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(OPENTYRIAN_DATA_SOURCE)
+ $(UNZIP) -d $(@D) $(OPENTYRIAN_DATA_DL_DIR)/$(OPENTYRIAN_DATA_SOURCE)
endef
define OPENTYRIAN_DATA_INSTALL_TARGET_CMDS
diff --git a/package/perl/perl.mk b/package/perl/perl.mk
index 02cfec8074..9c67bf3a87 100644
--- a/package/perl/perl.mk
+++ b/package/perl/perl.mk
@@ -24,7 +24,7 @@ PERL_EXTRA_DOWNLOADS = $(PERL_CROSS_SITE)/$(PERL_CROSS_SOURCE)
# as a separate package. Instead, it is downloaded and extracted
# together with perl
define PERL_CROSS_EXTRACT
- $(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(DL_DIR)/$(PERL_CROSS_SOURCE) | \
+ $(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(PERL_DL_DIR)/$(PERL_CROSS_SOURCE) | \
$(TAR) --strip-components=1 -C $(@D) $(TAR_OPTIONS) -
endef
PERL_POST_EXTRACT_HOOKS += PERL_CROSS_EXTRACT
diff --git a/package/python-keyring/python-keyring.mk b/package/python-keyring/python-keyring.mk
index ef8b8af261..a615973912 100644
--- a/package/python-keyring/python-keyring.mk
+++ b/package/python-keyring/python-keyring.mk
@@ -11,7 +11,7 @@ PYTHON_KEYRING_SETUP_TYPE = setuptools
PYTHON_KEYRING_LICENSE = python software foundation license
define PYTHON_KEYRING_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(PYTHON_KEYRING_SOURCE)
+ $(UNZIP) -d $(@D) $(PYTHON_KEYRING_DL_DIR)/$(PYTHON_KEYRING_SOURCE)
mv $(@D)/keyring-$(PYTHON_KEYRING_VERSION)/* $(@D)
$(RM) -r $(@D)/keyring-$(PYTHON_KEYRING_VERSION)
endef
diff --git a/package/python-simplegeneric/python-simplegeneric.mk b/package/python-simplegeneric/python-simplegeneric.mk
index cc84320428..d3bfa69a7e 100644
--- a/package/python-simplegeneric/python-simplegeneric.mk
+++ b/package/python-simplegeneric/python-simplegeneric.mk
@@ -16,7 +16,7 @@ PYTHON_SIMPLEGENERIC_LICENSE = ZPL-2.1
PYTHON_SIMPLEGENERIC_SETUP_TYPE = setuptools
define PYTHON_SIMPLEGENERIC_EXTRACT_CMDS
- unzip $(DL_DIR)/$(PYTHON_SIMPLEGENERIC_SOURCE) -d $(@D)
+ unzip $(PYTHON_SIMPLEGENERIC_DL_DIR)/$(PYTHON_SIMPLEGENERIC_SOURCE) -d $(@D)
mv $(@D)/simplegeneric-$(PYTHON_SIMPLEGENERIC_VERSION)/* $(@D)
rmdir $(@D)/simplegeneric-$(PYTHON_SIMPLEGENERIC_VERSION)
endef
diff --git a/package/rapidxml/rapidxml.mk b/package/rapidxml/rapidxml.mk
index 2bec8fe1f2..9d034d8205 100644
--- a/package/rapidxml/rapidxml.mk
+++ b/package/rapidxml/rapidxml.mk
@@ -15,7 +15,7 @@ RAPIDXML_INSTALL_TARGET = NO
RAPIDXML_INSTALL_STAGING = YES
define RAPIDXML_EXTRACT_CMDS
- $(UNZIP) -d $(@D) $(DL_DIR)/$(RAPIDXML_SOURCE)
+ $(UNZIP) -d $(@D) $(RAPIDXML_DL_DIR)/$(RAPIDXML_SOURCE)
mv $(@D)/rapidxml-$(RAPIDXML_VERSION)/* $(@D)/
rmdir $(@D)/rapidxml-$(RAPIDXML_VERSION)
endef
diff --git a/package/rpi-wifi-firmware/rpi-wifi-firmware.mk b/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
index 6c855a8e14..83473d5bb9 100644
--- a/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
+++ b/package/rpi-wifi-firmware/rpi-wifi-firmware.mk
@@ -12,7 +12,7 @@ RPI_WIFI_FIRMWARE_SITE = https://raw.githubusercontent.com/RPi-Distro/firmware-n
RPI_WIFI_FIRMWARE_LICENSE = PROPRIETARY
define RPI_WIFI_FIRMWARE_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(RPI_WIFI_FIRMWARE_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define RPI_WIFI_FIRMWARE_INSTALL_TARGET_CMDS
diff --git a/package/sam-ba/sam-ba.mk b/package/sam-ba/sam-ba.mk
index 244ede3ef1..fe107b0bb2 100644
--- a/package/sam-ba/sam-ba.mk
+++ b/package/sam-ba/sam-ba.mk
@@ -13,7 +13,7 @@ SAM_BA_LICENSE_FILES = doc/license.txt tcl_lib/boards.tcl \
applets/sam4c/libraries/libchip_sam4c/include/sam4c/sam4c32e-1.h
define HOST_SAM_BA_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(SAM_BA_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(SAM_BA_DL_DIR)/$(SAM_BA_SOURCE)
mv $(BUILD_DIR)/sam-ba_cdc_linux/* $(@D)
rmdir $(BUILD_DIR)/sam-ba_cdc_linux/
endef
diff --git a/package/spidev_test/spidev_test.mk b/package/spidev_test/spidev_test.mk
index 1d657803b4..bf8170cd2c 100644
--- a/package/spidev_test/spidev_test.mk
+++ b/package/spidev_test/spidev_test.mk
@@ -32,7 +32,7 @@ endef
SPIDEV_TEST_POST_PATCH_HOOKS += SPIDEV_ADD_LINUX_IOCTL
define SPIDEV_TEST_EXTRACT_CMDS
- cp $(DL_DIR)/$(SPIDEV_TEST_SOURCE) $(@D)/spidev_test.c
+ cp $(SPIDEV_TEST_DL_DIR)/$(SPIDEV_TEST_SOURCE) $(@D)/spidev_test.c
endef
define SPIDEV_TEST_BUILD_CMDS
diff --git a/package/tar/tar.mk b/package/tar/tar.mk
index cb2ddc0eca..ad3f5ef763 100644
--- a/package/tar/tar.mk
+++ b/package/tar/tar.mk
@@ -40,7 +40,7 @@ HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz
define HOST_TAR_EXTRACT_CMDS
mkdir -p $(@D)
cd $(@D) && \
- $(call suitable-extractor,$(HOST_TAR_SOURCE)) $(DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i --preserve-modification-time
+ $(call suitable-extractor,$(HOST_TAR_SOURCE)) $(TAR_DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i --preserve-modification-time
mv $(@D)/tar-$(TAR_VERSION)/* $(@D)
rmdir $(@D)/tar-$(TAR_VERSION)
endef
diff --git a/package/tesseract-ocr/tesseract-ocr.mk b/package/tesseract-ocr/tesseract-ocr.mk
index 9e315b239a..919a4360ed 100644
--- a/package/tesseract-ocr/tesseract-ocr.mk
+++ b/package/tesseract-ocr/tesseract-ocr.mk
@@ -62,7 +62,7 @@ TESSERACT_OCR_PRE_CONFIGURE_HOOKS += TESSERACT_OCR_PRECONFIGURE
# Language data files installation
define TESSERACT_OCR_INSTALL_LANG_DATA
$(foreach langfile,$(TESSERACT_OCR_DATA_FILES), \
- $(INSTALL) -D -m 0644 $(DL_DIR)/$(langfile) \
+ $(INSTALL) -D -m 0644 $(TESSERACT_OCR_DL_DIR)/$(langfile) \
$(TARGET_DIR)/usr/share/tessdata/$(langfile)
)
endef
diff --git a/package/ti-cgt-pru/ti-cgt-pru.mk b/package/ti-cgt-pru/ti-cgt-pru.mk
index f923e835dc..9c00801558 100644
--- a/package/ti-cgt-pru/ti-cgt-pru.mk
+++ b/package/ti-cgt-pru/ti-cgt-pru.mk
@@ -13,8 +13,8 @@ TI_CGT_PRU_LICENSE_FILES = PRU_Code_Generation_Tools_2.1.x_manifest.html \
PRU_CodeGen_Library_2.1_0222433C-30C1-442d-B5C6-2073BD97F80F.spdx.tag
define HOST_TI_CGT_PRU_EXTRACT_CMDS
- chmod +x $(DL_DIR)/$(TI_CGT_PRU_SOURCE)
- $(DL_DIR)/$(TI_CGT_PRU_SOURCE) --prefix $(@D) --mode unattended
+ chmod +x $(TI_CGT_PRU_DL_DIR)/$(TI_CGT_PRU_SOURCE)
+ $(TI_CGT_PRU_DL_DIR)/$(TI_CGT_PRU_SOURCE) --prefix $(@D) --mode unattended
mv $(@D)/ti-cgt-pru_$(TI_CGT_PRU_VERSION)/* $(@D)
rmdir $(@D)/ti-cgt-pru_$(TI_CGT_PRU_VERSION)/
endef
diff --git a/package/ti-gfx/ti-gfx.mk b/package/ti-gfx/ti-gfx.mk
index 428878a08b..9fad553780 100644
--- a/package/ti-gfx/ti-gfx.mk
+++ b/package/ti-gfx/ti-gfx.mk
@@ -94,8 +94,8 @@ TI_GFX_HDR_DIRS = OGLES2/EGL OGLES2/EWS OGLES2/GLES2 OGLES2/KHR \
OGLES/GLES bufferclass_ti/ pvr2d/ wsegl/
define TI_GFX_EXTRACT_CMDS
- chmod +x $(DL_DIR)/$(TI_GFX_SOURCE)
- printf "Y\nY\n qY\n\n" | $(DL_DIR)/$(TI_GFX_SOURCE) \
+ chmod +x $(TI_GFX_DL_DIR)/$(TI_GFX_SOURCE)
+ printf "Y\nY\n qY\n\n" | $(TI_GFX_DL_DIR)/$(TI_GFX_SOURCE) \
--prefix $(@D) \
--mode console
endef
diff --git a/package/ts4900-fpga/ts4900-fpga.mk b/package/ts4900-fpga/ts4900-fpga.mk
index ed951b8c8f..7bb62a3984 100644
--- a/package/ts4900-fpga/ts4900-fpga.mk
+++ b/package/ts4900-fpga/ts4900-fpga.mk
@@ -11,7 +11,7 @@ TS4900_FPGA_SITE = ftp://ftp.embeddedarm.com/ts-socket-macrocontrollers/ts-4900-
# https://github.com/embeddedarm/meta-ts/blob/f31860f1204b64f765a5380d3b93a2cf18234f90/recipes-extras/ts4900-fpga/ts4900-fpga.bb#L6
define TS4900_FPGA_EXTRACT_CMDS
- cp $(DL_DIR)/$(TS4900_FPGA_SOURCE) $(@D)
+ cp $(TS4900_FPGA_DL_DIR)/$(TS4900_FPGA_SOURCE) $(@D)
endef
define TS4900_FPGA_INSTALL_TARGET_CMDS
diff --git a/package/unscd/unscd.mk b/package/unscd/unscd.mk
index f0eb5d6ad6..36f2de271b 100644
--- a/package/unscd/unscd.mk
+++ b/package/unscd/unscd.mk
@@ -11,7 +11,7 @@ UNSCD_LICENSE = GPL-2.0
UNSCD_LICENSE_FILES = $(UNSCD_SOURCE)
define UNSCD_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(UNSCD_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define UNSCD_BUILD_CMDS
diff --git a/package/urg/urg.mk b/package/urg/urg.mk
index 966627fe36..1346079893 100644
--- a/package/urg/urg.mk
+++ b/package/urg/urg.mk
@@ -25,7 +25,9 @@ endif
URG_CONFIG_SCRIPTS = c_urg-config urg-config
define URG_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(URG_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(URG_DL_DIR)/$(URG_SOURCE)
+ test -d $(URG_DIR) || \
+ mv $(BUILD_DIR)/$(subst .zip,,$(URG_SOURCE)) $(URG_DIR)
endef
$(eval $(autotools-package))
diff --git a/package/waf/waf.mk b/package/waf/waf.mk
index e7ac891b39..639c26a3b0 100644
--- a/package/waf/waf.mk
+++ b/package/waf/waf.mk
@@ -9,7 +9,7 @@ WAF_SOURCE = waf-$(WAF_VERSION)
WAF_SITE = https://waf.io/
define HOST_WAF_EXTRACT_CMDS
- $(INSTALL) -D -m 0755 $(DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
+ $(INSTALL) -D -m 0755 $(WAF_DL_DIR)/waf-$(WAF_VERSION) $(@D)/waf
endef
define HOST_WAF_INSTALL_CMDS
diff --git a/package/whetstone/whetstone.mk b/package/whetstone/whetstone.mk
index d9b45638c5..3b6ec3419b 100644
--- a/package/whetstone/whetstone.mk
+++ b/package/whetstone/whetstone.mk
@@ -9,7 +9,7 @@ WHETSTONE_SOURCE = whetstone.c
WHETSTONE_SITE = http://www.netlib.org/benchmark
define WHETSTONE_EXTRACT_CMDS
- cp $(DL_DIR)/$($(PKG)_SOURCE) $(@D)/
+ cp $(WHETSTONE_DL_DIR)/$($(PKG)_SOURCE) $(@D)/
endef
define WHETSTONE_BUILD_CMDS
diff --git a/package/wilc1000-firmware/wilc1000-firmware.mk b/package/wilc1000-firmware/wilc1000-firmware.mk
index 5dcf19045b..6f504d67a0 100644
--- a/package/wilc1000-firmware/wilc1000-firmware.mk
+++ b/package/wilc1000-firmware/wilc1000-firmware.mk
@@ -11,7 +11,7 @@ WILC1000_FIRMWARE_SOURCE = v$(WILC1000_FIRMWARE_VERSION)_Firmware.zip
WILC1000_FIRMWARE_LICENSE = PROPRIETARY
define WILC1000_FIRMWARE_EXTRACT_CMDS
- $(UNZIP) -d $(BUILD_DIR) $(DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
+ $(UNZIP) -d $(BUILD_DIR) $(WILC1000_FIRMWARE_DL_DIR)/$(WILC1000_FIRMWARE_SOURCE)
mv $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware/* $(@D)
rmdir $(BUILD_DIR)/wireless-firmware-$(WILC1000_FIRMWARE_VERSION)_Firmware
endef
diff --git a/package/zynq-boot-bin/zynq-boot-bin.mk b/package/zynq-boot-bin/zynq-boot-bin.mk
index 3648c63654..c523e04f45 100644
--- a/package/zynq-boot-bin/zynq-boot-bin.mk
+++ b/package/zynq-boot-bin/zynq-boot-bin.mk
@@ -12,7 +12,7 @@ ZYNQ_BOOT_BIN_LICENSE = GPL-3.0+
HOST_ZYNQ_BOOT_BIN_DEPENDENCIES = host-python
define HOST_ZYNQ_BOOT_BIN_EXTRACT_CMDS
- cp $(DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
+ cp $(ZYNQ_BOOT_BIN_DL_DIR)/$(ZYNQ_BOOT_BIN_SOURCE) $(@D)
endef
define HOST_ZYNQ_BOOT_BIN_INSTALL_CMDS
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR)
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (3 preceding siblings ...)
2017-10-25 20:09 ` [Buildroot] [PATCH v2 04/13] packages: use new $($PKG)_DL_DIR) variable Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 06/13] support/download: make sure the download folder is created Peter Seiderer
` (7 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Let the infrastructure use the new variable $(PKG)_DL_DIR
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/aa1e182e779f17715c391dfd0868ec7096c600b8.patch
---
package/pkg-download.mk | 2 +-
package/pkg-generic.mk | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 233a9fafc7..b94a828579 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -95,7 +95,7 @@ define DOWNLOAD
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
-n $($(PKG)_RAW_BASE_NAME) \
-N $($(PKG)_RAWNAME) \
- -o $(DL_DIR)/$(notdir $(1)) \
+ -o $($(PKG)_DL_DIR)/$(notdir $(1)) \
$(if $($(PKG)_GIT_SUBMODULES),-r) \
$(DOWNLOAD_URIS) \
$(QUIET) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5989417902..676625e0a6 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -589,7 +589,7 @@ $(2)_TARGET_DIRCLEAN = $$($(2)_DIR)/.stamp_dircleaned
# default extract command
$(2)_EXTRACT_CMDS ?= \
- $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
+ $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$($(2)_DL_DIR)/$$($(2)_SOURCE) | \
$$(TAR) --strip-components=$$($(2)_STRIP_COMPONENTS) \
-C $$($(2)_DIR) \
$$(foreach x,$$($(2)_EXCLUDES),--exclude='$$(x)' ) \
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 06/13] support/download: make sure the download folder is created
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (4 preceding siblings ...)
2017-10-25 20:09 ` [Buildroot] [PATCH v2 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 07/13] pkg-generic: add a subdirectory to the DL_DIR Peter Seiderer
` (6 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
At the moment, it means that we make sure that BR2_DL_DIR is created, in
the future, it will make sure that BR2_DL_DIR/PKG_NAME/ is created.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/e19dd1a941063dc1429a5f5ea1b84f222e608f5d.patch
---
support/download/dl-wrapper | 1 +
1 file changed, 1 insertion(+)
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 67e9742767..49caf3717b 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -50,6 +50,7 @@ main() {
if [ -z "${output}" ]; then
error "no output specified, use -o\n"
fi
+ mkdir -p "$(dirname "${output}")"
# If the output file already exists and:
# - there's no .hash file: do not download it again and exit promptly
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 07/13] pkg-generic: add a subdirectory to the DL_DIR
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (5 preceding siblings ...)
2017-10-25 20:09 ` [Buildroot] [PATCH v2 06/13] support/download: make sure the download folder is created Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 08/13] pkg-download: support new subdir for mirrors Peter Seiderer
` (5 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
With all the previous changes, we are now ready to add a subdirectory to
the DL_DIR.
The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2}
This is needed for multiple reasons:
- Avoid patches with name like SHA1.patch laying flat in DL_DIR,
which makes it hard to know to which packages they apply
- Avoid possible collisions if two releases have the same name
(e.g: v01.tar)
- Allow the possibility to handle a git cache per package in the
newly created subdirectory.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/a53df6631be6d9fa43d8fd7fdb2f8b2903459334.patch
---
package/pkg-generic.mk | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 676625e0a6..d07ea0f885 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -441,7 +441,7 @@ endif
$(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR = $$(DL_DIR)
+$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
ifndef $(2)_SUBDIR
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 08/13] pkg-download: support new subdir for mirrors
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (6 preceding siblings ...)
2017-10-25 20:09 ` [Buildroot] [PATCH v2 07/13] pkg-generic: add a subdirectory to the DL_DIR Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 09/13] pkg-generic: introduce _SAME_SOURCE_AS Peter Seiderer
` (4 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Since we introduced subdirectory to the DL_DIR, we need to support them
in the PRIMARY and BACKUP mirror as they evolve to the new tree
structure.
We check first the new URI (with the subdir), and in case of failure, we
go check without.
By checking both URIs, we ensure that old mirror are usable.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/a7d5ad4a34322e7f77849e2d5ef38ebc23e42149.patch
- use urlencode
---
package/pkg-download.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index b94a828579..b76d0ec289 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -75,6 +75,7 @@ export BR_NO_CHECK_HASH_FOR =
ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE)/$(notdir $($(PKG)_DL_DIR)),urlencode) \
-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE),urlencode)
endif
@@ -83,6 +84,7 @@ DOWNLOAD_URIS += \
-u $($(PKG)_SITE_METHOD)+$(dir $(1))
ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
DOWNLOAD_URIS += \
+ -u $(call getschemeplusuri,$(BR2_BACKUP_SITE)/$(notdir $($(PKG)_DL_DIR)),urlencode) \
-u $(call getschemeplusuri,$(BR2_BACKUP_SITE),urlencode)
endif
endif
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 09/13] pkg-generic: introduce _SAME_SOURCE_AS
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (7 preceding siblings ...)
2017-10-25 20:09 ` [Buildroot] [PATCH v2 08/13] pkg-download: support new subdir for mirrors Peter Seiderer
@ 2017-10-25 20:09 ` Peter Seiderer
2017-10-25 20:10 ` [Buildroot] [PATCH v2 10/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Peter Seiderer
` (3 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:09 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
This per package variable can be used to specify that a package shares
the same sources as another package.
The use case here is for example, you have linux-headers and linux, you
don't want to download twice the kernel (because of the introduction of
the subdirs in DL_DIR).
So you want to specify in linux-headers, this variable, so it will use
the linux's source.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/8a24e162fea241cf36a2727c8e53da91e64c677c.patch
---
package/pkg-generic.mk | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index d07ea0f885..99494929e6 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -441,7 +441,9 @@ endif
$(2)_BASE_NAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
$(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME)
+$(2)_DL_DIR = $$(if $$($(2)_SAME_SOURCE_AS), \
+ $$(DL_DIR)/$$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_RAWNAME), \
+ $$(DL_DIR)/$$($(2)_RAWNAME))
$(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASE_NAME)
ifndef $(2)_SUBDIR
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 10/13] help/manual: update help about the new $(LIBFOO_DL_DIR)
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (8 preceding siblings ...)
2017-10-25 20:09 ` [Buildroot] [PATCH v2 09/13] pkg-generic: introduce _SAME_SOURCE_AS Peter Seiderer
@ 2017-10-25 20:10 ` Peter Seiderer
2017-10-25 20:10 ` [Buildroot] [PATCH v2 11/13] download: add flock call before dl-wrapper Peter Seiderer
` (2 subsequent siblings)
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:10 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/dd1b183dea6fe2388dff13964b9053edac151dca.patch
---
Config.in | 3 +++
docs/manual/adding-packages-generic.txt | 6 +++---
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/Config.in b/Config.in
index 8920b727ce..8bb8b5fda1 100644
--- a/Config.in
+++ b/Config.in
@@ -200,6 +200,9 @@ config BR2_DL_DIR
If the Linux shell environment has defined the BR2_DL_DIR
environment variable, then this overrides this configuration
item.
+ The directory is organized with a subdirectory for each package.
+ Each package has its own $(LIBFOO_DL_DIR) variable that can be used
+ to find the correct path.
The default is $(TOPDIR)/dl
diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 63ea51bf89..9bdedeeadb 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -263,7 +263,7 @@ information is (assuming the package name is +libfoo+) :
the file using this URL. Otherwise, Buildroot will assume the file
to be downloaded is located at +LIBFOO_SITE+. Buildroot will not do
anything with those additional files, except download them: it will
- be up to the package recipe to use them from +$(DL_DIR)+.
+ be up to the package recipe to use them from +$(LIBFOO_DL_DIR)+.
* +LIBFOO_SITE_METHOD+ determines the method used to fetch or copy the
package source code. In many cases, Buildroot guesses the method
@@ -540,8 +540,8 @@ In the action definitions, you can use the following variables:
* +$(@D)+, which contains the directory in which the package source
code has been uncompressed.
-* +$(DL_DIR)+ contains the path to the directory where all the downloads made
- by Buildroot are stored.
+* +$(LIBFOO_DL_DIR)+ contains the path to the directory where all the downloads
+ made by Buildroot are stored.
* +$(TARGET_CC)+, +$(TARGET_LD)+, etc. to get the target
cross-compilation utilities
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 11/13] download: add flock call before dl-wrapper
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (9 preceding siblings ...)
2017-10-25 20:10 ` [Buildroot] [PATCH v2 10/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Peter Seiderer
@ 2017-10-25 20:10 ` Peter Seiderer
2017-10-25 20:10 ` [Buildroot] [PATCH v2 12/13] download: git: introduce cache feature Peter Seiderer
2017-10-25 20:10 ` [Buildroot] [PATCH v2 13/13] WIP: support package with '-' Peter Seiderer
12 siblings, 0 replies; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:10 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
In order to introduce the cache mechanisms, we need to have a lock on
the $(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel
download (a shared DL_DIR for two buildroot instances).
To make sure the directory exists, the mkdir call has been removed from
the dl-wrapper and put in the infrastructure.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/2f4cd2529b0681eb673b3d1a5cb7801cd55122a9.patch
---
package/pkg-download.mk | 4 +++-
support/download/dl-wrapper | 1 -
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index b76d0ec289..307c415f64 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -19,6 +19,7 @@ SSH := $(call qstrip,$(BR2_SSH))
export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
DL_WRAPPER = support/download/dl-wrapper
+FLOCK = flock $($(PKG)_DL_DIR)/
# DL_DIR may have been set already from the environment
ifeq ($(origin DL_DIR),undefined)
@@ -91,7 +92,8 @@ endif
define DOWNLOAD
$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
- $(EXTRA_ENV) $(DL_WRAPPER) \
+ $(Q)mkdir -p $($(PKG)_DL_DIR)/
+ $(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
-c $($(PKG)_DL_VERSION) \
-f $(notdir $(1)) \
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 49caf3717b..67e9742767 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -50,7 +50,6 @@ main() {
if [ -z "${output}" ]; then
error "no output specified, use -o\n"
fi
- mkdir -p "$(dirname "${output}")"
# If the output file already exists and:
# - there's no .hash file: do not download it again and exit promptly
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 12/13] download: git: introduce cache feature
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (10 preceding siblings ...)
2017-10-25 20:10 ` [Buildroot] [PATCH v2 11/13] download: add flock call before dl-wrapper Peter Seiderer
@ 2017-10-25 20:10 ` Peter Seiderer
2018-02-05 13:46 ` Thomas Petazzoni
2017-10-25 20:10 ` [Buildroot] [PATCH v2 13/13] WIP: support package with '-' Peter Seiderer
12 siblings, 1 reply; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:10 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Now we keep the git clone that we download and generates our tarball
from there.
The main goal here is that if you change the version of a package (say
Linux), instead of cloning all over again, you will simply 'git fetch'
from the repo the missing objects, then generates the tarball again.
This should speed the 'source' part of the build significantly.
The drawback is that the DL_DIR will grow much larger; but time is more
important than disk space nowadays.
Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/ce0c2175f25ee870b40a0f374f7fc99f50bcd3ae.patch
- fix links in created tar packages (reported by Peter Seiderer)
---
support/download/git | 69 ++++++++++++++++++++++++++++++----------------------
1 file changed, 40 insertions(+), 29 deletions(-)
diff --git a/support/download/git b/support/download/git
index 58a2c6ad9d..5d09cc0d7a 100755
--- a/support/download/git
+++ b/support/download/git
@@ -39,28 +39,34 @@ _git() {
eval ${GIT} "${@}"
}
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
-#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
-git_done=0
-if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
- printf "Doing shallow clone\n"
- if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
- git_done=1
- else
- printf "Shallow clone failed, falling back to doing a full clone\n"
+# We want to check if a cache of the git clone of this repo already exists.
+git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
+
+# If the cache directory already exists, don't try to clone.
+if [ ! -d "${git_cache}" ]; then
+ # Try a shallow clone, since it is faster than a full clone - but that
+ # only works if the versionis a ref (tag or branch). Before trying to do a
+ # shallow clone we check if ${cset} is in the list provided by git
+ # ls-remote. If not we fall back on a full clone.
+ #
+ # Messages for the type of clone used are provided to ease debugging in
+ # case of problems
+ git_done=0
+ if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
+ printf "Doing shallow clone\n"
+ if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then
+ git_done=1
+ else
+ printf "Shallow clone failed, falling back to doing a full clone\n"
+ fi
+ fi
+ if [ ${git_done} -eq 0 ]; then
+ printf "Doing full clone\n"
+ _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'"
fi
-fi
-if [ ${git_done} -eq 0 ]; then
- printf "Doing full clone\n"
- _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
fi
-pushd "${basename}" >/dev/null
+pushd "${git_cache}" >/dev/null
# Try to get the special refs exposed by some forges (pull-requests for
# github, changes for gerrit...). There is no easy way to know whether
@@ -86,20 +92,25 @@ if [ ${recurse} -eq 1 ]; then
_git submodule update --init --recursive
fi
-# We do not want the .git dir; we keep other .git files, in case they
-# are the only files in their directory.
+# Generate the archive, sort with the C locale so that it is reproducible
+# We do not want the .git dir; we keep other .git
+# files, in case they are the only files in their directory.
# The .git dir would generate non reproducible tarballs as it depends on
# the state of the remote server. It also would generate large tarballs
# (gigabytes for some linux trees) when a full clone took place.
-rm -rf .git
+find . -not -type d \
+ -and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list"
+LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted"
-popd >/dev/null
-
-# Generate the archive, sort with the C locale so that it is reproducible
-find "${basename}" -not -type d >"${basename}.list"
-LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
# Create GNU-format tarballs, since that's the format of the tarballs on
# sources.buildroot.org and used in the *.hash files
-tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
- -T "${basename}.list.sorted" >"${output}.tar"
+tar cf - --transform="s/^\.$/${basename}/" \
+ --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
+ -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar"
gzip -6 -n <"${output}.tar" >"${output}"
+tar tf "${output}"
+
+rm -f "${BR2_DL_DIR}/${basename}.list"
+rm -f "${BR2_DL_DIR}/${basename}.list.sorted"
+
+popd >/dev/null
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 13/13] WIP: support package with '-'
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
` (11 preceding siblings ...)
2017-10-25 20:10 ` [Buildroot] [PATCH v2 12/13] download: git: introduce cache feature Peter Seiderer
@ 2017-10-25 20:10 ` Peter Seiderer
2018-02-05 13:33 ` Thomas Petazzoni
12 siblings, 1 reply; 24+ messages in thread
From: Peter Seiderer @ 2017-10-25 20:10 UTC (permalink / raw)
To: buildroot
From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
Changes v1 --> v2:
- from https://github.com/maximeh/buildroot/commit/39fb06a2b4d7662474520db849a56b9977650487.patch
- new WIP patch
---
package/pkg-download.mk | 2 +-
support/download/bzr | 3 ++-
support/download/cvs | 3 ++-
support/download/dl-wrapper | 15 ++++++++-------
support/download/file | 3 ++-
support/download/git | 7 ++++---
support/download/hg | 3 ++-
support/download/scp | 3 ++-
support/download/svn | 3 ++-
support/download/wget | 3 ++-
10 files changed, 27 insertions(+), 18 deletions(-)
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 307c415f64..3973778181 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -99,7 +99,7 @@ define DOWNLOAD
-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
-n $($(PKG)_RAW_BASE_NAME) \
-N $($(PKG)_RAWNAME) \
- -o $($(PKG)_DL_DIR)/$(notdir $(1)) \
+ -o $($(PKG)_DL_DIR) \
$(if $($(PKG)_GIT_SUBMODULES),-r) \
$(DOWNLOAD_URIS) \
$(QUIET) \
diff --git a/support/download/bzr b/support/download/bzr
index 5289a421cd..36f19cc0bb 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -20,6 +20,7 @@ verbose=
while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ f) filename="${OPTARG}";;
o) output="${OPTARG}";;
u) uri="${OPTARG}";;
c) cset="${OPTARG}";;
@@ -55,4 +56,4 @@ fi
_bzr export ${verbose} --root="'${basename}/'" --format=tgz \
${timestamp_opt} - "${@}" "'${uri}'" -r "'${cset}'" \
- >"${output}"
+ >"${output}/${filename}"
diff --git a/support/download/cvs b/support/download/cvs
index 3f77b849e4..959583dcc0 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -21,6 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-Q;;
o) output="${OPTARG}";;
+ f) filename="${OPTARG}";;
u) uri="${OPTARG#*://}";;
c) rev="${OPTARG}";;
N) rawname="${OPTARG}";;
@@ -56,4 +57,4 @@ export TZ=UTC
_cvs ${verbose} -z3 -d"'${uri}'" \
co "${@}" -d "'${basename}'" ${select} "'${rev}'" -P "'${rawname}'"
-tar czf "${output}" "${basename}"
+tar czf "${output}/${filename}" "${basename}"
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 67e9742767..c022d356fa 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,7 +19,7 @@
# We want to catch any unexpected failure, and exit immediately.
set -e
-export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:qf:e"
+export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:qf:eR:"
main() {
local OPT OPTARG
@@ -56,8 +56,8 @@ main() {
# - matches all its hashes: do not download it again and exit promptly
# - fails at least one of its hashes: force a re-download
# - there's no hash (but a .hash file): consider it a hard error
- if [ -e "${output}" ]; then
- if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then
+ if [ -e "${output}/${filename}" ]; then
+ if support/download/check-hash ${quiet} "${hfile}" "${output}" "${filename}"; then
exit 0
elif [ ${?} -ne 2 ]; then
# Do not remove the file, otherwise it might get re-downloaded
@@ -109,9 +109,10 @@ main() {
-c "${cset}" \
-n "${raw_base_name}" \
-N "${raw_name}" \
- -f "${filename}" \
+ -f "$(basename ${tmpf})" \
+ -R "${filename}" \
-u "${uri}" \
- -o "${tmpf}" \
+ -o "${tmpd}" \
${quiet} ${recurse} "${@}"
then
rm -rf "${tmpd:?}/*"
@@ -125,7 +126,7 @@ main() {
# Check if the downloaded file is sane, and matches the stored hashes
# for that file
- if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
+ if ! support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${filename}"; then
rm -rf "${tmpd:?}/*"
# cd back to keep path coherence
cd "${OLDPWD}"
@@ -181,7 +182,7 @@ main() {
# that 'mv' is atomic, because it then uses rename() that POSIX mandates
# to be atomic, see:
# http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html
- if ! mv -f "${tmp_output}" "${output}"; then
+ if ! mv -f "${tmp_output}" "${output}/${filename}"; then
rm -f "${tmp_output}"
exit 1
fi
diff --git a/support/download/file b/support/download/file
index a3e616a181..5d77d2b66c 100755
--- a/support/download/file
+++ b/support/download/file
@@ -22,6 +22,7 @@ verbose=-v
while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=;;
+ f) filename="${OPTARG}";;
o) output="${OPTARG}";;
u) source="${OPTARG#*://}";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
@@ -37,4 +38,4 @@ _localfiles() {
eval ${LOCALFILES} "${@}"
}
-_localfiles ${verbose} "${@}""'${source}'" "'${output}'"
+_localfiles ${verbose} "${@}""'${source}'" "'${output}/${filename}'"
diff --git a/support/download/git b/support/download/git
index 5d09cc0d7a..813784970b 100755
--- a/support/download/git
+++ b/support/download/git
@@ -22,6 +22,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q; exec >/dev/null;;
r) recurse=1;;
+ f) filename="${OPTARG}";;
o) output="${OPTARG}";;
u) uri="${OPTARG}";;
c) cset="${OPTARG}";;
@@ -106,9 +107,9 @@ LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list
# sources.buildroot.org and used in the *.hash files
tar cf - --transform="s/^\.$/${basename}/" \
--numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
- -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar"
-gzip -6 -n <"${output}.tar" >"${output}"
-tar tf "${output}"
+ -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}/${filename}.tar"
+gzip -n <"${output}/${filename}.tar" >"${output}/${filename}"
+tar tf "${output}/${filename}"
rm -f "${BR2_DL_DIR}/${basename}.list"
rm -f "${BR2_DL_DIR}/${basename}.list.sorted"
diff --git a/support/download/hg b/support/download/hg
index efb515fca5..586d62f3be 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -19,6 +19,7 @@ verbose=
while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ f) filename="${OPTARG}";;
o) output="${OPTARG}";;
u) uri="${OPTARG}";;
c) cset="${OPTARG}";;
@@ -40,4 +41,4 @@ _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'"
_hg archive ${verbose} --repository "'${basename}'" --type tgz \
--prefix "'${basename}'" --rev "'${cset}'" \
- - >"${output}"
+ - >"${output}/${filename}"
diff --git a/support/download/scp b/support/download/scp
index 8ecf2f4b22..92276dfef2 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -17,6 +17,7 @@ verbose=
while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ f) filename="${OPTARG}";;
o) output="${OPTARG}";;
u) uri="${OPTARG}";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
@@ -32,4 +33,4 @@ _scp() {
eval ${SCP} "${@}"
}
-_scp ${verbose} "${@}" "'${uri}'" "'${output}'"
+_scp ${verbose} "${@}" "'${uri}'" "'${output}/${filename}'"
diff --git a/support/download/svn b/support/download/svn
index 542b25c0a2..a326d0523e 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -19,6 +19,7 @@ verbose=
while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
case "${OPT}" in
q) verbose=-q;;
+ f) filename="${OPTARG}";;
o) output="${OPTARG}";;
u) uri="${OPTARG}";;
c) rev="${OPTARG}";;
@@ -38,4 +39,4 @@ _svn() {
_svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'"
-tar czf "${output}" "${basename}"
+tar czf "${output}/${filename}" "${basename}"
diff --git a/support/download/wget b/support/download/wget
index c69e6071aa..97aa43de73 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -21,6 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
q) verbose=-q;;
o) output="${OPTARG}";;
f) filename="${OPTARG}";;
+ R) remote_filename="${OPTARG}";;
u) url="${OPTARG}";;
e) encode="-e";;
:) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
@@ -40,4 +41,4 @@ _wget() {
# mirror
[ -n "${encode}" ] && filename=${filename//\?/%3F}
-_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
+_wget ${verbose} "${@}" -O "'${output}/${filename}'" "'${url}/${remote_filename}'"
--
2.14.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check
2017-10-25 20:09 ` [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check Peter Seiderer
@ 2018-01-08 20:22 ` Yann E. MORIN
2018-01-08 20:28 ` Thomas Petazzoni
1 sibling, 0 replies; 24+ messages in thread
From: Yann E. MORIN @ 2018-01-08 20:22 UTC (permalink / raw)
To: buildroot
Peter, Maxime, All,
On 2017-10-25 22:09 +0200, Peter Seiderer spake thusly:
> From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>
> This feature is not used by anyone in the core developpers and makes a
> drastic simplification of the pkg-download infrastructure harder.
>
> The future patch will move much of what's in the current pkg-download.mk
> file into the dl-wrapper which is a shell script.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Regards,
Yann E. MORIN.
> ---
> Changes v1 --> v2:
> - from https://github.com/maximeh/buildroot/commit/a4d0e83729a1ab534ff99d6f258058a1be1ed949.patch
> - remove more references to source-check (suggested by Thomas Petazzoni)
> ---
> Makefile | 7 +------
> package/pkg-download.mk | 44 --------------------------------------------
> package/pkg-generic.mk | 14 +-------------
> 3 files changed, 2 insertions(+), 63 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 79db7fe48a..45047f7e9a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -141,7 +141,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf
> # We're building in two situations: when MAKECMDGOALS is empty
> # (default target is to build), or when MAKECMDGOALS contains
> # something else than one of the nobuild_targets.
> -nobuild_targets := source %-source source-check \
> +nobuild_targets := source %-source \
> legal-info %-legal-info external-deps _external-deps \
> clean distclean help show-targets graph-depends \
> %-graph-depends %-show-depends %-show-version \
> @@ -754,10 +754,6 @@ _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps)
> external-deps:
> @$(MAKE1) -Bs $(EXTRAMAKEARGS) _external-deps | sort -u
>
> -# check if download URLs are outdated
> -.PHONY: source-check
> -source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check)
> -
> .PHONY: legal-info-clean
> legal-info-clean:
> @rm -fr $(LEGAL_INFO_DIR)
> @@ -1044,7 +1040,6 @@ help:
> @echo
> @echo 'Miscellaneous:'
> @echo ' source - download all sources needed for offline-build'
> - @echo ' source-check - check selected packages for valid download URLs'
> @echo ' external-deps - list external packages used'
> @echo ' legal-info - generate info about license compliance'
> @echo ' printvars - dump all the internal variables'
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index dc4ff1c8c7..2b845e77a2 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -68,9 +68,6 @@ export BR_NO_CHECK_HASH_FOR =
> # ssh authentication. DOWNLOAD_WGET is the normal wget-based download
> # mechanism.
> #
> -# The SOURCE_CHECK_* helpers are in charge of simply checking that the source
> -# is available for download. This can be used to make sure one will be able
> -# to get all the sources needed for one's build configuration.
> ################################################################################
>
> define DOWNLOAD_GIT
> @@ -86,12 +83,6 @@ define DOWNLOAD_GIT
> $($(PKG)_DL_OPTS)
> endef
>
> -# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
> -# repository
> -define SOURCE_CHECK_GIT
> - $(GIT) ls-remote --heads $($(PKG)_SITE) > /dev/null
> -endef
> -
> define DOWNLOAD_BZR
> $(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> @@ -103,10 +94,6 @@ define DOWNLOAD_BZR
> $($(PKG)_DL_OPTS)
> endef
>
> -define SOURCE_CHECK_BZR
> - $(BZR) ls --quiet $($(PKG)_SITE) > /dev/null
> -endef
> -
> define DOWNLOAD_CVS
> $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> @@ -119,11 +106,6 @@ define DOWNLOAD_CVS
> $($(PKG)_DL_OPTS)
> endef
>
> -# Not all CVS servers support ls/rls, use login to see if we can connect
> -define SOURCE_CHECK_CVS
> - $(CVS) -d:pserver:anonymous:@$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) login
> -endef
> -
> define DOWNLOAD_SVN
> $(EXTRA_ENV) $(DL_WRAPPER) -b svn \
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> @@ -135,10 +117,6 @@ define DOWNLOAD_SVN
> $($(PKG)_DL_OPTS)
> endef
>
> -define SOURCE_CHECK_SVN
> - $(SVN) ls $($(PKG)_SITE)@$($(PKG)_DL_VERSION) > /dev/null
> -endef
> -
> # SCP URIs should be of the form scp://[user@]host:filepath
> # Note that filepath is relative to the user's home directory, so you may want
> # to prepend the path with a slash: scp://[user@]host:/absolutepath
> @@ -152,10 +130,6 @@ define DOWNLOAD_SCP
> $($(PKG)_DL_OPTS)
> endef
>
> -define SOURCE_CHECK_SCP
> - $(SSH) $(call domain,$(1),:) ls '$(call notdomain,$(1),:)' > /dev/null
> -endef
> -
> define DOWNLOAD_HG
> $(EXTRA_ENV) $(DL_WRAPPER) -b hg \
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> @@ -167,12 +141,6 @@ define DOWNLOAD_HG
> $($(PKG)_DL_OPTS)
> endef
>
> -# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
> -# repository
> -define SOURCE_CHECK_HG
> - $(HG) incoming --force -l1 $($(PKG)_SITE) > /dev/null
> -endef
> -
> define DOWNLOAD_WGET
> $(EXTRA_ENV) $(DL_WRAPPER) -b wget \
> -o $(DL_DIR)/$(2) \
> @@ -183,10 +151,6 @@ define DOWNLOAD_WGET
> $($(PKG)_DL_OPTS)
> endef
>
> -define SOURCE_CHECK_WGET
> - $(WGET) --spider '$(call qstrip,$(1))'
> -endef
> -
> define DOWNLOAD_LOCALFILES
> $(EXTRA_ENV) $(DL_WRAPPER) -b cp \
> -o $(DL_DIR)/$(2) \
> @@ -197,10 +161,6 @@ define DOWNLOAD_LOCALFILES
> $($(PKG)_DL_OPTS)
> endef
>
> -define SOURCE_CHECK_LOCALFILES
> - test -e $(call stripurischeme,$(call qstrip,$(1)))
> -endef
> -
> ################################################################################
> # DOWNLOAD -- Download helper. Will try to download source from:
> # 1) BR2_PRIMARY_SITE if enabled
> @@ -221,10 +181,6 @@ define DOWNLOAD
> $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
> endef
>
> -define SOURCE_CHECK
> - $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),SOURCE_CHECK)
> -endef
> -
> define DOWNLOAD_INNER
> $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
> if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index cca94ba338..5989417902 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -706,10 +706,6 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE)
> endif # actual sources != sources
> endif # actual sources != ""
>
> -$(1)-source-check: PKG=$(2)
> -$(1)-source-check:
> - $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep))
> -
> $(1)-external-deps:
> @for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \
> echo `basename $$$$p` ; \
> @@ -734,9 +730,6 @@ $(1)-rsync: $$($(2)_TARGET_RSYNC)
> $(1)-source:
> $(1)-legal-source:
>
> -$(1)-source-check:
> - test -d $$($(2)_OVERRIDE_SRCDIR)
> -
> $(1)-external-deps:
> @echo "file://$$($(2)_OVERRIDE_SRCDIR)"
> endif
> @@ -762,9 +755,6 @@ $(1)-graph-rdepends: graph-depends-requirements
> $(1)-all-source: $(1)-source
> $(1)-all-source: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source)
>
> -$(1)-all-source-check: $(1)-source-check
> -$(1)-all-source-check: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check)
> -
> $(1)-all-external-deps: $(1)-external-deps
> $(1)-all-external-deps: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps)
>
> @@ -964,7 +954,6 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
> $(1)-all-external-deps \
> $(1)-all-legal-info \
> $(1)-all-source \
> - $(1)-all-source-check \
> $(1)-build \
> $(1)-clean-for-rebuild \
> $(1)-clean-for-reconfigure \
> @@ -989,8 +978,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
> $(1)-rsync \
> $(1)-show-depends \
> $(1)-show-version \
> - $(1)-source \
> - $(1)-source-check
> + $(1)-source
>
> ifneq ($$($(2)_SOURCE),)
> ifeq ($$($(2)_SITE),)
> --
> 2.14.2
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check
2017-10-25 20:09 ` [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check Peter Seiderer
2018-01-08 20:22 ` Yann E. MORIN
@ 2018-01-08 20:28 ` Thomas Petazzoni
2018-01-10 12:04 ` Thomas De Schampheleire
1 sibling, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2018-01-08 20:28 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 25 Oct 2017 22:09:51 +0200, Peter Seiderer wrote:
> From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>
> This feature is not used by anyone in the core developpers and makes a
> drastic simplification of the pkg-download infrastructure harder.
>
> The future patch will move much of what's in the current pkg-download.mk
> file into the dl-wrapper which is a shell script.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> Changes v1 --> v2:
> - from https://github.com/maximeh/buildroot/commit/a4d0e83729a1ab534ff99d6f258058a1be1ed949.patch
> - remove more references to source-check (suggested by Thomas Petazzoni)
> ---
> Makefile | 7 +------
> package/pkg-download.mk | 44 --------------------------------------------
> package/pkg-generic.mk | 14 +-------------
> 3 files changed, 2 insertions(+), 63 deletions(-)
Applied to master, thanks.
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check
2018-01-08 20:28 ` Thomas Petazzoni
@ 2018-01-10 12:04 ` Thomas De Schampheleire
2018-01-10 12:41 ` Thomas Petazzoni
2018-01-10 18:18 ` Yann E. MORIN
0 siblings, 2 replies; 24+ messages in thread
From: Thomas De Schampheleire @ 2018-01-10 12:04 UTC (permalink / raw)
To: buildroot
Hi all,
2018-01-08 21:28 GMT+01:00 Thomas Petazzoni
<thomas.petazzoni@free-electrons.com>:
> Hello,
>
> On Wed, 25 Oct 2017 22:09:51 +0200, Peter Seiderer wrote:
>> From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>>
>> This feature is not used by anyone in the core developpers and makes a
>> drastic simplification of the pkg-download infrastructure harder.
>>
>> The future patch will move much of what's in the current pkg-download.mk
>> file into the dl-wrapper which is a shell script.
>>
>> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>> ---
>> Changes v1 --> v2:
>> - from https://github.com/maximeh/buildroot/commit/a4d0e83729a1ab534ff99d6f258058a1be1ed949.patch
>> - remove more references to source-check (suggested by Thomas Petazzoni)
>> ---
>> Makefile | 7 +------
>> package/pkg-download.mk | 44 --------------------------------------------
>> package/pkg-generic.mk | 14 +-------------
>> 3 files changed, 2 insertions(+), 63 deletions(-)
>
> Applied to master, thanks.
>
I saw this on the agenda of the FOSDEM developer days, so thought it
would be discussed there before going forward with applying this. I
should have expressed my concerns on the patch immediately, sorry.
In fact, just recently I did start using 'make source-check' in our
environment. There are two use cases:
1. Our Buildroot tree contains a few packages with in-house source
code. They are typically Mercurial repositories. A developer changing
code in such a repository is expected to push to that repository and
then update the revision hash in the Buildroot .mk file (or if it
concerns a Linux/Uboot tree in the defconfig file).
Now, it sometimes happens that people update the hash but forget to
push the corresponding changes to the package's repository. This is of
course human error. In our delivery system (where people don't push
directly to the main repo but rather to some staging area where
changes are first automatically validated), it would be caught by
making a build, but this takes quite some time, order of 1 hour (we
have many defconfigs). Hence, we added an initial check to our
delivery system that just performs 'make source-check' for all
defconfigs. This only takes a few minutes. Deliveries that fail the
check are thus rejected very quickly and no computing resources are
lost.
2. Besides the in-house repositories, there is a use case even for
normal open-source packages. We have an internal BR2_PRIMARY_SITE set
up, that avoids package downloads over the internet. As we need to
guarantee reproducibility over a long time span, we cannot rely on
external sources and we need to mirror all packages used to that
primary site. Developers that enable new packages (or bump one) are
expected to upload the new sources to the primary site, but again may
forget to do so.
Again, this would be caught by the build step that takes ~1 hour,
while the source-check target would detect it in minutes.
Doing a 'make source' rather than 'make source-check' is an
alternative, but it takes longer and consumes much more network
bandwidth.
I would therefore like to find a way to reintroduce 'make
source-check', but of course taking into account the reasons of
removing it in the first place. Perhaps another implementation can be
found using the dl-wrapper.
Thoughts?
Thanks,
Thomas
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check
2018-01-10 12:04 ` Thomas De Schampheleire
@ 2018-01-10 12:41 ` Thomas Petazzoni
2018-01-10 12:51 ` Thomas De Schampheleire
2018-01-10 18:18 ` Yann E. MORIN
1 sibling, 1 reply; 24+ messages in thread
From: Thomas Petazzoni @ 2018-01-10 12:41 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 10 Jan 2018 13:04:03 +0100, Thomas De Schampheleire wrote:
> I saw this on the agenda of the FOSDEM developer days, so thought it
> would be discussed there before going forward with applying this. I
> should have expressed my concerns on the patch immediately, sorry.
No problem. It's never too late to revert something, or re-introduce it
in a different form.
However, my plan is in fact to get the Git download caching stuff
merged for 2018.02, hence before the meeting at FOSDEM. I've recently
worked on a few projects were the kernel is hosted in Git and regularly
changed, and the full git clone every time you update the kernel is
simply horrible, and makes Buildroot pretty much unusable.
> In fact, just recently I did start using 'make source-check' in our
> environment. There are two use cases:
> 1. Our Buildroot tree contains a few packages with in-house source
> code. They are typically Mercurial repositories. A developer changing
> code in such a repository is expected to push to that repository and
> then update the revision hash in the Buildroot .mk file (or if it
> concerns a Linux/Uboot tree in the defconfig file).
> Now, it sometimes happens that people update the hash but forget to
> push the corresponding changes to the package's repository. This is of
> course human error. In our delivery system (where people don't push
> directly to the main repo but rather to some staging area where
> changes are first automatically validated), it would be caught by
> making a build, but this takes quite some time, order of 1 hour (we
> have many defconfigs). Hence, we added an initial check to our
> delivery system that just performs 'make source-check' for all
> defconfigs. This only takes a few minutes. Deliveries that fail the
> check are thus rejected very quickly and no computing resources are
> lost.
>
> 2. Besides the in-house repositories, there is a use case even for
> normal open-source packages. We have an internal BR2_PRIMARY_SITE set
> up, that avoids package downloads over the internet. As we need to
> guarantee reproducibility over a long time span, we cannot rely on
> external sources and we need to mirror all packages used to that
> primary site. Developers that enable new packages (or bump one) are
> expected to upload the new sources to the primary site, but again may
> forget to do so.
> Again, this would be caught by the build step that takes ~1 hour,
> while the source-check target would detect it in minutes.
>
> Doing a 'make source' rather than 'make source-check' is an
> alternative, but it takes longer and consumes much more network
> bandwidth.
>
> I would therefore like to find a way to reintroduce 'make
> source-check', but of course taking into account the reasons of
> removing it in the first place. Perhaps another implementation can be
> found using the dl-wrapper.
We can definitely re-introduce a "make source-check" implementation. It
was mainly dropped because we had no identification anyone was really
using it.
Perhaps we could for now have "make source-check" implemented by doing
"make source", and later on re-optimize it?
As I said, I'd really to have this Git caching stuff merged in 2018.02.
The source-check removal has been on the mailing list since July 2017,
and nobody complained, until today. So while I do acknowledge that
there are some use cases for source-check, I'm not really willing to
delay merging the Git caching stuff just because source-check will be
missing. But maybe re-adding source-check is not that hard, and can be
done before 2018.02 as well.
Best regards,
Thomas Petazzoni
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check
2018-01-10 12:41 ` Thomas Petazzoni
@ 2018-01-10 12:51 ` Thomas De Schampheleire
0 siblings, 0 replies; 24+ messages in thread
From: Thomas De Schampheleire @ 2018-01-10 12:51 UTC (permalink / raw)
To: buildroot
Hi Thomas,
Thanks for your feedback...
2018-01-10 13:41 GMT+01:00 Thomas Petazzoni
<thomas.petazzoni@free-electrons.com>:
> Hello,
>
> On Wed, 10 Jan 2018 13:04:03 +0100, Thomas De Schampheleire wrote:
>
>> I saw this on the agenda of the FOSDEM developer days, so thought it
>> would be discussed there before going forward with applying this. I
>> should have expressed my concerns on the patch immediately, sorry.
>
> No problem. It's never too late to revert something, or re-introduce it
> in a different form.
>
> However, my plan is in fact to get the Git download caching stuff
> merged for 2018.02, hence before the meeting at FOSDEM. I've recently
> worked on a few projects were the kernel is hosted in Git and regularly
> changed, and the full git clone every time you update the kernel is
> simply horrible, and makes Buildroot pretty much unusable.
>
>> In fact, just recently I did start using 'make source-check' in our
>> environment. There are two use cases:
>> 1. Our Buildroot tree contains a few packages with in-house source
>> code. They are typically Mercurial repositories. A developer changing
>> code in such a repository is expected to push to that repository and
>> then update the revision hash in the Buildroot .mk file (or if it
>> concerns a Linux/Uboot tree in the defconfig file).
>> Now, it sometimes happens that people update the hash but forget to
>> push the corresponding changes to the package's repository. This is of
>> course human error. In our delivery system (where people don't push
>> directly to the main repo but rather to some staging area where
>> changes are first automatically validated), it would be caught by
>> making a build, but this takes quite some time, order of 1 hour (we
>> have many defconfigs). Hence, we added an initial check to our
>> delivery system that just performs 'make source-check' for all
>> defconfigs. This only takes a few minutes. Deliveries that fail the
>> check are thus rejected very quickly and no computing resources are
>> lost.
>>
>> 2. Besides the in-house repositories, there is a use case even for
>> normal open-source packages. We have an internal BR2_PRIMARY_SITE set
>> up, that avoids package downloads over the internet. As we need to
>> guarantee reproducibility over a long time span, we cannot rely on
>> external sources and we need to mirror all packages used to that
>> primary site. Developers that enable new packages (or bump one) are
>> expected to upload the new sources to the primary site, but again may
>> forget to do so.
>> Again, this would be caught by the build step that takes ~1 hour,
>> while the source-check target would detect it in minutes.
>>
>> Doing a 'make source' rather than 'make source-check' is an
>> alternative, but it takes longer and consumes much more network
>> bandwidth.
>>
>> I would therefore like to find a way to reintroduce 'make
>> source-check', but of course taking into account the reasons of
>> removing it in the first place. Perhaps another implementation can be
>> found using the dl-wrapper.
>
> We can definitely re-introduce a "make source-check" implementation. It
> was mainly dropped because we had no identification anyone was really
> using it.
>
> Perhaps we could for now have "make source-check" implemented by doing
> "make source", and later on re-optimize it?
>
> As I said, I'd really to have this Git caching stuff merged in 2018.02.
> The source-check removal has been on the mailing list since July 2017,
> and nobody complained, until today. So while I do acknowledge that
> there are some use cases for source-check, I'm not really willing to
> delay merging the Git caching stuff just because source-check will be
> missing. But maybe re-adding source-check is not that hard, and can be
> done before 2018.02 as well.
By no means do I want to delay other patches like git caching because of this.
I can definitely live with a temporary local patch on our tree, but of
course would like to find a method that can be upstreamed.
/Thomas
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check
2018-01-10 12:04 ` Thomas De Schampheleire
2018-01-10 12:41 ` Thomas Petazzoni
@ 2018-01-10 18:18 ` Yann E. MORIN
1 sibling, 0 replies; 24+ messages in thread
From: Yann E. MORIN @ 2018-01-10 18:18 UTC (permalink / raw)
To: buildroot
Thomas?, All,
On 2018-01-10 13:04 +0100, Thomas De Schampheleire spake thusly:
> 2018-01-08 21:28 GMT+01:00 Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com>:
> >> ---
> >> Makefile | 7 +------
> >> package/pkg-download.mk | 44 --------------------------------------------
> >> package/pkg-generic.mk | 14 +-------------
> >> 3 files changed, 2 insertions(+), 63 deletions(-)
> >
> > Applied to master, thanks.
>
> I saw this on the agenda of the FOSDEM developer days, so thought it
> would be discussed there before going forward with applying this. I
> should have expressed my concerns on the patch immediately, sorry.
>
> In fact, just recently I did start using 'make source-check' in our
> environment. There are two use cases:
> 1. Our Buildroot tree contains a few packages with in-house source
> code. They are typically Mercurial repositories. A developer changing
> code in such a repository is expected to push to that repository and
> then update the revision hash in the Buildroot .mk file (or if it
> concerns a Linux/Uboot tree in the defconfig file).
> Now, it sometimes happens that people update the hash but forget to
> push the corresponding changes to the package's repository. This is of
> course human error. In our delivery system (where people don't push
> directly to the main repo but rather to some staging area where
> changes are first automatically validated), it would be caught by
> making a build, but this takes quite some time, order of 1 hour (we
> have many defconfigs). Hence, we added an initial check to our
> delivery system that just performs 'make source-check' for all
> defconfigs. This only takes a few minutes. Deliveries that fail the
> check are thus rejected very quickly and no computing resources are
> lost.
>
> 2. Besides the in-house repositories, there is a use case even for
> normal open-source packages. We have an internal BR2_PRIMARY_SITE set
> up, that avoids package downloads over the internet. As we need to
> guarantee reproducibility over a long time span, we cannot rely on
> external sources and we need to mirror all packages used to that
> primary site. Developers that enable new packages (or bump one) are
> expected to upload the new sources to the primary site, but again may
> forget to do so.
> Again, this would be caught by the build step that takes ~1 hour,
> while the source-check target would detect it in minutes.
>
> Doing a 'make source' rather than 'make source-check' is an
> alternative, but it takes longer and consumes much more network
> bandwidth.
>
> I would therefore like to find a way to reintroduce 'make
> source-check', but of course taking into account the reasons of
> removing it in the first place. Perhaps another implementation can be
> found using the dl-wrapper.
>
> Thoughts?
The way I haved solved this issue (and is something I forgot to talk
about during my talk at ELCE) without source-check is that the automatic
build system (jenkins and gitlab-ci) have a job that does basically:
for t in ${targets}; do
make ${t}_defconfig
sed -r -e 's:^BR2_WGET="(.+)"$:BR2_WGET="/path/to/dl-wrapper \1":'
make source
done
(ditto for svn, git, hg, scp...)
Then the /path/to/dl-wrapper basically does:
#!/bin/bash
"${@}" # Do the actual download
if [[ ${url} =~ ^http://internal-server/mirror/ ]]; then
: ; # Already on server
elif [[ ${url} =~ ^http://internal-server/repositories/ ]]; then
# New revision, push to filer
scp "${url##*/}" "user at internal-server:path/to/mirror/"
elif [[ ${url} =~ ^http://backup-server/mirror/ ]]; then
# From backup, ARG...
send-email "ARG: mot on mirror, nor upstream, from backup ${pkg}"
scp "${url##*/}" "user at internal-server:path/to/mirror/"
else
# From upstream
send-email "WARN: new package from upstream: ${pkg}"
scp "${url##*/}" "user at internal-server:path/to/mirror/"
fi
Of course, this is extermely simplified, but you get the idea...
Then the real build jobs only have access to the internal mirror, and
never to upstream, whether it is internal or exgternal. So we can also
fill a directory for delivery and archiving when needed.
Even though we still have a Buildroot with source-check, we are not
using it because it is more powerfull to just wrap around the real
download.
Regards,
Yann E. MORIN.
> Thanks,
> Thomas
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 13/13] WIP: support package with '-'
2017-10-25 20:10 ` [Buildroot] [PATCH v2 13/13] WIP: support package with '-' Peter Seiderer
@ 2018-02-05 13:33 ` Thomas Petazzoni
0 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2018-02-05 13:33 UTC (permalink / raw)
To: buildroot
Hello,
On Wed, 25 Oct 2017 22:10:03 +0200, Peter Seiderer wrote:
> From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>
> ---
Why is that needed? I do remember some discussion about it during the
Buildroot Summer Camp 2017, but I don't remember what was the problem.
The empty commit log doesn't really help here.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 12/13] download: git: introduce cache feature
2017-10-25 20:10 ` [Buildroot] [PATCH v2 12/13] download: git: introduce cache feature Peter Seiderer
@ 2018-02-05 13:46 ` Thomas Petazzoni
0 siblings, 0 replies; 24+ messages in thread
From: Thomas Petazzoni @ 2018-02-05 13:46 UTC (permalink / raw)
To: buildroot
Hello,
On issue below.
On Wed, 25 Oct 2017 22:10:02 +0200, Peter Seiderer wrote:
> diff --git a/support/download/git b/support/download/git
> index 58a2c6ad9d..5d09cc0d7a 100755
> --- a/support/download/git
> +++ b/support/download/git
> @@ -39,28 +39,34 @@ _git() {
> eval ${GIT} "${@}"
> }
>
> -# Try a shallow clone, since it is faster than a full clone - but that only
> -# works if the version is a ref (tag or branch). Before trying to do a shallow
> -# clone we check if ${cset} is in the list provided by git ls-remote. If not
> -# we fall back on a full clone.
> -#
> -# Messages for the type of clone used are provided to ease debugging in case of
> -# problems
> -git_done=0
> -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
> - printf "Doing shallow clone\n"
> - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
> - git_done=1
> - else
> - printf "Shallow clone failed, falling back to doing a full clone\n"
> +# We want to check if a cache of the git clone of this repo already exists.
> +git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
> +
> +# If the cache directory already exists, don't try to clone.
> +if [ ! -d "${git_cache}" ]; then
> + # Try a shallow clone, since it is faster than a full clone - but that
> + # only works if the versionis a ref (tag or branch). Before trying to do a
> + # shallow clone we check if ${cset} is in the list provided by git
> + # ls-remote. If not we fall back on a full clone.
> + #
> + # Messages for the type of clone used are provided to ease debugging in
> + # case of problems
> + git_done=0
> + if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
> + printf "Doing shallow clone\n"
> + if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then
> + git_done=1
> + else
> + printf "Shallow clone failed, falling back to doing a full clone\n"
> + fi
> + fi
> + if [ ${git_done} -eq 0 ]; then
> + printf "Doing full clone\n"
> + _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'"
> fi
> -fi
> -if [ ${git_done} -eq 0 ]; then
> - printf "Doing full clone\n"
> - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
> fi
This looks all good, except that the rest of the script does:
if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then
printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}"
fi
And "origin" will be the first remote from which the Git cache was
started, but not necessarily the current Git repository that we should
pull from. This "origin" should be changed to ${uri} it seems.
This is not a problem today because we always clone, so origin is
always ${uri}. But with Git caching, it is no longer the case.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
http://bootlin.com
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 02/13] core/pkg-download: change all helpers to use common options
2017-10-25 20:09 ` [Buildroot] [PATCH v2 02/13] core/pkg-download: change all helpers to use common options Peter Seiderer
@ 2018-02-05 15:34 ` Luca Ceresoli
0 siblings, 0 replies; 24+ messages in thread
From: Luca Ceresoli @ 2018-02-05 15:34 UTC (permalink / raw)
To: buildroot
Hi,
On 25/10/2017 22:09, Peter Seiderer wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
>
> Currently all download helpers accepts the local output file, the remote
> locations, the changesets and so on... as positional arguments.
>
> This was well and nice when that's was all we needed.
>
> But then we added an option to quiesce their verbosity, and that was
> shoehorned with a trivial getopts, still keeping all the existing
> positional arguments as... positional arguments.
>
> Adding yet more options while keeping positional arguments will not be
> very easy, even if we do not envision any new option in the foreseeable
> future (but 640K ought to be enough for everyone, remember? ;-) ).
>
> Change all helpers to accept a set of generic options (-q for quiet and
> -o for the output file) as well as helper-specific options (like -r for
> the repository, -c for a changeset...).
>
> Maxime:
> Changed -R to -r for recurse (only for the git backend)
> Changed -r to -u for URI (for all backend)
> Change -R to -c for cset (for CVS and SVN backend)
> Add the export of the BR_BACKEND_DL_GETOPTS so all the backend wrapper
> can use the same option easily
> Now all the backends use the same common options.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
> Changes v1 --> v2:
> - from https://github.com/maximeh/buildroot/commit/a1facaac906794d027a0ed5837fb617f93cad662.patch
> - no more change to support/download/check-hash
> ---
> package/pkg-download.mk | 38 +++++++++++++++++++-------------------
> support/download/bzr | 25 ++++++++++++++-----------
> support/download/cp | 17 +++++++++--------
> support/download/cvs | 34 +++++++++++++++++++---------------
> support/download/dl-wrapper | 7 ++++++-
> support/download/git | 33 +++++++++++++++++----------------
> support/download/hg | 25 ++++++++++++++-----------
> support/download/scp | 19 ++++++++++---------
> support/download/svn | 25 ++++++++++++++-----------
> support/download/wget | 17 +++++++++--------
> 10 files changed, 131 insertions(+), 109 deletions(-)
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index 2b845e77a2..4d724e7494 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -77,9 +77,9 @@ define DOWNLOAD_GIT
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_RAW_BASE_NAME) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_RAW_BASE_NAME) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -88,9 +88,9 @@ define DOWNLOAD_BZR
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_RAW_BASE_NAME) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_RAW_BASE_NAME) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -99,10 +99,10 @@ define DOWNLOAD_CVS
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_RAWNAME) \
> - $($(PKG)_RAW_BASE_NAME) \
> + -u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
> + -c $($(PKG)_DL_VERSION) \
> + -N $($(PKG)_RAWNAME) \
> + -n $($(PKG)_RAW_BASE_NAME) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -111,9 +111,9 @@ define DOWNLOAD_SVN
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_RAW_BASE_NAME) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_RAW_BASE_NAME) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -126,7 +126,7 @@ define DOWNLOAD_SCP
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - '$(call stripurischeme,$(call qstrip,$(1)))' \
> + -u '$(call stripurischeme,$(call qstrip,$(1)))' \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -135,9 +135,9 @@ define DOWNLOAD_HG
> -o $(DL_DIR)/$($(PKG)_SOURCE) \
> $(QUIET) \
> -- \
> - $($(PKG)_SITE) \
> - $($(PKG)_DL_VERSION) \
> - $($(PKG)_RAW_BASE_NAME) \
> + -u $($(PKG)_SITE) \
> + -c $($(PKG)_DL_VERSION) \
> + -n $($(PKG)_RAW_BASE_NAME) \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -147,7 +147,7 @@ define DOWNLOAD_WGET
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - '$(call qstrip,$(1))' \
> + -u '$(call qstrip,$(1))' \
> $($(PKG)_DL_OPTS)
> endef
>
> @@ -157,7 +157,7 @@ define DOWNLOAD_LOCALFILES
> -H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
> $(QUIET) \
> -- \
> - $(call stripurischeme,$(call qstrip,$(1))) \
> + -u $(call stripurischeme,$(call qstrip,$(1))) \
> $($(PKG)_DL_OPTS)
> endef
>
> diff --git a/support/download/bzr b/support/download/bzr
> index 75b7b415c1..5289a421cd 100755
> --- a/support/download/bzr
> +++ b/support/download/bzr
> @@ -5,28 +5,31 @@ set -e
>
> # Download helper for bzr, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../bzr [-q] OUT_FILE REPO_URL REV BASENAME
> +# Options:
> +# -q Be quiet
> +# -o FILE Generate archive in FILE.
> +# -u URI Clone from repository at URI.
> +# -c CSET Use changeset (or revision) CSET.
> +# -n NAME Use basename NAME.
> #
> # Environment:
> # BZR : the bzr command to call
>
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-q;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + c) cset="${OPTARG}";;
> + n) basename="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -repo="${2}"
> -rev="${3}"
> -basename="${4}"
> -
> -shift 4 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -51,5 +54,5 @@ if [ ${bzr_version} -ge ${bzr_min_version} ]; then
> fi
>
> _bzr export ${verbose} --root="'${basename}/'" --format=tgz \
> - ${timestamp_opt} - "${@}" "'${repo}'" -r "'${rev}'" \
> + ${timestamp_opt} - "${@}" "'${uri}'" -r "'${cset}'" \
> >"${output}"
> diff --git a/support/download/cp b/support/download/cp
> index 0ee1f3ba82..52fe2de83d 100755
> --- a/support/download/cp
> +++ b/support/download/cp
> @@ -5,8 +5,10 @@ set -e
>
> # Download helper for cp, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../cp [-q] OUT_FILE SRC_FILE
> +# Options:
> +# -q Be quiet.
> +# -o FILE Copy to file FILE.
> +# -u FILE Copy from file FILE.
> #
> # Environment:
> # LOCALFILES: the cp command to call
> @@ -17,18 +19,17 @@ set -e
> # Make 'cp' verbose by default, so it behaves a bit like the others.
> verbose=-v
>
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=;;
> + o) output="${OPTARG}";;
> + u) source="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -source="${2}"
> -
> -shift 2 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> diff --git a/support/download/cvs b/support/download/cvs
> index 50050ab1c9..69d5c71f28 100755
> --- a/support/download/cvs
> +++ b/support/download/cvs
> @@ -5,28 +5,32 @@ set -e
>
> # Download helper for cvs, to be called from the download wrapper script
> #
> -# Call it as:
> -# .../cvs [-q] OUT_FILE CVS_URL REV PKG_NAME BASENAME
> +# Options:
> +# -q Be quiet
> +# -o FILE Generate archive in FILE.
> +# -u URI Checkout from repository at URI.
> +# -c REV Use revision REV.
> +# -N RAWNAME Use rawname (aka module name) RAWNAME.
> +# -n NAME Use basename NAME.
> #
> # Environment:
> # CVS : the cvs command to call
>
> verbose=
> -while getopts :q OPT; do
> +while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
> case "${OPT}" in
> q) verbose=-Q;;
> + o) output="${OPTARG}";;
> + u) uri="${OPTARG}";;
> + c) rev="${OPTARG}";;
> + N) rawname="${OPTARG}";;
> + n) basename="${OPTARG}";;
> + :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
> \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> esac
> done
> -shift $((OPTIND-1))
>
> -output="${1}"
> -repo="${2}"
> -rev="${3}"
> -rawname="${4}"
> -basename="${5}"
> -
> -shift 5 # Get rid of our options
> +shift $((OPTIND-1)) # Get rid of our options
>
> # Caller needs to single-quote its arguments to prevent them from
> # being expanded a second time (in case there are spaces in them)
> @@ -42,14 +46,14 @@ else
> select="-r"
> fi
>
> -# The absence of an initial : on ${repo} means access method undefined
> -if [[ ! "${repo}" =~ ^: ]]; then
> +# The absence of an initial : on ${uri} means access method undefined
> +if [[ ! "${uri}" =~ ^: ]]; then
> # defaults to anonymous pserver
> - repo=":pserver:anonymous@${repo}"
> + uri=":pserver:anonymous@${uri}"
> fi
>
> export TZ=UTC
> -_cvs ${verbose} -z3 -d"'${repo}'" \
> +_cvs ${verbose} -z3 -d"'${uri}'" \
> co "${@}" -d "'${basename}'" ${select} "'${rev}'" -P "'${rawname}'"
>
> tar czf "${output}" "${basename}"
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index f944b71db5..510e7ef852 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -19,6 +19,8 @@
> # We want to catch any unexpected failure, and exit immediately.
> set -e
>
> +export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:q"
A minor not here.
I don't like very much this being passed to the backends in the
environment: it makes them non-self-standing, so we cannot call them
directly for testing. I'd find it more correct that the backend scripts
source a "dl-common" file that sets BACKEND_DL_GETOPTS without exporting
it, and that could also contain other common utility code.
But hey, the backends are not currently self-standing anyway since they
do "eval ${GIT}" and the like!
[Tested with wget, git, hg, svn, bzr]
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
--
Luca
^ permalink raw reply [flat|nested] 24+ messages in thread
* [Buildroot] [PATCH v2 03/13] download: put most of the infra in dl-wrapper
2017-10-25 20:09 ` [Buildroot] [PATCH v2 03/13] download: put most of the infra in dl-wrapper Peter Seiderer
@ 2018-02-05 15:34 ` Luca Ceresoli
0 siblings, 0 replies; 24+ messages in thread
From: Luca Ceresoli @ 2018-02-05 15:34 UTC (permalink / raw)
To: buildroot
Hi,
On 25/10/2017 22:09, Peter Seiderer wrote:
> From: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
>
> The goal here is to simplify the infrastructure by putting most of the
> code in the dl-wrapper as it's easier to implement and to read.
>
> Most of the function where common already, this patch finalizes it by
^ functions were
> making the pkg-download.mk pass all the parameters needed to the
> dl-wrapper which in turns will pass everything to every backend.
>
> The backend will then cherry-pick what it needs from these arguments
> and act accordingly.
>
> It eases the transition to the addition of a sub directory per package
> in the DL_DIR, and later on, a git cache.
>
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
[...]
> -################################################################################
> -# DOWNLOAD -- Download helper. Will try to download source from:
> -# 1) BR2_PRIMARY_SITE if enabled
> -# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
> -# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
> -#
> -# Argument 1 is the source location
> -#
> -# E.G. use like this:
> -# $(call DOWNLOAD,$(FOO_SITE))
> -#
> -# For PRIMARY and BACKUP site, any ? in the URL is replaced by %3F. A ? in
> -# the URL is used to separate query arguments, but the PRIMARY and BACKUP
> -# sites serve just plain files.
> -################################################################################
> -
> -define DOWNLOAD
> - $(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
> -endef
> -
> -define DOWNLOAD_INNER
> - $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
> - if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
> - case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
> - file) $(call $(3)_LOCALFILES,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
> - scp) $(call $(3)_SCP,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
> - *) $(call $(3)_WGET,$(BR2_PRIMARY_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ;; \
[...]
> diff --git a/support/download/cp b/support/download/file
> similarity index 90%
> rename from support/download/cp
> rename to support/download/file
> index 52fe2de83d..a3e616a181 100755
[...]
> -_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
> +# Replace every '?' with '%3F' in the filename; only for the PRIMARY and BACKUP
> +# mirror
> +[ -n "${encode}" ] && filename=${filename//\?/%3F}
> +
> +_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
I suggest to retain the original comment above, which is clearer.
For the rest:
[Tested with wget, git, hg, svn, bzr]
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
--
Luca
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2018-02-05 15:34 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-10-25 20:09 [Buildroot] [PATCH v2 00/13] New DL_DIR organisation; git cache feature Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 01/13] pkg-{download, generic}: remove source-check Peter Seiderer
2018-01-08 20:22 ` Yann E. MORIN
2018-01-08 20:28 ` Thomas Petazzoni
2018-01-10 12:04 ` Thomas De Schampheleire
2018-01-10 12:41 ` Thomas Petazzoni
2018-01-10 12:51 ` Thomas De Schampheleire
2018-01-10 18:18 ` Yann E. MORIN
2017-10-25 20:09 ` [Buildroot] [PATCH v2 02/13] core/pkg-download: change all helpers to use common options Peter Seiderer
2018-02-05 15:34 ` Luca Ceresoli
2017-10-25 20:09 ` [Buildroot] [PATCH v2 03/13] download: put most of the infra in dl-wrapper Peter Seiderer
2018-02-05 15:34 ` Luca Ceresoli
2017-10-25 20:09 ` [Buildroot] [PATCH v2 04/13] packages: use new $($PKG)_DL_DIR) variable Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 05/13] pkg-{download, generic}: use new $($(PKG)_DL_DIR) Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 06/13] support/download: make sure the download folder is created Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 07/13] pkg-generic: add a subdirectory to the DL_DIR Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 08/13] pkg-download: support new subdir for mirrors Peter Seiderer
2017-10-25 20:09 ` [Buildroot] [PATCH v2 09/13] pkg-generic: introduce _SAME_SOURCE_AS Peter Seiderer
2017-10-25 20:10 ` [Buildroot] [PATCH v2 10/13] help/manual: update help about the new $(LIBFOO_DL_DIR) Peter Seiderer
2017-10-25 20:10 ` [Buildroot] [PATCH v2 11/13] download: add flock call before dl-wrapper Peter Seiderer
2017-10-25 20:10 ` [Buildroot] [PATCH v2 12/13] download: git: introduce cache feature Peter Seiderer
2018-02-05 13:46 ` Thomas Petazzoni
2017-10-25 20:10 ` [Buildroot] [PATCH v2 13/13] WIP: support package with '-' Peter Seiderer
2018-02-05 13:33 ` Thomas Petazzoni
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox