* [Buildroot] [PATCH 01/14] Makefile: add BR2_DL_DIR to EXTRA_ENV
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-07-02 15:24 ` Peter Korsgaard
2014-06-30 22:26 ` [Buildroot] [PATCH 02/14] pkg-infra: also set PKGDIR for the download step Yann E. MORIN
` (12 subsequent siblings)
13 siblings, 1 reply; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Also export BR2_DL_DIR for incoming download helper scripts.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
package/Makefile.in | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/Makefile.in b/package/Makefile.in
index 0233506..97053ba 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -305,6 +305,7 @@ HOST_MAKE_ENV=PATH=$(BR_PATH) \
# post-images)
EXTRA_ENV=\
PATH=$(BR_PATH) \
+ BR2_DL_DIR=$(BR2_DL_DIR) \
BUILD_DIR=$(BUILD_DIR)
################################################################################
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 02/14] pkg-infra: also set PKGDIR for the download step
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 01/14] Makefile: add BR2_DL_DIR to EXTRA_ENV Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-07-02 15:24 ` Peter Korsgaard
2014-06-30 22:26 ` [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script Yann E. MORIN
` (11 subsequent siblings)
13 siblings, 1 reply; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
This will be needed to get the hash file, to check the
downloaded files.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
---
package/pkg-generic.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 54193d2..ab1f8d2 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -602,6 +602,7 @@ $$($(2)_TARGET_PATCH): RAWNAME=$$(patsubst host-%,%,$(1))
$$($(2)_TARGET_PATCH): PKGDIR=$(pkgdir)
$$($(2)_TARGET_EXTRACT): PKG=$(2)
$$($(2)_TARGET_SOURCE): PKG=$(2)
+$$($(2)_TARGET_SOURCE): PKGDIR=$(pkgdir)
$$($(2)_TARGET_DIRCLEAN): PKG=$(2)
# Compute the name of the Kconfig option that correspond to the
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 01/14] Makefile: add BR2_DL_DIR to EXTRA_ENV Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 02/14] pkg-infra: also set PKGDIR for the download step Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-07-02 15:28 ` Peter Korsgaard
2014-06-30 22:26 ` [Buildroot] [PATCH 04/14] pkg-infra: move the svn " Yann E. MORIN
` (10 subsequent siblings)
13 siblings, 1 reply; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
The git download helper is getting a bit more complex. Fixing it in the
Makefile when it breaks (like the recent breakage with a non-existing
sha1-cset) proves to be challenging, to say the least.
Move it into a shell script in support/download/git, which will make
it much easier to read, maintain, fix and enhance in the future.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
package/pkg-download.mk | 17 +++--------------
support/download/git | 37 +++++++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+), 14 deletions(-)
create mode 100755 support/download/git
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index e07fd1b..c848f6a 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -12,7 +12,7 @@ WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
SVN := $(call qstrip,$(BR2_SVN))
CVS := $(call qstrip,$(BR2_CVS))
BZR := $(call qstrip,$(BR2_BZR))
-GIT := $(call qstrip,$(BR2_GIT))
+export GIT := $(call qstrip,$(BR2_GIT))
HG := $(call qstrip,$(BR2_HG)) $(QUIET)
SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
@@ -84,19 +84,8 @@ github = https://github.com/$(1)/$(2)/archive/$(3)
# problems
define DOWNLOAD_GIT
test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
- (pushd $(DL_DIR) > /dev/null && \
- ((test "`git ls-remote $($(PKG)_SITE) $($(PKG)_DL_VERSION)`" && \
- echo "Doing shallow clone" && \
- $(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME)) || \
- (echo "Doing full clone" && \
- $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME))) && \
- pushd $($(PKG)_BASE_NAME) > /dev/null && \
- $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ -o $(DL_DIR)/.$($(PKG)_SOURCE).tmp $($(PKG)_DL_VERSION) && \
- gzip -c $(DL_DIR)/.$($(PKG)_SOURCE).tmp > $(DL_DIR)/$($(PKG)_SOURCE) && \
- rm -f $(DL_DIR)/.$($(PKG)_SOURCE).tmp && \
- popd > /dev/null && \
- rm -rf $($(PKG)_DL_DIR) && \
- popd > /dev/null)
+ $(EXTRA_ENV) support/download/git $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+ $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
endef
# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
diff --git a/support/download/git b/support/download/git
new file mode 100755
index 0000000..96db3a9
--- /dev/null
+++ b/support/download/git
@@ -0,0 +1,37 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for git
+# Call it with:
+# $1: git repo
+# $2: git cset
+# $3: package's basename (eg. foobar-1.2.3)
+# $4: output file
+# And this environment:
+# BR2_DL_DIR: path to Buildroot's download dir
+# GIT : the git command to call
+
+repo="${1}"
+cset="${2}"
+basename="${3}"
+output="${4}"
+
+repodir="${BR2_DL_DIR}/${basename}"
+
+if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
+ printf "Doing shallow clone\n"
+ ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${repodir}"
+else
+ printf "Doing full clone\n"
+ ${GIT} clone --bare "${repo}" "${repodir}"
+fi
+
+pushd "${repodir}"
+${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
+gzip -c "${output}.tmp" >"${output}"
+rm -f "${output}.tmp"
+popd
+
+rm -rf "${repodir}"
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-06-30 22:26 ` [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script Yann E. MORIN
@ 2014-07-02 15:28 ` Peter Korsgaard
2014-07-02 15:44 ` Thomas Petazzoni
2014-07-02 20:25 ` Yann E. MORIN
0 siblings, 2 replies; 29+ messages in thread
From: Peter Korsgaard @ 2014-07-02 15:28 UTC (permalink / raw)
To: buildroot
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> The git download helper is getting a bit more complex. Fixing it in the
> Makefile when it breaks (like the recent breakage with a non-existing
> sha1-cset) proves to be challenging, to say the least.
> Move it into a shell script in support/download/git, which will make
> it much easier to read, maintain, fix and enhance in the future.
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Reviewed-by: Samuel Martin <s.martin49@gmail.com>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Thanks. I've verified that the (non-gzip'ed) tarball is identical to
what we had before, but I noticed that we no longer delete the temporary
repo in DL_DIR.
Did you do that change on purpose? I don't think we want to keep it, do
we?
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 15:28 ` Peter Korsgaard
@ 2014-07-02 15:44 ` Thomas Petazzoni
2014-07-02 17:19 ` Thomas De Schampheleire
2014-07-02 20:25 ` Yann E. MORIN
1 sibling, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2014-07-02 15:44 UTC (permalink / raw)
To: buildroot
Dear Peter Korsgaard,
On Wed, 02 Jul 2014 17:28:04 +0200, Peter Korsgaard wrote:
> Thanks. I've verified that the (non-gzip'ed) tarball is identical to
> what we had before, but I noticed that we no longer delete the temporary
> repo in DL_DIR.
>
> Did you do that change on purpose? I don't think we want to keep it, do
> we?
Hum, I'm confused, didn't we say that we should no longer do any
temporary thing in $(DL_DIR) in order to allow parallel builds of
separate Buildroot instances to not mess up with each other? I think we
said that the process should be:
1/ Clone the repo in $(BUILD_DIR)
2/ Create the tarball of the repo in $(BUILD_DIR)
3/ Move the tarball from $(BUILD_DIR) to $(DL_DIR) with a temporary
unique file name.
4/ Rename the tarball in $(DL_DIR) to its final name
Steps (3) and (4) are separated so that if $(DL_DIR) and $(BUILD_DIR)
are in separate filesystems, the rename to the final name remains an
atomic operation.
And yes, the git download helper from Yann doesn't seem to implement
this logic (or I got lost with the variable names, which is very
possible).
Yann?
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 15:44 ` Thomas Petazzoni
@ 2014-07-02 17:19 ` Thomas De Schampheleire
2014-07-02 17:22 ` Thomas Petazzoni
0 siblings, 1 reply; 29+ messages in thread
From: Thomas De Schampheleire @ 2014-07-02 17:19 UTC (permalink / raw)
To: buildroot
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> schreef:
>Dear Peter Korsgaard,
>
>On Wed, 02 Jul 2014 17:28:04 +0200, Peter Korsgaard wrote:
>
>> Thanks. I've verified that the (non-gzip'ed) tarball is identical to
>> what we had before, but I noticed that we no longer delete the temporary
>> repo in DL_DIR.
>>
>> Did you do that change on purpose? I don't think we want to keep it, do
>> we?
>
>Hum, I'm confused, didn't we say that we should no longer do any
>temporary thing in $(DL_DIR) in order to allow parallel builds of
>separate Buildroot instances to not mess up with each other? I think we
>said that the process should be:
>
> 1/ Clone the repo in $(BUILD_DIR)
> 2/ Create the tarball of the repo in $(BUILD_DIR)
> 3/ Move the tarball from $(BUILD_DIR) to $(DL_DIR) with a temporary
> unique file name.
> 4/ Rename the tarball in $(DL_DIR) to its final name
>
>Steps (3) and (4) are separated so that if $(DL_DIR) and $(BUILD_DIR)
>are in separate filesystems, the rename to the final name remains an
>atomic operation.
>
>And yes, the git download helper from Yann doesn't seem to implement
>this logic (or I got lost with the variable names, which is very
>possible).
This patch (3/14) only moves the existing logic to a separate script,
just like for the other helpers. Later patches in the same series
improve the logic. In particular, not using DL_DIR as scratchpad is
implemented in patch 11/14 http://patchwork.ozlabs.org/patch/365791/
Best regards,
Thomas
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 17:19 ` Thomas De Schampheleire
@ 2014-07-02 17:22 ` Thomas Petazzoni
2014-07-02 17:31 ` Thomas De Schampheleire
0 siblings, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2014-07-02 17:22 UTC (permalink / raw)
To: buildroot
Dear Thomas De Schampheleire,
On Wed, 2 Jul 2014 19:19:42 +0200, Thomas De Schampheleire wrote:
> >> Thanks. I've verified that the (non-gzip'ed) tarball is identical to
> >> what we had before, but I noticed that we no longer delete the temporary
> >> repo in DL_DIR.
> >>
> >> Did you do that change on purpose? I don't think we want to keep it, do
> >> we?
> >
> >Hum, I'm confused, didn't we say that we should no longer do any
> >temporary thing in $(DL_DIR) in order to allow parallel builds of
> >separate Buildroot instances to not mess up with each other? I think we
> >said that the process should be:
> >
> > 1/ Clone the repo in $(BUILD_DIR)
> > 2/ Create the tarball of the repo in $(BUILD_DIR)
> > 3/ Move the tarball from $(BUILD_DIR) to $(DL_DIR) with a temporary
> > unique file name.
> > 4/ Rename the tarball in $(DL_DIR) to its final name
> >
> >Steps (3) and (4) are separated so that if $(DL_DIR) and $(BUILD_DIR)
> >are in separate filesystems, the rename to the final name remains an
> >atomic operation.
> >
> >And yes, the git download helper from Yann doesn't seem to implement
> >this logic (or I got lost with the variable names, which is very
> >possible).
>
> This patch (3/14) only moves the existing logic to a separate script,
> just like for the other helpers. Later patches in the same series
> improve the logic. In particular, not using DL_DIR as scratchpad is
> implemented in patch 11/14 http://patchwork.ozlabs.org/patch/365791/
Ah, ok, thanks for the clarification. However that doesn't explain why
the temporary repo in $(DL_DIR) is not being removed. That's a
regression compared to the original code in the .mk file, no?
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 17:22 ` Thomas Petazzoni
@ 2014-07-02 17:31 ` Thomas De Schampheleire
2014-07-02 20:29 ` Yann E. MORIN
2014-07-02 20:31 ` Thomas Petazzoni
0 siblings, 2 replies; 29+ messages in thread
From: Thomas De Schampheleire @ 2014-07-02 17:31 UTC (permalink / raw)
To: buildroot
On Wed, Jul 2, 2014 at 7:22 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Wed, 2 Jul 2014 19:19:42 +0200, Thomas De Schampheleire wrote:
>
>> >> Thanks. I've verified that the (non-gzip'ed) tarball is identical to
>> >> what we had before, but I noticed that we no longer delete the temporary
>> >> repo in DL_DIR.
>> >>
>> >> Did you do that change on purpose? I don't think we want to keep it, do
>> >> we?
>> >
>> >Hum, I'm confused, didn't we say that we should no longer do any
>> >temporary thing in $(DL_DIR) in order to allow parallel builds of
>> >separate Buildroot instances to not mess up with each other? I think we
>> >said that the process should be:
>> >
>> > 1/ Clone the repo in $(BUILD_DIR)
>> > 2/ Create the tarball of the repo in $(BUILD_DIR)
>> > 3/ Move the tarball from $(BUILD_DIR) to $(DL_DIR) with a temporary
>> > unique file name.
>> > 4/ Rename the tarball in $(DL_DIR) to its final name
>> >
>> >Steps (3) and (4) are separated so that if $(DL_DIR) and $(BUILD_DIR)
>> >are in separate filesystems, the rename to the final name remains an
>> >atomic operation.
>> >
>> >And yes, the git download helper from Yann doesn't seem to implement
>> >this logic (or I got lost with the variable names, which is very
>> >possible).
>>
>> This patch (3/14) only moves the existing logic to a separate script,
>> just like for the other helpers. Later patches in the same series
>> improve the logic. In particular, not using DL_DIR as scratchpad is
>> implemented in patch 11/14 http://patchwork.ozlabs.org/patch/365791/
>
> Ah, ok, thanks for the clarification. However that doesn't explain why
> the temporary repo in $(DL_DIR) is not being removed. That's a
> regression compared to the original code in the .mk file, no?
Patch 11 does:
-rm -rf "${repodir}"
+rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
+exit ${ret}
and
-repodir="${BR2_DL_DIR}/${basename}"
+repodir="${basename}.tmp-git-checkout"
+tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
So from the code (haven't verified this now) there is no repo at all
in DL_DIR, so that repo does not need to be removed. The repo $repodir
_is_ removed, including all of the other temporary stuff tmp_tar and
tmp_output.
Peter, did you test this with all patches applied?
Thanks,
Thomas
^ permalink raw reply [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 17:31 ` Thomas De Schampheleire
@ 2014-07-02 20:29 ` Yann E. MORIN
2014-07-02 20:31 ` Thomas Petazzoni
1 sibling, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-07-02 20:29 UTC (permalink / raw)
To: buildroot
Thomas?, Peter, All,
On 2014-07-02 19:31 +0200, Thomas De Schampheleire spake thusly:
> On Wed, Jul 2, 2014 at 7:22 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
> > Dear Thomas De Schampheleire,
> >
> > On Wed, 2 Jul 2014 19:19:42 +0200, Thomas De Schampheleire wrote:
> >
> >> >> Thanks. I've verified that the (non-gzip'ed) tarball is identical to
> >> >> what we had before, but I noticed that we no longer delete the temporary
> >> >> repo in DL_DIR.
> >> >>
> >> >> Did you do that change on purpose? I don't think we want to keep it, do
> >> >> we?
> >> >
> >> >Hum, I'm confused, didn't we say that we should no longer do any
> >> >temporary thing in $(DL_DIR) in order to allow parallel builds of
> >> >separate Buildroot instances to not mess up with each other? I think we
> >> >said that the process should be:
> >> >
> >> > 1/ Clone the repo in $(BUILD_DIR)
> >> > 2/ Create the tarball of the repo in $(BUILD_DIR)
> >> > 3/ Move the tarball from $(BUILD_DIR) to $(DL_DIR) with a temporary
> >> > unique file name.
> >> > 4/ Rename the tarball in $(DL_DIR) to its final name
> >> >
> >> >Steps (3) and (4) are separated so that if $(DL_DIR) and $(BUILD_DIR)
> >> >are in separate filesystems, the rename to the final name remains an
> >> >atomic operation.
> >> >
> >> >And yes, the git download helper from Yann doesn't seem to implement
> >> >this logic (or I got lost with the variable names, which is very
> >> >possible).
> >>
> >> This patch (3/14) only moves the existing logic to a separate script,
> >> just like for the other helpers. Later patches in the same series
> >> improve the logic. In particular, not using DL_DIR as scratchpad is
> >> implemented in patch 11/14 http://patchwork.ozlabs.org/patch/365791/
> >
> > Ah, ok, thanks for the clarification. However that doesn't explain why
> > the temporary repo in $(DL_DIR) is not being removed. That's a
> > regression compared to the original code in the .mk file, no?
>
> Patch 11 does:
>
> -rm -rf "${repodir}"
> +rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
> +exit ${ret}
>
> and
>
> -repodir="${BR2_DL_DIR}/${basename}"
> +repodir="${basename}.tmp-git-checkout"
> +tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
> +tmp_output="$( mktemp "${output}.XXXXXX" )"
>
> So from the code (haven't verified this now) there is no repo at all
> in DL_DIR, so that repo does not need to be removed. The repo $repodir
> _is_ removed, including all of the other temporary stuff tmp_tar and
> tmp_output.
>
> Peter, did you test this with all patches applied?
Well, even if the full series behaves correctly, the patch 3 should not
change the existing behaviour. Any change in behaviour is indeed a
regression, and thus a bug.
But I could not observe any change in behaviour here.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 17:31 ` Thomas De Schampheleire
2014-07-02 20:29 ` Yann E. MORIN
@ 2014-07-02 20:31 ` Thomas Petazzoni
2014-07-02 20:37 ` Yann E. MORIN
1 sibling, 1 reply; 29+ messages in thread
From: Thomas Petazzoni @ 2014-07-02 20:31 UTC (permalink / raw)
To: buildroot
Dear Thomas De Schampheleire,
On Wed, 2 Jul 2014 19:31:08 +0200, Thomas De Schampheleire wrote:
> > Ah, ok, thanks for the clarification. However that doesn't explain why
> > the temporary repo in $(DL_DIR) is not being removed. That's a
> > regression compared to the original code in the .mk file, no?
>
> Patch 11 does:
>
> -rm -rf "${repodir}"
> +rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
> +exit ${ret}
>
> and
>
> -repodir="${BR2_DL_DIR}/${basename}"
> +repodir="${basename}.tmp-git-checkout"
> +tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
> +tmp_output="$( mktemp "${output}.XXXXXX" )"
>
> So from the code (haven't verified this now) there is no repo at all
> in DL_DIR, so that repo does not need to be removed. The repo $repodir
> _is_ removed, including all of the other temporary stuff tmp_tar and
> tmp_output.
>
> Peter, did you test this with all patches applied?
I think what Peter complains about is that with just PATCH 03/14
applied, the Git download method no longer works properly as it leaves
the repo in $(DL_DIR).
The fact that this problem may or may not get solved by a later patch
in the series is more-or-less irrelevant here: Peter is just saying
that the series is not perfectly bisectable, and he stopped reviewing
when he saw an issue in PATCH 03/14.
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 20:31 ` Thomas Petazzoni
@ 2014-07-02 20:37 ` Yann E. MORIN
2014-07-02 21:42 ` Peter Korsgaard
0 siblings, 1 reply; 29+ messages in thread
From: Yann E. MORIN @ 2014-07-02 20:37 UTC (permalink / raw)
To: buildroot
Thomas?, All,
On 2014-07-02 22:31 +0200, Thomas Petazzoni spake thusly:
> On Wed, 2 Jul 2014 19:31:08 +0200, Thomas De Schampheleire wrote:
> > > Ah, ok, thanks for the clarification. However that doesn't explain why
> > > the temporary repo in $(DL_DIR) is not being removed. That's a
> > > regression compared to the original code in the .mk file, no?
> >
> > Patch 11 does:
> >
> > -rm -rf "${repodir}"
> > +rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
> > +exit ${ret}
> >
> > and
> >
> > -repodir="${BR2_DL_DIR}/${basename}"
> > +repodir="${basename}.tmp-git-checkout"
> > +tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
> > +tmp_output="$( mktemp "${output}.XXXXXX" )"
> >
> > So from the code (haven't verified this now) there is no repo at all
> > in DL_DIR, so that repo does not need to be removed. The repo $repodir
> > _is_ removed, including all of the other temporary stuff tmp_tar and
> > tmp_output.
> >
> > Peter, did you test this with all patches applied?
>
> I think what Peter complains about is that with just PATCH 03/14
> applied, the Git download method no longer works properly as it leaves
> the repo in $(DL_DIR).
>
> The fact that this problem may or may not get solved by a later patch
> in the series is more-or-less irrelevant here:
Agreed. If that patch changes the behaviour, it is incorrect.
> Peter is just saying
> that the series is not perfectly bisectable, and he stopped reviewing
> when he saw an issue in PATCH 03/14.
And I do not see this issue...
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 20:37 ` Yann E. MORIN
@ 2014-07-02 21:42 ` Peter Korsgaard
0 siblings, 0 replies; 29+ messages in thread
From: Peter Korsgaard @ 2014-07-02 21:42 UTC (permalink / raw)
To: buildroot
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
Hi,
>> I think what Peter complains about is that with just PATCH 03/14
>> applied, the Git download method no longer works properly as it leaves
>> the repo in $(DL_DIR).
Indeed.
>> The fact that this problem may or may not get solved by a later patch
>> in the series is more-or-less irrelevant here:
> Agreed. If that patch changes the behaviour, it is incorrect.
>> Peter is just saying
>> that the series is not perfectly bisectable, and he stopped reviewing
>> when he saw an issue in PATCH 03/14.
> And I do not see this issue...
Sorry, I got confused. The problem actually turned out to be the use of
pushd/popd with a /bin/sh, on here /bin/sh is dash which doesn't support
it.
The fix is to use !#/bin/bash
--
Bye, Peter Korsgaard
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script
2014-07-02 15:28 ` Peter Korsgaard
2014-07-02 15:44 ` Thomas Petazzoni
@ 2014-07-02 20:25 ` Yann E. MORIN
1 sibling, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-07-02 20:25 UTC (permalink / raw)
To: buildroot
Peter, All,
On 2014-07-02 17:28 +0200, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
>
> > The git download helper is getting a bit more complex. Fixing it in the
> > Makefile when it breaks (like the recent breakage with a non-existing
> > sha1-cset) proves to be challenging, to say the least.
>
> > Move it into a shell script in support/download/git, which will make
> > it much easier to read, maintain, fix and enhance in the future.
>
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Peter Korsgaard <jacmet@uclibc.org>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
> > Reviewed-by: Samuel Martin <s.martin49@gmail.com>
> > Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>
> Thanks. I've verified that the (non-gzip'ed) tarball is identical to
> what we had before, but I noticed that we no longer delete the temporary
> repo in DL_DIR.
>
> Did you do that change on purpose? I don't think we want to keep it, do
> we?
For sure, that's not on-purpose.
Now, I fail to see how the remporary repository would linger after a
successfull download. Here's what the script does:
repodir="${BR2_DL_DIR}/${basename}"
if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
printf "Doing shallow clone\n"
${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${repodir}"
else
printf "Doing full clone\n"
${GIT} clone --bare "${repo}" "${repodir}"
fi
pushd "${repodir}"
${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar
"${cset}"
gzip -c "${output}.tmp" >"${output}"
rm -f "${output}.tmp"
popd
rm -rf "${repodir}"
So, unless there is something I'm missing, the last statement should get
rid of the temp repo...
Lemme try...
Nope, the repository is properly removed here:
$ make libllcp-source
[--SNIP--]
>>> libllcp cf0c4b3c9df98851c6092c130192130c3f5a46bd Downloading
Doing full clone
Cloning into bare repository '/home/ymorin/dev/buildroot/O/test-dl/libllcp-cf0c4b3c9df98851c6092c130192130c3f5a46bd'...
remote: Counting objects: 1202, done.
Receiving objects: 100% (1202/1202), 267.40 KiB | 145.00 KiB/s, done.
Resolving deltas: 100% (812/812), done.
Checking connectivity... done.
~/dev/buildroot/O/test-dl/libllcp-cf0c4b3c9df98851c6092c130192130c3f5a46bd ~/dev/buildroot/buildroot
~/dev/buildroot/buildroot
$ ls -l /home/ymorin/dev/buildroot/O/test-dl/
-rw-rw-r-- 1 ymorin ymorin 52K Jul 2 22:23 libllcp-cf0c4b3c9df98851c6092c130192130c3f5a46bd.tar.gz
So, no lingering clone...
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 04/14] pkg-infra: move the svn download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (2 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 03/14] pkg-infra: move the git download helper to a script Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-07-01 14:07 ` Ryan Barnett
2014-06-30 22:26 ` [Buildroot] [PATCH 05/14] pkg-infra: move the cvs " Yann E. MORIN
` (9 subsequent siblings)
13 siblings, 1 reply; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
---
package/pkg-download.mk | 9 +++------
support/download/svn | 25 +++++++++++++++++++++++++
2 files changed, 28 insertions(+), 6 deletions(-)
create mode 100755 support/download/svn
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index c848f6a..e823f68 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -9,7 +9,7 @@
# Download method commands
WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
-SVN := $(call qstrip,$(BR2_SVN))
+export SVN := $(call qstrip,$(BR2_SVN))
CVS := $(call qstrip,$(BR2_CVS))
BZR := $(call qstrip,$(BR2_BZR))
export GIT := $(call qstrip,$(BR2_GIT))
@@ -133,11 +133,8 @@ endef
define DOWNLOAD_SVN
test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
- (pushd $(DL_DIR) > /dev/null && \
- $(SVN) export $($(PKG)_SITE)@$($(PKG)_DL_VERSION) $($(PKG)_DL_DIR) && \
- $(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \
- rm -rf $($(PKG)_DL_DIR) && \
- popd > /dev/null)
+ $(EXTRA_ENV) support/download/svn $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+ $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
endef
define SOURCE_CHECK_SVN
diff --git a/support/download/svn b/support/download/svn
new file mode 100755
index 0000000..717fad3
--- /dev/null
+++ b/support/download/svn
@@ -0,0 +1,25 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for svn
+# Call it with:
+# $1: svn repo
+# $2: svn revision
+# $3: package's basename (eg. foobar-1.2.3)
+# $4: output file
+# And this environment:
+# SVN : the svn command to call
+# BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+basename="${3}"
+output="${4}"
+
+pushd "${BR2_DL_DIR}"
+${SVN} export "${repo}@${rev}" "${basename}"
+tar czf "${output}" "${basename}"
+rm -rf "${basename}"
+popd
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 04/14] pkg-infra: move the svn download helper to a script
2014-06-30 22:26 ` [Buildroot] [PATCH 04/14] pkg-infra: move the svn " Yann E. MORIN
@ 2014-07-01 14:07 ` Ryan Barnett
0 siblings, 0 replies; 29+ messages in thread
From: Ryan Barnett @ 2014-07-01 14:07 UTC (permalink / raw)
To: buildroot
Yann,
On Mon, Jun 30, 2014 at 5:26 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Maintaining the download helpers in the Makefile has proved to be a bit
> complex, so move it to a shell script.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Reviewed-by: Samuel Martin <s.martin49@gmail.com>
> ---
> package/pkg-download.mk | 9 +++------
> support/download/svn | 25 +++++++++++++++++++++++++
> 2 files changed, 28 insertions(+), 6 deletions(-)
> create mode 100755 support/download/svn
>
> diff --git a/package/pkg-download.mk b/package/pkg-download.mk
> index c848f6a..e823f68 100644
> --- a/package/pkg-download.mk
> +++ b/package/pkg-download.mk
> @@ -9,7 +9,7 @@
>
> # Download method commands
> WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
> -SVN := $(call qstrip,$(BR2_SVN))
> +export SVN := $(call qstrip,$(BR2_SVN))
> CVS := $(call qstrip,$(BR2_CVS))
> BZR := $(call qstrip,$(BR2_BZR))
> export GIT := $(call qstrip,$(BR2_GIT))
> @@ -133,11 +133,8 @@ endef
>
> define DOWNLOAD_SVN
> test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
> - (pushd $(DL_DIR) > /dev/null && \
> - $(SVN) export $($(PKG)_SITE)@$($(PKG)_DL_VERSION) $($(PKG)_DL_DIR) && \
> - $(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \
> - rm -rf $($(PKG)_DL_DIR) && \
> - popd > /dev/null)
> + $(EXTRA_ENV) support/download/svn $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
> + $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
> endef
>
> define SOURCE_CHECK_SVN
> diff --git a/support/download/svn b/support/download/svn
> new file mode 100755
> index 0000000..717fad3
> --- /dev/null
> +++ b/support/download/svn
> @@ -0,0 +1,25 @@
> +#!/bin/sh
> +
> +# We want to catch any command failure, and exit immediately
> +set -e
> +
> +# Download helper for svn
> +# Call it with:
> +# $1: svn repo
> +# $2: svn revision
> +# $3: package's basename (eg. foobar-1.2.3)
> +# $4: output file
> +# And this environment:
> +# SVN : the svn command to call
> +# BR2_DL_DIR: path to Buildroot's download dir
> +
> +repo="${1}"
> +rev="${2}"
> +basename="${3}"
> +output="${4}"
> +
> +pushd "${BR2_DL_DIR}"
> +${SVN} export "${repo}@${rev}" "${basename}"
Glad to see you incorporated this change!
> +tar czf "${output}" "${basename}"
> +rm -rf "${basename}"
> +popd
Reviewed-by: Ryan Barnett <ryan.barnett@rockwellcollins.com>
^ permalink raw reply [flat|nested] 29+ messages in thread
* [Buildroot] [PATCH 05/14] pkg-infra: move the cvs download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (3 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 04/14] pkg-infra: move the svn " Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 06/14] pkg-infra: move the hg " Yann E. MORIN
` (8 subsequent siblings)
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
---
package/pkg-download.mk | 11 ++++-------
support/download/cvs | 27 +++++++++++++++++++++++++++
2 files changed, 31 insertions(+), 7 deletions(-)
create mode 100755 support/download/cvs
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index e823f68..9f3afac 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -10,7 +10,7 @@
# Download method commands
WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
export SVN := $(call qstrip,$(BR2_SVN))
-CVS := $(call qstrip,$(BR2_CVS))
+export CVS := $(call qstrip,$(BR2_CVS))
BZR := $(call qstrip,$(BR2_BZR))
export GIT := $(call qstrip,$(BR2_GIT))
HG := $(call qstrip,$(BR2_HG)) $(QUIET)
@@ -114,12 +114,9 @@ endef
define DOWNLOAD_CVS
test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
- (pushd $(DL_DIR) > /dev/null && \
- $(CVS) -z3 -d:pserver:anonymous@$(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
- co -d $($(PKG)_BASE_NAME) -r :$($(PKG)_DL_VERSION) -P $($(PKG)_RAWNAME) && \
- $(TAR) czf $($(PKG)_SOURCE) $($(PKG)_BASE_NAME)/ && \
- rm -rf $($(PKG)_DL_DIR) && \
- popd > /dev/null)
+ $(EXTRA_ENV) support/download/cvs $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
+ $($(PKG)_DL_VERSION) $($(PKG)_RAWNAME) \
+ $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
endef
# Not all CVS servers support ls/rls, use login to see if we can connect
diff --git a/support/download/cvs b/support/download/cvs
new file mode 100755
index 0000000..06b8647
--- /dev/null
+++ b/support/download/cvs
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for cvs
+# Call it with:
+# $1: cvs repo
+# $2: cvs revision
+# $3: package's name (eg. foobar)
+# $4: package's basename (eg. foobar-1.2.3)
+# $5: output file
+# And this environment:
+# CVS : the cvs command to call
+# BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+rawname="${3}"
+basename="${4}"
+output="${5}"
+
+cd "${BR2_DL_DIR}"
+${CVS} -z3 -d":pserver:anonymous@${repo}" \
+ co -d "${basename}" -r ":${rev}" -P "${rawname}"
+tar czf "${output}" "${basename}"
+rm -rf "${basename}"
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 06/14] pkg-infra: move the hg download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (4 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 05/14] pkg-infra: move the cvs " Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 07/14] pkg-infra: move the wget " Yann E. MORIN
` (7 subsequent siblings)
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail, when the 'hg archive'
step is interrupted, now working fine]
---
package/pkg-download.mk | 11 +++--------
support/download/hg | 25 +++++++++++++++++++++++++
2 files changed, 28 insertions(+), 8 deletions(-)
create mode 100755 support/download/hg
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 9f3afac..3290b6b 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -13,7 +13,7 @@ export SVN := $(call qstrip,$(BR2_SVN))
export CVS := $(call qstrip,$(BR2_CVS))
BZR := $(call qstrip,$(BR2_BZR))
export GIT := $(call qstrip,$(BR2_GIT))
-HG := $(call qstrip,$(BR2_HG)) $(QUIET)
+export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
@@ -161,13 +161,8 @@ endef
define DOWNLOAD_HG
test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
- (pushd $(DL_DIR) > /dev/null && \
- rm -rf $($(PKG)_BASE_NAME) && \
- $(HG) clone --noupdate --rev $($(PKG)_DL_VERSION) $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
- $(HG) archive --repository $($(PKG)_BASE_NAME) --type tgz --prefix $($(PKG)_BASE_NAME)/ \
- --rev $($(PKG)_DL_VERSION) $(DL_DIR)/$($(PKG)_SOURCE) && \
- rm -rf $($(PKG)_DL_DIR) && \
- popd > /dev/null)
+ $(EXTRA_ENV) support/download/hg $($(PKG)_SITE) $($(PKG)_DL_VERSION) \
+ $($(PKG)_BASE_NAME) $(DL_DIR)/$($(PKG)_SOURCE)
endef
# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
diff --git a/support/download/hg b/support/download/hg
new file mode 100755
index 0000000..70b49cf
--- /dev/null
+++ b/support/download/hg
@@ -0,0 +1,25 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for hg
+# Call it with:
+# $1: hg repo
+# $2: hg cset
+# $3: package's basename (eg. foobar-1.2.3)
+# $4: output file
+# And this environment:
+# HG : the hg command to call
+# BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+cset="${2}"
+basename="${3}"
+output="${4}"
+
+cd "${BR2_DL_DIR}"
+${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
+${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
+ --rev "${cset}" "${output}"
+rm -rf "${basename}"
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 07/14] pkg-infra: move the wget download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (5 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 06/14] pkg-infra: move the hg " Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 08/14] pkg-infra: move the bzr " Yann E. MORIN
` (6 subsequent siblings)
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
---
package/pkg-download.mk | 13 +++----------
support/download/wget | 21 +++++++++++++++++++++
2 files changed, 24 insertions(+), 10 deletions(-)
create mode 100755 support/download/wget
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 3290b6b..7554582 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -8,7 +8,7 @@
################################################################################
# Download method commands
-WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
+export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
export SVN := $(call qstrip,$(BR2_SVN))
export CVS := $(call qstrip,$(BR2_CVS))
BZR := $(call qstrip,$(BR2_BZR))
@@ -175,17 +175,10 @@ define SHOW_EXTERNAL_DEPS_HG
echo $($(PKG)_SOURCE)
endef
-# Download a file using wget. Only download the file if it doesn't
-# already exist in the download directory. If the download fails,
-# remove the file (because wget -O creates a 0-byte file even if the
-# download fails). To handle an interrupted download as well, download
-# to a temporary file first. The temporary file will be overwritten
-# the next time the download is tried.
+
define DOWNLOAD_WGET
test -e $(DL_DIR)/$(2) || \
- ($(WGET) -O $(DL_DIR)/$(2).tmp '$(call qstrip,$(1))' && \
- mv $(DL_DIR)/$(2).tmp $(DL_DIR)/$(2)) || \
- (rm -f $(DL_DIR)/$(2).tmp ; exit 1)
+ $(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2)
endef
define SOURCE_CHECK_WGET
diff --git a/support/download/wget b/support/download/wget
new file mode 100755
index 0000000..7865517
--- /dev/null
+++ b/support/download/wget
@@ -0,0 +1,21 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for wget
+# Call it with:
+# $1: URL
+# $2: output file
+# And this environment:
+# WGET : the wget command to call
+
+url="${1}"
+output="${2}"
+
+if ${WGET} -O "${output}.tmp" "${url}"; then
+ mv "${output}.tmp" "${output}"
+else
+ rm -f "${output}.tmp"
+ exit 1
+fi
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 08/14] pkg-infra: move the bzr download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (6 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 07/14] pkg-infra: move the wget " Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 09/14] pkg-infra: move the scp " Yann E. MORIN
` (5 subsequent siblings)
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
package/pkg-download.mk | 4 ++--
support/download/bzr | 19 +++++++++++++++++++
2 files changed, 21 insertions(+), 2 deletions(-)
create mode 100755 support/download/bzr
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 7554582..e420b05 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -11,7 +11,7 @@
export WGET := $(call qstrip,$(BR2_WGET)) $(QUIET)
export SVN := $(call qstrip,$(BR2_SVN))
export CVS := $(call qstrip,$(BR2_CVS))
-BZR := $(call qstrip,$(BR2_BZR))
+export BZR := $(call qstrip,$(BR2_BZR))
export GIT := $(call qstrip,$(BR2_GIT))
export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
@@ -101,7 +101,7 @@ endef
define DOWNLOAD_BZR
test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
- $(BZR) export $(DL_DIR)/$($(PKG)_SOURCE) $($(PKG)_SITE) -r $($(PKG)_DL_VERSION)
+ $(EXTRA_ENV) support/download/bzr $($(PKG)_SITE) $($(PKG)_DL_VERSION) $(DL_DIR)/$($(PKG)_SOURCE)
endef
define SOURCE_CHECK_BZR
diff --git a/support/download/bzr b/support/download/bzr
new file mode 100755
index 0000000..68121f4
--- /dev/null
+++ b/support/download/bzr
@@ -0,0 +1,19 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for bzr
+# Call it with:
+# $1: bzr repo
+# $2: bzr revision
+# $3: output file
+# And this environment:
+# BZR : the bzr command to call
+# BR2_DL_DIR: path to Buildroot's download dir
+
+repo="${1}"
+rev="${2}"
+output="${3}"
+
+${BZR} export "${output}" "${repo}" -r "${rev}"
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 09/14] pkg-infra: move the scp download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (7 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 08/14] pkg-infra: move the bzr " Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 10/14] pkg-infra: move the cp (aka localfiles) " Yann E. MORIN
` (4 subsequent siblings)
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
package/pkg-download.mk | 5 +++--
support/download/scp | 16 ++++++++++++++++
2 files changed, 19 insertions(+), 2 deletions(-)
create mode 100755 support/download/scp
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index e420b05..eec91b8 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -14,7 +14,7 @@ export CVS := $(call qstrip,$(BR2_CVS))
export BZR := $(call qstrip,$(BR2_BZR))
export GIT := $(call qstrip,$(BR2_GIT))
export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
-SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
+export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
@@ -147,7 +147,8 @@ endef
# to prepend the path with a slash: scp://[user@]host:/absolutepath
define DOWNLOAD_SCP
test -e $(DL_DIR)/$(2) || \
- $(SCP) '$(call stripurischeme,$(call qstrip,$(1)))' $(DL_DIR)/$(2)
+ $(EXTRA_ENV) support/download/scp '$(call stripurischeme,$(call qstrip,$(1)))' \
+ $(DL_DIR)/$(2)
endef
define SOURCE_CHECK_SCP
diff --git a/support/download/scp b/support/download/scp
new file mode 100755
index 0000000..9f8a4f0
--- /dev/null
+++ b/support/download/scp
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for scp
+# Call it with:
+# $1: URL
+# $2: output file
+# And this environment:
+# SCP : the scp command to call
+
+url="${1}"
+output="${2}"
+
+${SCP} "${url}" "${output}"
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 10/14] pkg-infra: move the cp (aka localfiles) download helper to a script
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (8 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 09/14] pkg-infra: move the scp " Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 11/14] pkg-infra: don't use DL_DIR as scratchpad for temporary downloads Yann E. MORIN
` (3 subsequent siblings)
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Maintaining the download helpers in the Makefile has proved to be a bit
complex, so move it to a shell script.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
package/pkg-download.mk | 5 +++--
support/download/cp | 16 ++++++++++++++++
2 files changed, 19 insertions(+), 2 deletions(-)
create mode 100755 support/download/cp
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index eec91b8..d3cd0c1 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -16,7 +16,7 @@ export GIT := $(call qstrip,$(BR2_GIT))
export HG := $(call qstrip,$(BR2_HG)) $(QUIET)
export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET)
SSH := $(call qstrip,$(BR2_SSH)) $(QUIET)
-LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
+export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
# Default spider mode is 'DOWNLOAD'. Other possible values are 'SOURCE_CHECK'
# used by the _source-check target and 'SHOW_EXTERNAL_DEPS', used by the
@@ -192,7 +192,8 @@ endef
define DOWNLOAD_LOCALFILES
test -e $(DL_DIR)/$(2) || \
- $(LOCALFILES) $(call stripurischeme,$(call qstrip,$(1))) $(DL_DIR)
+ $(EXTRA_ENV) support/download/cp $(call stripurischeme,$(call qstrip,$(1))) \
+ $(DL_DIR)
endef
define SOURCE_CHECK_LOCALFILES
diff --git a/support/download/cp b/support/download/cp
new file mode 100755
index 0000000..d820846
--- /dev/null
+++ b/support/download/cp
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+# We want to catch any command failure, and exit immediately
+set -e
+
+# Download helper for cp
+# Call it with:
+# $1: source file
+# $2: output file
+# And this environment:
+# LOCALFILES: the cp command to call
+
+source="${1}"
+output="${2}"
+
+${LOCALFILES} "${source}" "${output}"
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 11/14] pkg-infra: don't use DL_DIR as scratchpad for temporary downloads
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (9 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 10/14] pkg-infra: move the cp (aka localfiles) " Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 12/14] pkg-infra: add possiblity to check downloaded files against known hashes Yann E. MORIN
` (2 subsequent siblings)
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
DL_DIR can be a very precious place for some users: they use it to store
all the downloaded archives to share across all their Buildroot (and
maybe non-Buildroot) builds.
We do not want to trash this location with our temporary downloads (e.g.
git, Hg, svn, cvs repository clones/checkouts, or wget, bzr tep tarballs).
Turns out that we already have some kind of scratchpad, the BUILD_DIR.
Although it is not really a disposable location, that's the best we have
so far.
Also, we create the temporary tarballs with mktemp using the final tarrball,
as template, since we want the temporary to be on the same filesystem as
the final location, so the 'mv' is just a plain, atomic rename(2), and we
are not left with a half-copied file as the final location.
Using mktemp ensures all temp file names are unique, so it allows for
parallel downloads from different build dirs at the same time, without
cloberring each downloads.
Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared
locations, sometime with little place (eg. tmpfs), and some of the
repositories we clone/checkout can be very big.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Tested-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
[tested a particular scenario that used to fail: two separate builds
using a shared DL_DIR, ccache enabled, so that they run almost
synchronously. These would download the same file at the same time,
corrupting each other. With the patches in this series, all works
fine.]
---
Change v7 -> v8:
- use a cleaner and saner code path to detect errors
- convert the cp and scp helpers, too (Thomas DS)
Changes v6 -> v7:
- drop leftovers from a previous attempt in bzr (Thomas)
Changes v5 -> v6:
- ensure we can do parallel downloads from multiple build-dirs (Thomas)
Changes v4 -> v5
- explain why we create the temp files where we create them (Arnout)
Changes v3 -> v4:
- remove spurious bump in package/fis (Samuel)
---
support/download/bzr | 25 ++++++++++++++++++++++---
support/download/cp | 12 +++++++++++-
support/download/cvs | 34 +++++++++++++++++++++++++++-------
support/download/git | 33 ++++++++++++++++++++++++++-------
support/download/hg | 35 ++++++++++++++++++++++++++++-------
support/download/scp | 14 +++++++++++++-
support/download/svn | 33 ++++++++++++++++++++++++++-------
support/download/wget | 26 ++++++++++++++++++++------
8 files changed, 173 insertions(+), 39 deletions(-)
diff --git a/support/download/bzr b/support/download/bzr
index 68121f4..2a927d6 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -9,11 +9,30 @@ set -e
# $2: bzr revision
# $3: output file
# And this environment:
-# BZR : the bzr command to call
-# BR2_DL_DIR: path to Buildroot's download dir
+# BZR : the bzr command to call
+# BUILD_DIR: path to Buildroot's build dir
repo="${1}"
rev="${2}"
output="${3}"
-${BZR} export "${output}" "${repo}" -r "${rev}"
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - the we move to a temp file in the final location, so it is
+# on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${BZR} export "${tmp_dl}" "${repo}" -r "${rev}"; then
+ if mv "${tmp_dl}" "${tmp_output}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+ fi
+fi
+
+# Cleanup
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/cp b/support/download/cp
index d820846..eeb5856 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -13,4 +13,14 @@ set -e
source="${1}"
output="${2}"
-${LOCALFILES} "${source}" "${output}"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+ret=1
+if ${LOCALFILES} "${source}" "${tmp_output}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+fi
+
+# Cleanup
+rm -f "${tmp_output}"
+exit ${ret}
diff --git a/support/download/cvs b/support/download/cvs
index 06b8647..ee32958 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -11,8 +11,8 @@ set -e
# $4: package's basename (eg. foobar-1.2.3)
# $5: output file
# And this environment:
-# CVS : the cvs command to call
-# BR2_DL_DIR: path to Buildroot's download dir
+# CVS : the cvs command to call
+# BUILD_DIR: path to Buildroot's build dir
repo="${1}"
rev="${2}"
@@ -20,8 +20,28 @@ rawname="${3}"
basename="${4}"
output="${5}"
-cd "${BR2_DL_DIR}"
-${CVS} -z3 -d":pserver:anonymous@${repo}" \
- co -d "${basename}" -r ":${rev}" -P "${rawname}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
+repodir="${basename}.tmp-cvs-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+# on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${CVS} -z3 -d":pserver:anonymous@${repo}" \
+ co -d "${repodir}" -r ":${rev}" -P "${rawname}"; then
+ if tar czf "${tmp_output}" "${repodir}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+ fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/git b/support/download/git
index 96db3a9..79ce15e 100755
--- a/support/download/git
+++ b/support/download/git
@@ -10,15 +10,28 @@ set -e
# $3: package's basename (eg. foobar-1.2.3)
# $4: output file
# And this environment:
-# BR2_DL_DIR: path to Buildroot's download dir
-# GIT : the git command to call
+# GIT : the git command to call
+# BUILD_DIR: path to Buildroot's build dir
repo="${1}"
cset="${2}"
basename="${3}"
output="${4}"
-repodir="${BR2_DL_DIR}/${basename}"
+repodir="${basename}.tmp-git-checkout"
+tmp_tar="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create the uncomporessed tarball in tht same trashable location
+# - then we create a temporary compressed tarball in the final location, so
+# it is on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then
printf "Doing shallow clone\n"
@@ -28,10 +41,16 @@ else
${GIT} clone --bare "${repo}" "${repodir}"
fi
+ret=1
pushd "${repodir}"
-${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}"
-gzip -c "${output}.tmp" >"${output}"
-rm -f "${output}.tmp"
+if ${GIT} archive --prefix="${basename}/" -o "${tmp_tar}" \
+ --format=tar "${cset}"; then
+ if gzip -c "${tmp_tar}" >"${tmp_output}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+ fi
+fi
popd
-rm -rf "${repodir}"
+rm -rf "${repodir}" "${tmp_tar}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/hg b/support/download/hg
index 70b49cf..1568ff0 100755
--- a/support/download/hg
+++ b/support/download/hg
@@ -10,16 +10,37 @@ set -e
# $3: package's basename (eg. foobar-1.2.3)
# $4: output file
# And this environment:
-# HG : the hg command to call
-# BR2_DL_DIR: path to Buildroot's download dir
+# HG : the hg command to call
+# BUILD_DIR: path to Buildroot's build dir
repo="${1}"
cset="${2}"
basename="${3}"
output="${4}"
-cd "${BR2_DL_DIR}"
-${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}"
-${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \
- --rev "${cset}" "${output}"
-rm -rf "${basename}"
+repodir="${basename}.tmp-hg-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+# on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}"; then
+ if ${HG} archive --repository "${repodir}" --type tgz \
+ --prefix "${basename}" --rev "${cset}" \
+ "${tmp_output}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+ fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/scp b/support/download/scp
index 9f8a4f0..e9ca013 100755
--- a/support/download/scp
+++ b/support/download/scp
@@ -12,5 +12,17 @@ set -e
url="${1}"
output="${2}"
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
-${SCP} "${url}" "${output}"
+ret=1
+if ${SCP} "${url}" "${tmp_dl}"; then
+ if mv "${tmp_dl}" "${tmp_output}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+ fi
+fi
+
+# Cleanup
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/svn b/support/download/svn
index 717fad3..258e676 100755
--- a/support/download/svn
+++ b/support/download/svn
@@ -10,16 +10,35 @@ set -e
# $3: package's basename (eg. foobar-1.2.3)
# $4: output file
# And this environment:
-# SVN : the svn command to call
-# BR2_DL_DIR: path to Buildroot's download dir
+# SVN : the svn command to call
+# BUILD_DIR: path to Buildroot's build dir
repo="${1}"
rev="${2}"
basename="${3}"
output="${4}"
-pushd "${BR2_DL_DIR}"
-${SVN} export "${repo}@${rev}" "${basename}"
-tar czf "${output}" "${basename}"
-rm -rf "${basename}"
-popd
+repodir="${basename}.tmp-svn-checkout"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+cd "${BUILD_DIR}"
+# Remove leftovers from a previous failed run
+rm -rf "${repodir}"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we create a temporary tarball in the final location, so it is
+# on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${SVN} export "${repo}@${rev}" "${repodir}"; then
+ if tar czf "${tmp_output}" "${repodir}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+ fi
+fi
+
+# Cleanup
+rm -rf "${repodir}" "${tmp_output}"
+exit ${ret}
diff --git a/support/download/wget b/support/download/wget
index 7865517..f1a8f92 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -8,14 +8,28 @@ set -e
# $1: URL
# $2: output file
# And this environment:
-# WGET : the wget command to call
+# WGET : the wget command to call
+# BUILD_DIR: path to Buildroot's build dir
url="${1}"
output="${2}"
-if ${WGET} -O "${output}.tmp" "${url}"; then
- mv "${output}.tmp" "${output}"
-else
- rm -f "${output}.tmp"
- exit 1
+tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )"
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# Play tic-tac-toe with temp files
+# - first, we download to a trashable location (the build-dir)
+# - then we copy to a temporary tarball in the final location, so it is
+# on the same filesystem as the final file
+# - finally, we atomically rename to the final file
+
+ret=1
+if ${WGET} -O "${tmp_dl}" "${url}"; then
+ if mv "${tmp_dl}" "${tmp_output}"; then
+ mv "${tmp_output}" "${output}"
+ ret=0
+ fi
fi
+
+rm -f "${tmp_dl}" "${tmp_output}"
+exit ${ret}
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 12/14] pkg-infra: add possiblity to check downloaded files against known hashes
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (10 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 11/14] pkg-infra: don't use DL_DIR as scratchpad for temporary downloads Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 13/14] manual: add documentation about packages' hashes Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 14/14] package/ca-certificates: add tarball's hashes Yann E. MORIN
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Some of the packages that Buildroot might build are sensitive packages,
related to security: openssl, dropbear, ca-certificates...
Some of those packages are downloaded over plain http, because there is
no way to get them over a secure channel, such as https.
In these dark times of pervasive surveillance, the potential for harm that
a tampered-with package could generate, we may want to check the integrity
of those sensitive packages.
So, each package may now provide a list of hashes for all files that needs
to be downloaded, and Buildroot will just fail if any downloaded file does
not match its known hash, in which case it is removed.
Hashes can be any of the md5, sha1 or sha2 variants, and will be checked
even if the file was pre-downloaded.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Gustavo Zacarias <gustavo@zacarias.com.ar>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
---
Changes v7 -> v8
- expand MITM to its full-length meaning (Thomas DS)
- typo (Thomas DS)
Changes v4 -> v5:
- fix detection of comments and empty lines
---
Note: this is not a bullet-proof solution, since Buildroot may itself be
compromised. But since we do sign our releases, then we secure the list of
hashes at the same time. Only random snapshots from the repository may be
at risk of tampering, although this is highly doubtfull, given how git
stores its data.
---
package/pkg-download.mk | 20 ++++++++++--
support/download/check-hash | 76 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 93 insertions(+), 3 deletions(-)
create mode 100755 support/download/check-hash
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index d3cd0c1..7f208d5 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -58,6 +58,17 @@ domainseparator=$(if $(1),$(1),/)
# github(user,package,version): returns site of GitHub repository
github = https://github.com/$(1)/$(2)/archive/$(3)
+# Helper for checking a tarball's checksum
+# If the hash does not match, remove the incorrect file
+# $(1): the path to the file with the hashes
+# $(2): the full path to the file to check
+define VERIFY_HASH
+ if ! support/download/check-hash $(1) $(2); then \
+ rm -f $(2); \
+ exit 1; \
+ fi
+endef
+
################################################################################
# The DOWNLOAD_* helpers are in charge of getting a working copy
# of the source repository for their corresponding SCM,
@@ -148,7 +159,8 @@ endef
define DOWNLOAD_SCP
test -e $(DL_DIR)/$(2) || \
$(EXTRA_ENV) support/download/scp '$(call stripurischeme,$(call qstrip,$(1)))' \
- $(DL_DIR)/$(2)
+ $(DL_DIR)/$(2) && \
+ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
endef
define SOURCE_CHECK_SCP
@@ -179,7 +191,8 @@ endef
define DOWNLOAD_WGET
test -e $(DL_DIR)/$(2) || \
- $(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2)
+ $(EXTRA_ENV) support/download/wget '$(call qstrip,$(1))' $(DL_DIR)/$(2) && \
+ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
endef
define SOURCE_CHECK_WGET
@@ -193,7 +206,8 @@ endef
define DOWNLOAD_LOCALFILES
test -e $(DL_DIR)/$(2) || \
$(EXTRA_ENV) support/download/cp $(call stripurischeme,$(call qstrip,$(1))) \
- $(DL_DIR)
+ $(DL_DIR) && \
+ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_NAME).hash,$(DL_DIR)/$(2))
endef
define SOURCE_CHECK_LOCALFILES
diff --git a/support/download/check-hash b/support/download/check-hash
new file mode 100755
index 0000000..b13aad9
--- /dev/null
+++ b/support/download/check-hash
@@ -0,0 +1,76 @@
+#!/bin/sh
+set -e
+
+# Helper to check a file matches its known hash
+# Call it with:
+# $1: the full path to the file to check
+# $2: the path of the file containing all the the expected hashes
+
+h_file="${1}"
+file="${2}"
+
+# Does the hash-file exist?
+if [ ! -f "${h_file}" ]; then
+ exit 0
+fi
+
+# Check one hash for a file
+# $1: known hash
+# $2: file (full path)
+check_one_hash() {
+ _h="${1}"
+ _known="${2}"
+ _file="${3}"
+
+ # Note: sha3 is not supported, since there is currently no implementation
+ # (the NIST has yet to publish the parameters).
+ case "${_h}" in
+ md5|sha1) ;;
+ sha224|sha256|sha384|sha512) ;;
+ *) # Unknown hash, exit with error
+ printf "ERROR: unknown hash '%s' for '%s'\n" \
+ "${_h}" "${_file##*/}" >&2
+ exit 1
+ ;;
+ esac
+
+ # Do the hashes match?
+ _hash=$( ${_h}sum "${_file}" |cut -d ' ' -f 1 )
+ if [ "${_hash}" = "${_known}" ]; then
+ printf "%s: OK (%s: %s)\n" "${_file##*/}" "${_h}" "${_hash}"
+ return 0
+ fi
+
+ printf "ERROR: %s has wrong %s hash:\n" "${_file##*/}" "${_h}" >&2
+ printf "ERROR: expected: %s\n" "${_known}" >&2
+ printf "ERROR: got : %s\n" "${_hash}" >&2
+ printf "ERROR: Incomplete download, or man-in-the-middle (MITM) attack\n" >&2
+
+ exit 1
+}
+
+# Do we know one or more hashes for that file?
+nb_checks=0
+while read t h f; do
+ case "${t}" in
+ ''|'#'*)
+ # Skip comments and empty lines
+ continue
+ ;;
+ *)
+ if [ "${f}" = "${file##*/}" ]; then
+ check_one_hash "${t}" "${h}" "${file}"
+ : $((nb_checks++))
+ fi
+ ;;
+ esac
+done <"${h_file}"
+
+if [ ${nb_checks} -eq 0 ]; then
+ if [ -n "${BR2_ENFORCE_CHECK_HASH}" ]; then
+ printf "ERROR: No hash found for %s\n" "${file}" >&2
+ exit 1
+ else
+ printf "WARNING: No hash found for %s\n" "${file}" >&2
+ fi
+fi
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 13/14] manual: add documentation about packages' hashes
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (11 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 12/14] pkg-infra: add possiblity to check downloaded files against known hashes Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
2014-06-30 22:26 ` [Buildroot] [PATCH 14/14] package/ca-certificates: add tarball's hashes Yann E. MORIN
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
Although md5 is, for legacy reasons, a supported hash type,
it is not documented on purpose, since it is now known to
be weak.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Baruch Siach <baruch@tkos.co.il>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Reviewed-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
---
Changes v6 -> v7:
- use single-line title (Samuel)
---
docs/manual/adding-packages-directory.txt | 66 +++++++++++++++++++++++++++++++
support/download/check-hash | 1 +
2 files changed, 67 insertions(+)
diff --git a/docs/manual/adding-packages-directory.txt b/docs/manual/adding-packages-directory.txt
index 69b012c..c2f0a1a 100644
--- a/docs/manual/adding-packages-directory.txt
+++ b/docs/manual/adding-packages-directory.txt
@@ -357,3 +357,69 @@ different way, using different infrastructures:
Further formatting details: see xref:writing-rules-mk[the writing
rules].
+
+[[adding-packages-hash]]
+=== The +.hash+ file
+
+Optionally, you can add a third file, named +libfoo.hash+, that contains
+the hashes of the downloaded files for the +libfoo+ package.
+
+The hashes stored in that file are used to validate the integrity of the
+downloaded files.
+
+The format of this file is one line for each file for which to check the
+hash, each line being space-separated, with these three fields:
+
+* the type of hash, one of:
+** +sha1+, +sha224+, +sha256+, +sha384+, +sha512+
+* the hash of the file:
+** for +sha1+, 40 hexadecimal characters
+** for +sha224+, 56 hexadecimal characters
+** for +sha256+, 64 hexadecimal characters
+** for +sha384+, 96 hexadecimal characters
+** for +sha512+, 128 hexadecimal characters
+* the name of the file, without any directory component
+
+Lines starting with a +#+ sign are considered comments, and ignored. Empty
+lines are ignored.
+
+There can be more than one hash for a single file, each on its own line. In
+this case, all hashes must match.
+
+Ideally, the hashes stored in this file should match the hashes published by
+upstream, e.g. on their website, in the e-mail announcement... If upstream
+provides more than one type of hash (say, +sha1+ and +sha512+), then it is
+best to add all those hashes in the +.hash+ file. If upstream does not
+provide any hash, then compute at least one yourself, and mention this in a
+comment line above the hashes.
+
+*Note:* the number of spaces does not matter, so one can use spaces to
+properly align the different fields.
+
+The example below defines a +sha1+ and a +sha256+ published by upstream for
+the main +libfoo-1.2.3.tar.bz2+ tarball, plus two locally-computed hashes,
+a +sha256+ for a downloaded patch, and a +sha1+ for a downloaded binary blob:
+
+----
+# Hashes from: http://www.foosoftware.org/download/libfoo-1.2.3.tar.bz2.{sha1,sha256}:
+sha1 486fb55c3efa71148fe07895fd713ea3a5ae343a libfoo-1.2.3.tar.bz2
+sha256 efc8103cc3bcb06bda6a781532d12701eb081ad83e8f90004b39ab81b65d4369 libfoo-1.2.3.tar.bz2
+
+# No upstream hashes for the following:
+sha256 ff52101fb90bbfc3fe9475e425688c660f46216d7e751c4bbdb1dc85cdccacb9 libfoo-fix-blabla.patch
+sha1 2d608f3c318c6b7557d551a5a09314f03452f1a1 libfoo-data.bin
+----
+
+If the +.hash+ file is present, and it contains one or more hashes for a
+downloaded file, the hash(es) computed by Buildroot (after download) must
+match the hash(es) stored in the +.hash+ file. If one or more hashes do
+not match, Buildroot considers this an error, deletes the downloaded file,
+and aborts.
+
+If the +.hash+ file is present, but it does not contain a hash for a
+downloaded file, no check is done for that file. If you set the
+environment variable +BR2_ENFORCE_CHECK_HASH+ to a non-empty value, and
+there is no hash for a downloaded file, Buildroot considers this an
+error, deletes the downloaded file, and aborts.
+
+If the +.hash+ file is missing, then no check is done at all.
diff --git a/support/download/check-hash b/support/download/check-hash
index b13aad9..07799bb 100755
--- a/support/download/check-hash
+++ b/support/download/check-hash
@@ -22,6 +22,7 @@ check_one_hash() {
_known="${2}"
_file="${3}"
+ # Note: md5 is supported, but undocumented on purpose.
# Note: sha3 is not supported, since there is currently no implementation
# (the NIST has yet to publish the parameters).
case "${_h}" in
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread* [Buildroot] [PATCH 14/14] package/ca-certificates: add tarball's hashes
2014-06-30 22:26 [Buildroot] [PATCH 0/14 v9] Some download-related changes (branch yem/check-downloads) Yann E. MORIN
` (12 preceding siblings ...)
2014-06-30 22:26 ` [Buildroot] [PATCH 13/14] manual: add documentation about packages' hashes Yann E. MORIN
@ 2014-06-30 22:26 ` Yann E. MORIN
13 siblings, 0 replies; 29+ messages in thread
From: Yann E. MORIN @ 2014-06-30 22:26 UTC (permalink / raw)
To: buildroot
ca-certificates contains sensitive security-related information,
and we want to ensure the archive that we download has not been
compromised.
Add the sha1 and sha256 hashes from Debian's packaging.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Martin Bark <martin@barkynet.com>
Reviewed-by: Samuel Martin <s.martin49@gmail.com>
---
Changes v4 -> v5:
- update hashes since we've bumped the version
---
package/ca-certificates/ca-certificates.hash | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 package/ca-certificates/ca-certificates.hash
diff --git a/package/ca-certificates/ca-certificates.hash b/package/ca-certificates/ca-certificates.hash
new file mode 100644
index 0000000..bcd0723
--- /dev/null
+++ b/package/ca-certificates/ca-certificates.hash
@@ -0,0 +1,3 @@
+# hashes from: $(CA_CERTIFICATES_SITE)/ca-certificates_$(CA_CERTIFICATES_VERSION).dsc :
+sha1 ad57a45f0422fafd78a2e8191e5204f2306cc91b ca-certificates_20140223.tar.xz
+sha256 815b7cd97200b0d76450bb3e7d9b65997ac494ab6467b17369f65b2ef94bcb0c ca-certificates_20140223.tar.xz
--
1.9.1
^ permalink raw reply related [flat|nested] 29+ messages in thread