[Buildroot] [PATCH] pkg-download: Make a shallow clone for git downloads

[Buildroot] [PATCH] pkg-download: Make a shallow clone for git downloads

[Arnout Vandecappelle]

When downloading from git, we clone the whole repository and then only
use the latest commit.  That's a lot of redundant stuff.  So instead,
make a shallow clone.  Unfortunately that's only possible when
downloading a branch or tag, so fall back to the old method if git gives
an error.

This speeds up the cloning of a linux git from more than 2 hours to
20 minutes on a 200KB/s link).

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

---
 package/pkg-download.mk |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index fa9a3e8..6c57244 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -73,10 +73,13 @@ domainseparator=$(if $(1),$(1),/)
 # "external dependencies" of a given build configuration.
 ################################################################################
 
+# Try a shallow clone - but that only works if the version is a ref (tag or
+# branch). Fall back on a full clone if it's a generic sha1.
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
 	(pushd $(DL_DIR) > /dev/null && \
-	$(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
+	 ($(GIT) clone --depth 1 -b $($(PKG)_DL_VERSION) --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) || \
+	  $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME)) && \
 	pushd $($(PKG)_BASE_NAME) > /dev/null && \
 	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
 		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
-- 
tg: (29266a3..) t/git-clone-shallow (depends on: master)

[Buildroot] [PATCH] pkg-download: Make a shallow clone for git downloads

[Luca Ceresoli]

Arnout Vandecappelle (Essensium/Mind) wrote:
> When downloading from git, we clone the whole repository and then only
> use the latest commit.  That's a lot of redundant stuff.  So instead,
> make a shallow clone.  Unfortunately that's only possible when
> downloading a branch or tag, so fall back to the old method if git gives
> an error.
>
> This speeds up the cloning of a linux git from more than 2 hours to
> 20 minutes on a 200KB/s link).
>
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Acked-by: Luca Ceresoli <luca@lucaceresoli.net>

But as discussed during the Buildroot Developer Days, the git
downloader is growing in size and it's probably worth moving it to a
helper script.

Luca

[Buildroot] [PATCH] pkg-download: Make a shallow clone for git downloads

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> writes:

 Arnout> When downloading from git, we clone the whole repository and then only
 Arnout> use the latest commit.  That's a lot of redundant stuff.  So instead,
 Arnout> make a shallow clone.  Unfortunately that's only possible when
 Arnout> downloading a branch or tag, so fall back to the old method if git gives
 Arnout> an error.

 Arnout> This speeds up the cloning of a linux git from more than 2 hours to
 Arnout> 20 minutes on a 200KB/s link).

And just think what difference it will make on the hotel Grumps
connection ;)

Committed, thanks.

-- 
Bye, Peter Korsgaard