[Buildroot] [PATCH] Makefile.package.in: GIT: add support to fetch git submodules

[Buildroot] [PATCH] Makefile.package.in: GIT: add support to fetch git submodules

[Alexandre Pereira da Silva]

Signed-off-by: Alexandre Pereira da Silva <aletes.xgr@gmail.com>
---
 package/Makefile.package.in |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index ea44c6c..863e866 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -147,13 +147,15 @@ domainseparator=$(if $(1),$(1),/)
 define DOWNLOAD_GIT
 	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
 	(pushd $(DL_DIR) > /dev/null && \
-	$(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
+	$(GIT) clone -n $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
 	pushd $($(PKG)_BASE_NAME) > /dev/null && \
-	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
-		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
+	$(GIT) checkout $($(PKG)_VERSION) && \
+	$(GIT) submodule init && \
+	$(GIT) submodule update && \
+	$(TAR) czf $(DL_DIR)/$($(PKG)_SOURCE) . &&\
 	popd > /dev/null && \
 	rm -rf $($(PKG)_DL_DIR) && \
-	popd > /dev/null)
+	popd > /dev/null)    
 endef
 
 # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
-- 
1.7.9

[Buildroot] [PATCH] Makefile.package.in: GIT: add support to fetch git submodules

[Arnout Vandecappelle]

On Tuesday 28 February 2012 13:25:19 Alexandre Pereira da Silva wrote:
> Signed-off-by: Alexandre Pereira da Silva <aletes.xgr@gmail.com>
> ---
>  package/Makefile.package.in |   10 ++++++----
>  1 files changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/package/Makefile.package.in b/package/Makefile.package.in
> index ea44c6c..863e866 100644
> --- a/package/Makefile.package.in
> +++ b/package/Makefile.package.in
> @@ -147,13 +147,15 @@ domainseparator=$(if $(1),$(1),/)
>  define DOWNLOAD_GIT
>  	test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
>  	(pushd $(DL_DIR) > /dev/null && \
> -	$(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
> +	$(GIT) clone -n $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
>  	pushd $($(PKG)_BASE_NAME) > /dev/null && \
> -	$(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
> -		gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
> +	$(GIT) checkout $($(PKG)_VERSION) && \
> +	$(GIT) submodule init && \
> +	$(GIT) submodule update && \
> +	$(TAR) czf $(DL_DIR)/$($(PKG)_SOURCE) . &&\
>  	popd > /dev/null && \
>  	rm -rf $($(PKG)_DL_DIR) && \
> -	popd > /dev/null)
> +	popd > /dev/null)    
>  endef
>  
>  # TODO: improve to check that the given PKG_DL_VERSION exists on the remote

 I'm a bit concerned with the impact this will have on performance, e.g.
when cloning a linux git.  Could you evaluate that?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH] Makefile.package.in: GIT: add support to fetch git submodules

[Alexandre Pereira da Silva]

> ?I'm a bit concerned with the impact this will have on performance, e.g.
> when cloning a linux git. ?Could you evaluate that?
>
> ?Regards,
> ?Arnout

Arnout,

I just saw a problem in this first implementation. I am storing not
only the checkout but the git repository as well. I will exclude those
files and resend the patch.

I did some tests, with the optimization that I will implement. Here
are the figures comparing the time needed to checkout a git repository
versus the time to just archive it, excluding any network overhead:
rm -rf -- *; rm -f ../linux.tar.gz; time (git checkout -f master &&
tar czf ../linux.tar.gz --exclude-vcs . && git submodule init && git
submodule update); rm -f ../linux.tar.gz; time git archive -o
../linux.tar.gz master
Checking out files: 100% (37083/37083), done.
Already on 'master'

real	1m24.828s
user	1m9.588s
sys	0m13.233s

real	1m0.601s
user	1m7.676s
sys	0m1.904s

So, there is a 24s overhead or 40% more time needed to checkout the copy.

We could avoid this extra delay if we check for the presence of any
git submodule. If this kind of delay in the download phase is
important I can go in that direction.

Regards

[Buildroot] [PATCH] Makefile.package.in: GIT: add support to fetch git submodules

[Frank Hunleth]

Hi Alexandre and all,

On Tue, Feb 28, 2012 at 6:54 PM, Alexandre Pereira da Silva
<aletes.xgr@gmail.com> wrote:
>>  I'm a bit concerned with the impact this will have on performance, e.g.
>> when cloning a linux git.  Could you evaluate that?
>>
>>  Regards,
>>  Arnout
>
> I just saw a problem in this first implementation. I am storing not
> only the checkout but the git repository as well. I will exclude those
> files and resend the patch.

I was wondering if there had been any progress or further thoughts on
adding support for fetching git submodules. I just ran into a
situation where your patch would have been convenient.

Thanks,
Frank