* [Buildroot] [PATCH] Improved git download method performance.
@ 2012-01-06 22:56 Theodore Witkamp
2012-01-06 23:40 ` Michael S. Zick
2012-01-15 18:45 ` Arnout Vandecappelle
0 siblings, 2 replies; 4+ messages in thread
From: Theodore Witkamp @ 2012-01-06 22:56 UTC (permalink / raw)
To: buildroot
Instead of cloning the a complete repository, git archive is used first.
If git archive fails because the remote host does not support it, git clone is
used with with --depth 1 so only minimal history is copied.
Also the $($(PKG)_SOURCE) default is changed to .tar instead of .tar.gz.
This removes unneeded compression and decompression.
Relative file paths are now relative to $(TOPDIR) not $(DL_DIR).
---
package/Makefile.package.in | 35 ++++++++++++++++++++---------------
1 files changed, 20 insertions(+), 15 deletions(-)
diff --git a/package/Makefile.package.in b/package/Makefile.package.in
index e85eb15..14f1ba1 100644
--- a/package/Makefile.package.in
+++ b/package/Makefile.package.in
@@ -146,14 +146,14 @@ domainseparator=$(if $(1),$(1),/)
define DOWNLOAD_GIT
test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
- (pushd $(DL_DIR) > /dev/null && \
- $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
- pushd $($(PKG)_BASE_NAME) > /dev/null && \
- $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
- gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
- popd > /dev/null && \
- rm -rf $($(PKG)_DL_DIR) && \
- popd > /dev/null)
+ $(GIT) archive --format=tar --remote $($(PKG)_SITE) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+ > $(DL_DIR)/$($(PKG)_SOURCE) || \
+ ( \
+ $(GIT) clone -v --depth 1 --bare $($(PKG)_SITE) $($(PKG)_DL_DIR) && \
+ $(GIT) archive --format=tar --remote $($(PKG)_DL_DIR) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
+ > $(DL_DIR)/$($(PKG)_SOURCE) && \
+ rm -rf $($(PKG)_DL_DIR) \
+ )
endef
# TODO: improve to check that the given PKG_DL_VERSION exists on the remote
@@ -504,13 +504,6 @@ ifneq ($$($(2)_OVERRIDE_SRCDIR),)
$(2)_VERSION = custom
endif
-ifndef $(2)_SOURCE
- ifdef $(3)_SOURCE
- $(2)_SOURCE = $($(3)_SOURCE)
- else
- $(2)_SOURCE ?= $$($(2)_BASE_NAME).tar.gz
- endif
-endif
ifndef $(2)_PATCH
ifdef $(3)_PATCH
@@ -536,6 +529,18 @@ ifndef $(2)_SITE_METHOD
endif
endif
+ifndef $(2)_SOURCE
+ ifdef $(3)_SOURCE
+ $(2)_SOURCE = $($(3)_SOURCE)
+ else
+ ifeq ($$($(2)_SITE_METHOD),git)
+ $(2)_SOURCE ?= $$($(2)_BASE_NAME).tar
+ else
+ $(2)_SOURCE ?= $$($(2)_BASE_NAME).tar.gz
+ endif
+ endif
+endif
+
ifeq ($$($(2)_SITE_METHOD),local)
ifeq ($$($(2)_OVERRIDE_SRCDIR),)
$(2)_OVERRIDE_SRCDIR = $($(2)_SITE)
--
1.7.4.1
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] Improved git download method performance.
2012-01-06 22:56 [Buildroot] [PATCH] Improved git download method performance Theodore Witkamp
@ 2012-01-06 23:40 ` Michael S. Zick
2012-01-07 9:42 ` Thomas Petazzoni
2012-01-15 18:45 ` Arnout Vandecappelle
1 sibling, 1 reply; 4+ messages in thread
From: Michael S. Zick @ 2012-01-06 23:40 UTC (permalink / raw)
To: buildroot
On Fri January 6 2012, Theodore Witkamp wrote:
>
> Instead of cloning the a complete repository, git archive is used first.
> If git archive fails because the remote host does not support it, git clone is
> used with with --depth 1 so only minimal history is copied.
>
> Also the $($(PKG)_SOURCE) default is changed to .tar instead of .tar.gz.
> This removes unneeded compression and decompression.
>
But some parts of the world charge per/byte still.
So maybe you want to offer a Wallet/CPU-Cycles preference or choice.
;-)
Mike
> Relative file paths are now relative to $(TOPDIR) not $(DL_DIR).
> ---
> package/Makefile.package.in | 35 ++++++++++++++++++++---------------
> 1 files changed, 20 insertions(+), 15 deletions(-)
>
> diff --git a/package/Makefile.package.in b/package/Makefile.package.in
> index e85eb15..14f1ba1 100644
> --- a/package/Makefile.package.in
> +++ b/package/Makefile.package.in
> @@ -146,14 +146,14 @@ domainseparator=$(if $(1),$(1),/)
>
> define DOWNLOAD_GIT
> test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
> - (pushd $(DL_DIR) > /dev/null && \
> - $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
> - pushd $($(PKG)_BASE_NAME) > /dev/null && \
> - $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
> - gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
> - popd > /dev/null && \
> - rm -rf $($(PKG)_DL_DIR) && \
> - popd > /dev/null)
> + $(GIT) archive --format=tar --remote $($(PKG)_SITE) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
> + > $(DL_DIR)/$($(PKG)_SOURCE) || \
> + ( \
> + $(GIT) clone -v --depth 1 --bare $($(PKG)_SITE) $($(PKG)_DL_DIR) && \
> + $(GIT) archive --format=tar --remote $($(PKG)_DL_DIR) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
> + > $(DL_DIR)/$($(PKG)_SOURCE) && \
> + rm -rf $($(PKG)_DL_DIR) \
> + )
> endef
>
> # TODO: improve to check that the given PKG_DL_VERSION exists on the remote
> @@ -504,13 +504,6 @@ ifneq ($$($(2)_OVERRIDE_SRCDIR),)
> $(2)_VERSION = custom
> endif
>
> -ifndef $(2)_SOURCE
> - ifdef $(3)_SOURCE
> - $(2)_SOURCE = $($(3)_SOURCE)
> - else
> - $(2)_SOURCE ?= $$($(2)_BASE_NAME).tar.gz
> - endif
> -endif
>
> ifndef $(2)_PATCH
> ifdef $(3)_PATCH
> @@ -536,6 +529,18 @@ ifndef $(2)_SITE_METHOD
> endif
> endif
>
> +ifndef $(2)_SOURCE
> + ifdef $(3)_SOURCE
> + $(2)_SOURCE = $($(3)_SOURCE)
> + else
> + ifeq ($$($(2)_SITE_METHOD),git)
> + $(2)_SOURCE ?= $$($(2)_BASE_NAME).tar
> + else
> + $(2)_SOURCE ?= $$($(2)_BASE_NAME).tar.gz
> + endif
> + endif
> +endif
> +
> ifeq ($$($(2)_SITE_METHOD),local)
> ifeq ($$($(2)_OVERRIDE_SRCDIR),)
> $(2)_OVERRIDE_SRCDIR = $($(2)_SITE)
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] Improved git download method performance.
2012-01-06 23:40 ` Michael S. Zick
@ 2012-01-07 9:42 ` Thomas Petazzoni
0 siblings, 0 replies; 4+ messages in thread
From: Thomas Petazzoni @ 2012-01-07 9:42 UTC (permalink / raw)
To: buildroot
Le Fri, 6 Jan 2012 17:40:08 -0600,
"Michael S. Zick" <minimod@morethan.org> a ?crit :
> > Also the $($(PKG)_SOURCE) default is changed to .tar instead
> > of .tar.gz. This removes unneeded compression and decompression.
>
> But some parts of the world charge per/byte still.
The move to .tar only changes the fact that tarball are saved
uncompressed in the $(DL_DIR). It does not change anything to the
amount of data transferred over the network, if I understand correctly.
But it's true that the size difference of a Linux kernel tarball
uncompressed and compressed is quite significant.
Regards,
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
^ permalink raw reply [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH] Improved git download method performance.
2012-01-06 22:56 [Buildroot] [PATCH] Improved git download method performance Theodore Witkamp
2012-01-06 23:40 ` Michael S. Zick
@ 2012-01-15 18:45 ` Arnout Vandecappelle
1 sibling, 0 replies; 4+ messages in thread
From: Arnout Vandecappelle @ 2012-01-15 18:45 UTC (permalink / raw)
To: buildroot
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
On Friday 06 January 2012 23:56:58 Theodore Witkamp wrote:
> Instead of cloning the a complete repository, git archive is used first.
> If git archive fails because the remote host does not support it, git clone is
> used with with --depth 1 so only minimal history is copied.
Great idea! BTW, is there any upstream that actually supports git archive?
> Also the $($(PKG)_SOURCE) default is changed to .tar instead of .tar.gz.
> This removes unneeded compression and decompression.
Bad idea. As noted elsewhere, this uses extra disk space (remember that
.tar.gz will stay in the dl directory). With gzip the cpu overhead is
fairly small - unless you have an SSD and a slow CPU, disk access will
probably dominate (since downloads/extracts are anyway never in parallel
with anything else).
> Relative file paths are now relative to $(TOPDIR) not $(DL_DIR).
That's of course a better idea, but may be an incompatible change for some
people. So Peter, make sure it gets a big fat warning in the Changelog.
> ---
> package/Makefile.package.in | 35 ++++++++++++++++++++---------------
> 1 files changed, 20 insertions(+), 15 deletions(-)
>
> diff --git a/package/Makefile.package.in b/package/Makefile.package.in
> index e85eb15..14f1ba1 100644
> --- a/package/Makefile.package.in
> +++ b/package/Makefile.package.in
> @@ -146,14 +146,14 @@ domainseparator=$(if $(1),$(1),/)
>
> define DOWNLOAD_GIT
> test -e $(DL_DIR)/$($(PKG)_SOURCE) || \
> - (pushd $(DL_DIR) > /dev/null && \
> - $(GIT) clone --bare $($(PKG)_SITE) $($(PKG)_BASE_NAME) && \
> - pushd $($(PKG)_BASE_NAME) > /dev/null && \
> - $(GIT) archive --format=tar --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
> - gzip -c > $(DL_DIR)/$($(PKG)_SOURCE) && \
> - popd > /dev/null && \
> - rm -rf $($(PKG)_DL_DIR) && \
> - popd > /dev/null)
> + $(GIT) archive --format=tar --remote $($(PKG)_SITE) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
> + > $(DL_DIR)/$($(PKG)_SOURCE) || \
> + ( \
> + $(GIT) clone -v --depth 1 --bare $($(PKG)_SITE) $($(PKG)_DL_DIR) && \
> + $(GIT) archive --format=tar --remote $($(PKG)_DL_DIR) --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) \
> + > $(DL_DIR)/$($(PKG)_SOURCE) && \
> + rm -rf $($(PKG)_DL_DIR) \
> + )
I would like to see the two git archive commands refactored into a single
make variable, but that's not a requirement for acceptance. Something like
define GIT_ARCHIVE_CMD
$(GIT) archive --format=tar --remote $${remote} --prefix=$($(PKG)_BASE_NAME)/ $($(PKG)_DL_VERSION) | \
gzip -c > $(DL_DIR)/$($(PKG)_SOURCE)
endif
...
remote=$($(PKG)_SITE); $(GIT_ARCHIVE_CMD) || \
...
Also please use some indentation in the shell scripts.
[snip, since the rest of the changes are unnecessary]
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286540
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120115/28471606/attachment-0001.html>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-01-15 18:45 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-06 22:56 [Buildroot] [PATCH] Improved git download method performance Theodore Witkamp
2012-01-06 23:40 ` Michael S. Zick
2012-01-07 9:42 ` Thomas Petazzoni
2012-01-15 18:45 ` Arnout Vandecappelle
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox