[Buildroot] Getting package sources from git/svn/bzr

[Buildroot] Getting package sources from git/svn/bzr

[Luca Ceresoli]

Hi,

I'm trying to understand how a buildroot generic (or autotools-based)
package can download sources from a git repository instead of just
grabbing a tarball.

In an old post Peter wrote:
> See the tremor handling in package/multimedia/libvorbis/libvorbis.mk
> for an example. You could argue about the usefullness of creating a
> dated tarball of the checkout or not.
(http://lists.busybox.net/pipermail/buildroot/2009-August/028932.html)

The odd thing is that tremor does *not* checkout anything: it just
downloads a tarball which presumably contains the snapshot (see log below).
I didn't find any other supposedly-working example in the current tree.

So I'd renew the question: is git/svn/bzr repository download actually
working? I spent a little time in trying, but after a few dirty hacks I
couldn't produce anything really working.

Am I missing something obvious?

I suspect it is not currently possible, and a clean way to do it would
require some changes in package/Makefile.package.in, making the download
step overridable by the package, such as:

@@ -118,19 +118,19 @@
 # Retrieve the archive
 $(BUILD_DIR)/%/.stamp_downloaded:
 # support make source-check/external-deps
 ifeq ($(SPIDER),)
 # Only show the download message if it isn't already downloaded
 	$(Q)(test -e $(DL_DIR)/$($(PKG)_SOURCE) && \
 		(test -z $($(PKG)_PATCH) || test -e $(DL_DIR)$($(PKG)_PATCH))) || \
 		$(call MESSAGE,"Downloading")
 endif
-	$(call DOWNLOAD,$($(PKG)_SITE),$($(PKG)_SOURCE))
+	$($(PKG)_DOWNLOAD_CMDS)
 	$(if $($(PKG)_PATCH),$(call DOWNLOAD,$($(PKG)_SITE),$($(PKG)_PATCH)))

where of course $(PKG)_DOWNLOAD_CMDS would default to $(call DOWNLOAD,...).

Of course a more complete approach would be to have a DOWNLOAD function
that actually understands git/svn/bzr, but this may be less interesting
until there are only very few packages using such a feature in mainline
buildroot.

Is my view correct?

Thanks,
Luca


Tremor download log follows.

-------------------------------8<-------------------------------

>>> tremor 16259 Downloading
--2010-06-30 10:03:47--  http://garr.dl.sourceforge.net/sourceforge/tremor/Tremor-svn-r16259.tar.bz2
Connecting to 172.20.0.254:8080... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://downloads.sourceforge.net/sourceforge/tremor/Tremor-svn-r16259.tar.bz2?download&failedmirror=garr.dl.sourceforge.net [following]
--2010-06-30 10:03:48--  http://downloads.sourceforge.net/sourceforge/tremor/Tremor-svn-r16259.tar.bz2?download&failedmirror=garr.dl.sourceforge.net
Connecting to 172.20.0.254:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
2010-06-30 10:03:48 ERROR 404: Not Found.

--2010-06-30 10:03:48--  http://buildroot.net/downloads/sources//Tremor-svn-r16259.tar.bz2
Connecting to 172.20.0.254:8080... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: http://sources.buildroot.net//Tremor-svn-r16259.tar.bz2 [following]
--2010-06-30 10:03:50--  http://sources.buildroot.net//Tremor-svn-r16259.tar.bz2
Connecting to 172.20.0.254:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 209977 (205K) [application/x-bzip2]
Saving to: `/home/murray/src/Tremor-svn-r16259.tar.bz2'

     0K .......... .......... .......... .......... .......... 24% 4.29M 0s
    50K .......... .......... .......... .......... .......... 48% 25.2M 0s
   100K .......... .......... .......... .......... .......... 73% 11.1M 0s
   150K .......... .......... .......... .......... .......... 97% 11.1M 0s
   200K .....                                                 100% 8.39M=0.02s

2010-06-30 10:03:52 (8.82 MB/s) - `/home/murray/src/Tremor-svn-r16259.tar.bz2' saved [209977/209977]

>>> tremor 16259 Extracting

-------------------------------8<-------------------------------

[Buildroot] Getting package sources from git/svn/bzr

[Thomas Petazzoni]

Hello,

On Wed, 30 Jun 2010 10:44:08 +0200
"Luca Ceresoli" <luca@lucaceresoli.net> wrote:

> I'm trying to understand how a buildroot generic (or autotools-based)
> package can download sources from a git repository instead of just
> grabbing a tarball.

As of today, this isn't supported.

> In an old post Peter wrote:
> > See the tremor handling in package/multimedia/libvorbis/libvorbis.mk
> > for an example. You could argue about the usefullness of creating a
> > dated tarball of the checkout or not.
> (http://lists.busybox.net/pipermail/buildroot/2009-August/028932.html)
> 
> The odd thing is that tremor does *not* checkout anything: it just
> downloads a tarball which presumably contains the snapshot (see log
> below). I didn't find any other supposedly-working example in the
> current tree.

Yes, it just a normal HTTP download, where the file happens to be a SVN
snapshot.

> So I'd renew the question: is git/svn/bzr repository download actually
> working? I spent a little time in trying, but after a few dirty hacks
> I couldn't produce anything really working.
> 
> Am I missing something obvious?

No, it isn't supported as of today.

Quotient Remainder has proposed something last month to integrate Git
support (see
http://lists.busybox.net/pipermail/buildroot/2010-June/034942.html). I
unfortunately haven't had the time to carefully review his proposal,
but I'm not sure I like the general approach.

> I suspect it is not currently possible, and a clean way to do it would
> require some changes in package/Makefile.package.in, making the
> download step overridable by the package, such as:
> 
> @@ -118,19 +118,19 @@
>  # Retrieve the archive
>  $(BUILD_DIR)/%/.stamp_downloaded:
>  # support make source-check/external-deps
>  ifeq ($(SPIDER),)
>  # Only show the download message if it isn't already downloaded
>  	$(Q)(test -e $(DL_DIR)/$($(PKG)_SOURCE) && \
>  		(test -z $($(PKG)_PATCH) || test -e
> $(DL_DIR)$($(PKG)_PATCH))) || \ $(call MESSAGE,"Downloading")
>  endif
> -	$(call DOWNLOAD,$($(PKG)_SITE),$($(PKG)_SOURCE))
> +	$($(PKG)_DOWNLOAD_CMDS)
>  	$(if $($(PKG)_PATCH),$(call
> DOWNLOAD,$($(PKG)_SITE),$($(PKG)_PATCH)))
> 
> where of course $(PKG)_DOWNLOAD_CMDS would default to $(call
> DOWNLOAD,...).

Or just to improve the existing DOWNLOAD step so that if the URL starts
with git://, svn:// or even something like git at http:// (for Git repos
available through http), it knows what to do.

The issue I have with this is *when* to do a "git pull" or "svn
update". With normal packages, we download the particular version of a
particular tarball, and we uncompress it in $(BUILD_DIR)/package-X.Y.Z.
So, if the package gets its version bumped in the .mk file, we download
the new version package-X.Y.A, we extract it in a new directory
$(BUILD_DIR)/package-X.Y.A, and we're done.

If you just give the URL of a Git or SVN repo, how is this going to
work ? If you download the HEAD of a Git repo in a directory named
$(BUILD_DIR)/mypackage, when should the package be redownloaded ? How
is the version bump going to be done ?

IMO, the simplest solution is to do something like for SVN

MYPACKAGE_VERSION=23678 # is in fact the SVN revision
MYPACKAGE_SITE=svn://foo.bar.com/projects/mypackage/trunk

and similarly for Git :

MYPACKAGE_VERSION=902108904ad7daec88b367dd5fcd1c46e71ac3e6
MYPACKAGE_SITE=git://foo.bar.com/projects/mypackage.git

This way, a fixed version is choosen, so the build is reproducible. I
would find it rather disappointing to have "random" versions being
built by Buildroot.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Getting package sources from git/svn/bzr

[Quotient Remainder]

On Thu, 2010-07-01 at 09:43 +0200, Thomas Petazzoni wrote:
> On Wed, 30 Jun 2010 10:44:08 +0200
> "Luca Ceresoli" <luca@lucaceresoli.net> wrote:
> 
> 
> > So I'd renew the question: is git/svn/bzr repository download actually
> > working? I spent a little time in trying, but after a few dirty hacks
> > I couldn't produce anything really working.
> > 
> > Am I missing something obvious?
> 
> No, it isn't supported as of today.
> 
> Quotient Remainder has proposed something last month to integrate Git
> support (see
> http://lists.busybox.net/pipermail/buildroot/2010-June/034942.html). I
> unfortunately haven't had the time to carefully review his proposal,
> but I'm not sure I like the general approach.
> 

I'll admit I'm not all that fond of my approach either but I felt I'd
try to get some detailed discussion the subject started at least.  Since
I submitted it I noticed a few errors but I have since taken a new,
though similar, approach but this is not ready for public review as yet.

This reply is both a response to some of the specific issues raised by
Thomas and some clarifications on the direction I was taking.  My needs
are currently git-centred but many concepts should apply to other VCS.

> Or just to improve the existing DOWNLOAD step so that if the URL starts
> with git://, svn:// or even something like git at http:// (for Git repos
> available through http), it knows what to do.

This is what my patch does; instead of always calling wget, DOWNLOAD is
changed to extract the protocol at the start of the URL and then call
DOWNLOAD_WGET, DOWNLOAD_GIT, DOWNLOAD_SVN, allowing aliases like
DOWNLOAD_HTTP [for WGET) and DOWNLOAD_SSH (for GIT).  I hadn't
specifically though about git over HTTP but at a stretch we could use
something non-standard like githttp://git at site.org/... and parse out the
user.

One problem I had is that DOWNLOAD is always evaluated by the shell (in
the makefile command context - tabbed in) so it's not straightforward to
do conditional stuff.  This could be greatly simplified if $(call
DOWNLOAD ...) was called in the makefile directive context
(non-indented).  As you can see, makefiles (and their terminology) are
not my major.  As stated previously, I wanted to be as non-invasive as
possible so did not do this myself.

> The issue I have with this is *when* to do a "git pull" or "svn
> update". With normal packages, we download the particular version of a
> particular tarball, and we uncompress it in $(BUILD_DIR)/package-X.Y.Z.
> So, if the package gets its version bumped in the .mk file, we download
> the new version package-X.Y.A, we extract it in a new directory
> $(BUILD_DIR)/package-X.Y.A, and we're done.
> ...
> If you just give the URL of a Git or SVN repo, how is this going to 
> work ? If you download the HEAD of a Git repo in a directory named
> $(BUILD_DIR)/mypackage, when should the package be redownloaded ? How 
> is the version bump going to be done ?

I wanted git to work using both the "clone" command to get a working
copy and "archive --remote" command to get a snapshot.

For most users, the snapshot feature would be used.  Since all this does
is get a tarball to the $(DL_DIR) the pull problem doesn't arise, if you
change the version then there will be no archive in $(DL_DIR) with the
new version, e.g dl/package-b78686aaadeacdda67abade29f.tgz

The clone context introduces complications though.  "make source" will
clone the repository to $(BUILD_DIR) and check out the requested
version.  The extract stage is not necessary and for my own purposes no
patches should be applied so I skip the patch stage also - others may
feel different.  My reasoning is that if you're working on source you
will not want anyone else patches to be automatically applied.

Running "make source" again should in some way recognise source
controlled packages and checkout the specified versions of each package.
By this I mean that PACKAGE_VERSION in the package makefile may have
changed since "make source" was last run.

As time passes developers may wish to get the latest updates from
upstream.  For this I feel that a new command "make source-update" or
similar would be needed.  This would be analogous to "git pull".


One of the messy parts of my approach is that there is no good way to
choose on a package-by-package level whether to get a snapshot or clone.
Any ideas on this are most welcome.


> MYPACKAGE_VERSION=902108904ad7daec88b367dd5fcd1c46e71ac3e6
> MYPACKAGE_SITE=git://foo.bar.com/projects/mypackage.git
> 
> This way, a fixed version is choosen, so the build is reproducible. I
> would find it rather disappointing to have "random" versions being
> built by Buildroot.
> 

For my own purposes I need to have all package versions in a single
top-level file (a bit like a partial manifest to be used in combination
with the buildroot config):
	PACKAGE1_VERSION=3298703984
	PACKAGE2_VERSION=dsaf8977dsaadfs09
and so on.

We can support this by having a default version in the package makefile,
which can be overridden by the top-level file:
	PACKAGE1_VERSION ?= djkhsad87

The top-level file can be a "-include" in the top-level Makefile or
wherever is appropriate.


Anyway, that's a selection of my thoughts on this.  All comments - bad
and not so bad - are welcomed!

[Buildroot] Getting package sources from git/svn/bzr

[Michael S. Zick]

On Thu July 1 2010, Thomas Petazzoni wrote:
>
- - - -
> 
> Or just to improve the existing DOWNLOAD step so that if the URL starts
> with git://, svn:// or even something like git at http:// (for Git repos
> available through http), it knows what to do.
> 
> The issue I have with this is *when* to do a "git pull" or "svn
> update". With normal packages, we download the particular version of a
> particular tarball, and we uncompress it in $(BUILD_DIR)/package-X.Y.Z.
> So, if the package gets its version bumped in the .mk file, we download
> the new version package-X.Y.A, we extract it in a new directory
> $(BUILD_DIR)/package-X.Y.A, and we're done.
> 

And with a lot of things, they have version specific patches;
So how would that be made to work grabbing random versions?

Of course some patches are not version specific, they usually deal
with some sort of structural or build environment differences.
Which does not help this situation, now both version specific and
non-version specific patches have to be dealt with along with
how to tell the difference.

> If you just give the URL of a Git or SVN repo, how is this going to
> work ? If you download the HEAD of a Git repo in a directory named
> $(BUILD_DIR)/mypackage, when should the package be redownloaded ? How
> is the version bump going to be done ?
> 
> IMO, the simplest solution is to do something like for SVN
> 
> MYPACKAGE_VERSION=23678 # is in fact the SVN revision
> MYPACKAGE_SITE=svn://foo.bar.com/projects/mypackage/trunk
> 
> and similarly for Git :
> 
> MYPACKAGE_VERSION=902108904ad7daec88b367dd5fcd1c46e71ac3e6
> MYPACKAGE_SITE=git://foo.bar.com/projects/mypackage.git
> 
> This way, a fixed version is choosen, so the build is reproducible. I
> would find it rather disappointing to have "random" versions being
> built by Buildroot.
> 

Version control systems often employ tags or revision numbers that
point into the version control storage in such a way that a constant
content image can be pulled.

Planning a syntax that allows a version tag or a revision number to
be included, perhaps sourced from a menuconfig string field should
deal with the repeatability of a build.

Mercurial takes schemes of http://, https://, ssh:// for its network
operations that a repository-remote build system might use.

Here is a small one you can browse - note that the web interface allows
you to pull an archive from any specific revision number (or tag, or branch
but I don't have any tags (other than tip) or branches (other than default))
http://hg.minimodding.com

Same with the remote commands - they can pull a clone or a bundle by tag/branch.

Mike
> Best regards,
> 
> Thomas

[Buildroot] Getting package sources from git/svn/bzr/hg

[Michael S. Zick]

On Thu July 1 2010, Quotient Remainder wrote:
> On Thu, 2010-07-01 at 09:43 +0200, Thomas Petazzoni wrote:
> > On Wed, 30 Jun 2010 10:44:08 +0200
> > "Luca Ceresoli" <luca@lucaceresoli.net> wrote:
> > 
> > 
> > > So I'd renew the question: is git/svn/bzr repository download actually
> > > working? I spent a little time in trying, but after a few dirty hacks
> > > I couldn't produce anything really working.
> > > 
> > > Am I missing something obvious?
> > 
> > No, it isn't supported as of today.
> > 
> > Quotient Remainder has proposed something last month to integrate Git
> > support (see
> > http://lists.busybox.net/pipermail/buildroot/2010-June/034942.html). I
> > unfortunately haven't had the time to carefully review his proposal,
> > but I'm not sure I like the general approach.
> > 
> 
> I'll admit I'm not all that fond of my approach either but I felt I'd
> try to get some detailed discussion the subject started at least.  Since
> I submitted it I noticed a few errors but I have since taken a new,
> though similar, approach but this is not ready for public review as yet.
> 
> This reply is both a response to some of the specific issues raised by
> Thomas and some clarifications on the direction I was taking.  My needs
> are currently git-centred but many concepts should apply to other VCS.
> 
> > Or just to improve the existing DOWNLOAD step so that if the URL starts
> > with git://, svn:// or even something like git at http:// (for Git repos
> > available through http), it knows what to do.
> 
> This is what my patch does; instead of always calling wget, DOWNLOAD is
> changed to extract the protocol at the start of the URL and then call
> DOWNLOAD_WGET, DOWNLOAD_GIT, DOWNLOAD_SVN, allowing aliases like
> DOWNLOAD_HTTP [for WGET) and DOWNLOAD_SSH (for GIT).  I hadn't
> specifically though about git over HTTP but at a stretch we could use
> something non-standard like githttp://git at site.org/... and parse out the
> user.
> 
> One problem I had is that DOWNLOAD is always evaluated by the shell (in
> the makefile command context - tabbed in) so it's not straightforward to
> do conditional stuff.  This could be greatly simplified if $(call
> DOWNLOAD ...) was called in the makefile directive context
> (non-indented).  As you can see, makefiles (and their terminology) are
> not my major.  As stated previously, I wanted to be as non-invasive as
> possible so did not do this myself.
> 
> > The issue I have with this is *when* to do a "git pull" or "svn
> > update". With normal packages, we download the particular version of a
> > particular tarball, and we uncompress it in $(BUILD_DIR)/package-X.Y.Z.
> > So, if the package gets its version bumped in the .mk file, we download
> > the new version package-X.Y.A, we extract it in a new directory
> > $(BUILD_DIR)/package-X.Y.A, and we're done.
> > ...
> > If you just give the URL of a Git or SVN repo, how is this going to 
> > work ? If you download the HEAD of a Git repo in a directory named
> > $(BUILD_DIR)/mypackage, when should the package be redownloaded ? How 
> > is the version bump going to be done ?
> 
> I wanted git to work using both the "clone" command to get a working
> copy and "archive --remote" command to get a snapshot.
> 
> For most users, the snapshot feature would be used.  Since all this does
> is get a tarball to the $(DL_DIR) the pull problem doesn't arise, if you
> change the version then there will be no archive in $(DL_DIR) with the
> new version, e.g dl/package-b78686aaadeacdda67abade29f.tgz
> 
> The clone context introduces complications though.  "make source" will
> clone the repository to $(BUILD_DIR) and check out the requested
> version.  The extract stage is not necessary and for my own purposes no
> patches should be applied so I skip the patch stage also - others may
> feel different.  My reasoning is that if you're working on source you
> will not want anyone else patches to be automatically applied.
> 
> Running "make source" again should in some way recognise source
> controlled packages and checkout the specified versions of each package.
> By this I mean that PACKAGE_VERSION in the package makefile may have
> changed since "make source" was last run.
> 
> As time passes developers may wish to get the latest updates from
> upstream.  For this I feel that a new command "make source-update" or
> similar would be needed.  This would be analogous to "git pull".
> 
> 
> One of the messy parts of my approach is that there is no good way to
> choose on a package-by-package level whether to get a snapshot or clone.
> Any ideas on this are most welcome.
> 
> 
> > MYPACKAGE_VERSION=902108904ad7daec88b367dd5fcd1c46e71ac3e6
> > MYPACKAGE_SITE=git://foo.bar.com/projects/mypackage.git
> > 
> > This way, a fixed version is choosen, so the build is reproducible. I
> > would find it rather disappointing to have "random" versions being
> > built by Buildroot.
> > 
> 
> For my own purposes I need to have all package versions in a single
> top-level file (a bit like a partial manifest to be used in combination
> with the buildroot config):
> 	PACKAGE1_VERSION=3298703984
> 	PACKAGE2_VERSION=dsaf8977dsaadfs09
> and so on.
>

Grab a sample tar-ball from my project under Mercurial -
http://hg.minimodding.com

In the top level of the archive you will find a file named: .hg_archival.txt 

Content looks like:
repo: e9e6cd658042687e847b8247448b1d9fd485bd6c
node: a719b06876935ac6803f2e73171a12f8068d61b9
branch: default
latesttag: null
latesttagdistance: 31

With a bit of scripted editing, that could be turned into a manifest of
repeatable check-out points used in a given build for everything that
came from Mercurial repositories.

One show-stopping problem with this:
The .hg_archival.txt file is a per-repository option, not an unconditional
output of Mercurial for every repository it maintains. (Drat! More conditionals.)

I personally can't think of why someone would turn it off but I think it defaults 
to "off" so you can't trust that every repo owner remembers to turn it on.

Mike

> We can support this by having a default version in the package makefile,
> which can be overridden by the top-level file:
> 	PACKAGE1_VERSION ?= djkhsad87
> 
> The top-level file can be a "-include" in the top-level Makefile or
> wherever is appropriate.
> 
> 
> Anyway, that's a selection of my thoughts on this.  All comments - bad
> and not so bad - are welcomed!
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 
> 

[Buildroot] Getting package sources from git/svn/bzr

[Peter Korsgaard]

>>>>> "Luca" == Luca Ceresoli <luca@lucaceresoli.net> writes:

 Luca> Hi,
 Luca> I'm trying to understand how a buildroot generic (or autotools-based)
 Luca> package can download sources from a git repository instead of just
 Luca> grabbing a tarball.

 Luca> In an old post Peter wrote:
 >> See the tremor handling in package/multimedia/libvorbis/libvorbis.mk
 >> for an example. You could argue about the usefullness of creating a
 >> dated tarball of the checkout or not.
 Luca> (http://lists.busybox.net/pipermail/buildroot/2009-August/028932.html)

 Luca> The odd thing is that tremor does *not* checkout anything: it just
 Luca> downloads a tarball which presumably contains the snapshot (see log below).
 Luca> I didn't find any other supposedly-working example in the current tree.

Ups, seems like Sven's commit (1dc19c44570031e28ad4) from back in
December broke it - I'll fix.

 Luca> So I'd renew the question: is git/svn/bzr repository download actually
 Luca> working? I spent a little time in trying, but after a few dirty hacks I
 Luca> couldn't produce anything really working.

No, there's no generic support for it.

 Luca> Of course a more complete approach would be to have a DOWNLOAD function
 Luca> that actually understands git/svn/bzr, but this may be less interesting
 Luca> until there are only very few packages using such a feature in mainline
 Luca> buildroot.

I've pondered that as well. One of the issues is how to encode the
git/svn/bzr specific parameters in an URL, so we can easily extract it,
and figuring out if a http:// url means something for wget or
svn-over-http or git-over-http.

 Luca> Is my view correct?

yes.

-- 
Bye, Peter Korsgaard

[Buildroot] Getting package sources from git/svn/bzr

[Luca Ceresoli]

Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 30 Jun 2010 10:44:08 +0200
> "Luca Ceresoli" <luca@lucaceresoli.net> wrote:
> 
> > I'm trying to understand how a buildroot generic (or autotools-based)
> > package can download sources from a git repository instead of just
> > grabbing a tarball.
> 
> As of today, this isn't supported.
> 
> > In an old post Peter wrote:
> > > See the tremor handling in package/multimedia/libvorbis/libvorbis.mk
> > > for an example. You could argue about the usefullness of creating a
> > > dated tarball of the checkout or not.
> > (http://lists.busybox.net/pipermail/buildroot/2009-August/028932.html)
> > 
> > The odd thing is that tremor does *not* checkout anything: it just
> > downloads a tarball which presumably contains the snapshot (see log
> > below). I didn't find any other supposedly-working example in the
> > current tree.
> 
> Yes, it just a normal HTTP download, where the file happens to be a SVN
> snapshot.
> 
> > So I'd renew the question: is git/svn/bzr repository download actually
> > working? I spent a little time in trying, but after a few dirty hacks
> > I couldn't produce anything really working.
> > 
> > Am I missing something obvious?
> 
> No, it isn't supported as of today.
> 
> Quotient Remainder has proposed something last month to integrate Git
> support (see
> http://lists.busybox.net/pipermail/buildroot/2010-June/034942.html). I
> unfortunately haven't had the time to carefully review his proposal,
> but I'm not sure I like the general approach.

Sorry for having missed that one. I read it now, and I agree with your
comments.

> 
> > I suspect it is not currently possible, and a clean way to do it would
> > require some changes in package/Makefile.package.in, making the
> > download step overridable by the package, such as:
> > 
> > @@ -118,19 +118,19 @@
> >  # Retrieve the archive
> >  $(BUILD_DIR)/%/.stamp_downloaded:
> >  # support make source-check/external-deps
> >  ifeq ($(SPIDER),)
> >  # Only show the download message if it isn't already downloaded
> >  	$(Q)(test -e $(DL_DIR)/$($(PKG)_SOURCE) && \
> >  		(test -z $($(PKG)_PATCH) || test -e
> > $(DL_DIR)$($(PKG)_PATCH))) || \ $(call MESSAGE,"Downloading")
> >  endif
> > -	$(call DOWNLOAD,$($(PKG)_SITE),$($(PKG)_SOURCE))
> > +	$($(PKG)_DOWNLOAD_CMDS)
> >  	$(if $($(PKG)_PATCH),$(call
> > DOWNLOAD,$($(PKG)_SITE),$($(PKG)_PATCH)))
> > 
> > where of course $(PKG)_DOWNLOAD_CMDS would default to $(call
> > DOWNLOAD,...).
> 
> Or just to improve the existing DOWNLOAD step so that if the URL starts
> with git://, svn:// or even something like git at http:// (for Git repos
> available through http), it knows what to do.

I'll add a couple annoying variations.

We use gitorious to host corporate projects, and those have strange URLs
such as "gitorious at myserver:myrepo", which technically might also be
"weird_user_name at myserver:myrepo", without any "git" substring at all.
That would be ok for gitorious (and maybe desired by some crazy sysadmin),
but would require buildroot a little more effort. Maybe this way:

define DOWNLOAD
 ifneq ($($(3)_DOWNLOAD_PROTOCOL),)
  # use user-provided protocol
  $(call DOWNLOAD_$($(3)_DOWNLOAD_PROTOCOL),$(1),$(2),$(3))
 else
  # try automatic detection - AS PER QUOTIENT'S PATCH
  $(call DOWNLOAD_$(call UPPERCASE,$(firstword \
           $(subst ://, ,$(call qstrip,$(1))))),$(1),$(2),$(3))
 endif
endef #DOWNLOAD

A more complex use case is git submodules.
I find them useful when development and system integration happen
simultaneously, which is typical for embedded products.
But I'll be back on this in a while.

> 
> The issue I have with this is *when* to do a "git pull" or "svn
> update". With normal packages, we download the particular version of a
> particular tarball, and we uncompress it in $(BUILD_DIR)/package-X.Y.Z.
> So, if the package gets its version bumped in the .mk file, we download
> the new version package-X.Y.A, we extract it in a new directory
> $(BUILD_DIR)/package-X.Y.A, and we're done.
> 
> If you just give the URL of a Git or SVN repo, how is this going to
> work ? If you download the HEAD of a Git repo in a directory named
> $(BUILD_DIR)/mypackage, when should the package be redownloaded ? How
> is the version bump going to be done ?
> 
> IMO, the simplest solution is to do something like for SVN
> 
> MYPACKAGE_VERSION=23678 # is in fact the SVN revision
> MYPACKAGE_SITE=svn://foo.bar.com/projects/mypackage/trunk
> 
> and similarly for Git :
> 
> MYPACKAGE_VERSION=902108904ad7daec88b367dd5fcd1c46e71ac3e6
> MYPACKAGE_SITE=git://foo.bar.com/projects/mypackage.git
> 
> This way, a fixed version is choosen, so the build is reproducible. I
> would find it rather disappointing to have "random" versions being
> built by Buildroot.

Definitely.
This also solves the question of when to redownload.


Now let me go back to the git submodule use case.

In this scenario the git submodule is something that should be located in
a place where it is not deleted by make (dist)clean, which IMHO could be
inside packages/.

The "download" would just happen to be a regular git clone or git archive,
supposing no one will need to build the current submodule working copy
(including local, uncommitted changes).
The URL could be a directory inside the package/$(PKG)/. Buildroot can
realize that it is a submodule by checking for the existence of a .git/
subdir.
Oh, and this whole thing would only work if BR itself in a git clone, not
if one grabs buildroot from a tarball.

Luca

[Buildroot] Getting package sources from git/svn/bzr

[Luca Ceresoli]

Quotient Remainder wrote:
> On Thu, 2010-07-01 at 09:43 +0200, Thomas Petazzoni wrote:
> > On Wed, 30 Jun 2010 10:44:08 +0200
> > "Luca Ceresoli" <luca@lucaceresoli.net> wrote:
> > 
> > 
> > > So I'd renew the question: is git/svn/bzr repository download actually
> > > working? I spent a little time in trying, but after a few dirty hacks
> > > I couldn't produce anything really working.
> > > 
> > > Am I missing something obvious?
> > 
> > No, it isn't supported as of today.
> > 
> > Quotient Remainder has proposed something last month to integrate Git
> > support (see
> > http://lists.busybox.net/pipermail/buildroot/2010-June/034942.html). I
> > unfortunately haven't had the time to carefully review his proposal,
> > but I'm not sure I like the general approach.
> > 
> 
> I'll admit I'm not all that fond of my approach either but I felt I'd
> try to get some detailed discussion the subject started at least.  Since
> I submitted it I noticed a few errors but I have since taken a new,
> though similar, approach but this is not ready for public review as yet.
> 
> This reply is both a response to some of the specific issues raised by
> Thomas and some clarifications on the direction I was taking.  My needs
> are currently git-centred but many concepts should apply to other VCS.
> 
> > Or just to improve the existing DOWNLOAD step so that if the URL starts
> > with git://, svn:// or even something like git at http:// (for Git repos
> > available through http), it knows what to do.
> 
> This is what my patch does; instead of always calling wget, DOWNLOAD is
> changed to extract the protocol at the start of the URL and then call
> DOWNLOAD_WGET, DOWNLOAD_GIT, DOWNLOAD_SVN, allowing aliases like
> DOWNLOAD_HTTP [for WGET) and DOWNLOAD_SSH (for GIT).  I hadn't
> specifically though about git over HTTP but at a stretch we could use
> something non-standard like githttp://git at site.org/... and parse out the
> user.
> 
> One problem I had is that DOWNLOAD is always evaluated by the shell (in
> the makefile command context - tabbed in) so it's not straightforward to
> do conditional stuff.  This could be greatly simplified if $(call
> DOWNLOAD ...) was called in the makefile directive context
> (non-indented).  As you can see, makefiles (and their terminology) are
> not my major.  As stated previously, I wanted to be as non-invasive as
> possible so did not do this myself.
> 
> > The issue I have with this is *when* to do a "git pull" or "svn
> > update". With normal packages, we download the particular version of a
> > particular tarball, and we uncompress it in $(BUILD_DIR)/package-X.Y.Z.
> > So, if the package gets its version bumped in the .mk file, we download
> > the new version package-X.Y.A, we extract it in a new directory
> > $(BUILD_DIR)/package-X.Y.A, and we're done.
> > ...
> > If you just give the URL of a Git or SVN repo, how is this going to 
> > work ? If you download the HEAD of a Git repo in a directory named
> > $(BUILD_DIR)/mypackage, when should the package be redownloaded ? How 
> > is the version bump going to be done ?
> 
> I wanted git to work using both the "clone" command to get a working
> copy and "archive --remote" command to get a snapshot.
> 
> For most users, the snapshot feature would be used.  Since all this does
> is get a tarball to the $(DL_DIR) the pull problem doesn't arise, if you
> change the version then there will be no archive in $(DL_DIR) with the
> new version, e.g dl/package-b78686aaadeacdda67abade29f.tgz
> 
> The clone context introduces complications though.  "make source" will
> clone the repository to $(BUILD_DIR) and check out the requested
> version.  The extract stage is not necessary and for my own purposes no
> patches should be applied so I skip the patch stage also - others may
> feel different.  My reasoning is that if you're working on source you
> will not want anyone else patches to be automatically applied.

The difference between the two methods (clone vs archive) is not clear
to me. I got how each works, but what advantages do they have over each
other?

> 
> Running "make source" again should in some way recognise source
> controlled packages and checkout the specified versions of each package.
> By this I mean that PACKAGE_VERSION in the package makefile may have
> changed since "make source" was last run.
> 
> As time passes developers may wish to get the latest updates from
> upstream.  For this I feel that a new command "make source-update" or
> similar would be needed.  This would be analogous to "git pull".
> 
> 
> One of the messy parts of my approach is that there is no good way to
> choose on a package-by-package level whether to get a snapshot or clone.
> Any ideas on this are most welcome.
> 
> 
> > MYPACKAGE_VERSION=902108904ad7daec88b367dd5fcd1c46e71ac3e6
> > MYPACKAGE_SITE=git://foo.bar.com/projects/mypackage.git
> > 
> > This way, a fixed version is choosen, so the build is reproducible. I
> > would find it rather disappointing to have "random" versions being
> > built by Buildroot.
> > 
> 
> For my own purposes I need to have all package versions in a single
> top-level file (a bit like a partial manifest to be used in combination
> with the buildroot config):
> 	PACKAGE1_VERSION=3298703984
> 	PACKAGE2_VERSION=dsaf8977dsaadfs09
> and so on.

I don't like this as a way to mainline package version management and
reproducibility. Can you explain why you find it useful?
If this is just a way to override the package/*/*.mk defaults, why not
just changing them directly?

Luca

> 
> We can support this by having a default version in the package makefile,
> which can be overridden by the top-level file:
> 	PACKAGE1_VERSION ?= djkhsad87
> 
> The top-level file can be a "-include" in the top-level Makefile or
> wherever is appropriate.
> 
> 
> Anyway, that's a selection of my thoughts on this.  All comments - bad
> and not so bad - are welcomed!
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

[Buildroot] Getting package sources from git/svn/bzr

[Quotient Remainder]

Hello Luca,

On Fri, 2010-07-02 at 12:58 +0200, Luca Ceresoli wrote:
> 
> The difference between the two methods (clone vs archive) is not clear
> to me. I got how each works, but what advantages do they have over each
> other?

Well, I'm thinking that most people will just get an archive because
they just want to compile the code, but some will want to make edits and
contribute their changes so they will need to do a clone.  I suppose
it's a first step in making buildroot into a system that can be used as
a day-to-day development environment as well as a build system.

In my particular environment, certification requires the source of all
software to be trusted so _all_ packages will have to be downloaded from
local git repositories.  I realise that very few will have a similar
requirement but I believe that adding the features being discussed would
benefit a wider audience.

> I don't like this as a way to mainline package version management and
> reproducibility. Can you explain why you find it useful?
> If this is just a way to override the package/*/*.mk defaults, why not
> just changing them directly?
> 
> Luca
> 

Take the example of a project that has a few releases in existence.  One
day I decide I want to get release X, which had say 50 packages each
with its own version.  So instead of having to check out Buildroot
(containing per-package versions) as of the date of release X, perhaps
losing some of the improvements since then, I just want to be able to
check out one file that lists all package versions.  Then if I want to
move to release Y, I just get that version file as at version Y's
release.  Only the packages that have been changed will need to be
updated so it will cut down on the amount of code to be transmitted.

I'd also need to get the config files from that release to determine
what should be built.

You know, maybe the in-package version and entire Buildroot clone _is_
the way to go since you can be sure the config files are compatible with
the overall Buildroot environment - a state not guaranteed by my
proposal.
There should to be a way of outputting these versions to create a
manifest of some sort though.

[Buildroot] Getting package sources from git/svn/bzr

[Luca Ceresoli]

Quotient Remainder wrote:
> Hello Luca,
>
> On Fri, 2010-07-02 at 12:58 +0200, Luca Ceresoli wrote:
> >
> > The difference between the two methods (clone vs archive) is not clear
> > to me. I got how each works, but what advantages do they have over each
> > other?
>
> Well, I'm thinking that most people will just get an archive because
> they just want to compile the code, but some will want to make edits and
> contribute their changes so they will need to do a clone. I suppose
> it's a first step in making buildroot into a system that can be used as
> a day-to-day development environment as well as a build system.

While git archive --remote is definitely a clever thing for buildroot,
I notice it does not work in many cases.

Github does not support git archive --remote
(http://support.github.com/discussions/feature-requests/145-deploying-without-cloning)

Gitosis does not work as well.

It seems like git.kernel.org and git.pengutronix.de do not as well. It might
be caused my the damn corporate firewall, but sniffing suggest it's not so.

Luca