* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-09-27 14:43 [Buildroot] [RFC] package: Redownload HEAD packages every build Clayton Shotwell
@ 2013-09-27 14:51 ` Clayton Shotwell
2013-09-27 15:11 ` Thomas De Schampheleire
` (2 subsequent siblings)
3 siblings, 0 replies; 10+ messages in thread
From: Clayton Shotwell @ 2013-09-27 14:51 UTC (permalink / raw)
To: buildroot
Clayton Shotwell <clshotwe@rockwellcollins.com> wrote on 09/27/2013
09:43:35 AM:
> Subject: [RFC] package: Redownload HEAD packages every build
>
> Adding a check to remove a downloaded package if the version is
> HEAD. This causes the package to be re-downloaded with updated
> software. This feature is very useful during package development.
I apologize for messing up the commit message. That will be fixed in the
next version.
Thanks,
Clayton
Clayton Shotwell
Software Engineer
clshotwe at rockwellcollins.com
www.rockwellcollins.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130927/bdca5831/attachment.html>
^ permalink raw reply [flat|nested] 10+ messages in thread* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-09-27 14:43 [Buildroot] [RFC] package: Redownload HEAD packages every build Clayton Shotwell
2013-09-27 14:51 ` Clayton Shotwell
@ 2013-09-27 15:11 ` Thomas De Schampheleire
2013-10-01 16:08 ` Arnout Vandecappelle
2013-10-01 18:51 ` Thomas Petazzoni
3 siblings, 0 replies; 10+ messages in thread
From: Thomas De Schampheleire @ 2013-09-27 15:11 UTC (permalink / raw)
To: buildroot
Clayton Shotwell <clshotwe@rockwellcollins.com> wrote:
>Adding a check to remove a downloaded package if the version is HEAD. This causes the package to be re-downloaded with updated software. This feature is very useful during package development.
>
>Signed-off-by: Clayton Shotwell <clshotwe@rockwellcollins.com>
>---
> package/pkg-generic.mk | 7 +++++++
> 1 files changed, 7 insertions(+), 0 deletions(-)
>
>diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>index bfc4dc1..859c4a9 100644
>--- a/package/pkg-generic.mk
>+++ b/package/pkg-generic.mk
>@@ -27,6 +27,13 @@
> # Retrieve the archive
> $(BUILD_DIR)/%/.stamp_downloaded:
> ifeq ($(DL_MODE),DOWNLOAD)
>+# Remove the pre-downloaded tar file if the package is a HEAD version
>+ $(Q)( \
>+ if test $($(PKG)_VERSION) == HEAD -a -e $(DL_DIR)/$($(PKG)_SOURCE); then \
>+ rm $(DL_DIR)/$($(PKG)_SOURCE); \
>+ $(call MESSAGE,"Removing $(PKG) HEAD source file"); \
>+ fi; \
>+ )
> # Only show the download message if it isn't already downloaded
> $(Q)if test ! -e $(DL_DIR)/$($(PKG)_SOURCE); then \
> $(call MESSAGE,"Downloading") ; \
Not sure if this patch would be accepted, but in case it is I think you should also check for 'tip' which is the mercurial name for HEAD. Note that it may be safer to also check the FOO_SITE_METHOD variable for hg or git, to avoid that tip would match with a git tag or branch, and similarly that HEAD would match a mercurial tag or branch.
Best regards,
Thomas
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-09-27 14:43 [Buildroot] [RFC] package: Redownload HEAD packages every build Clayton Shotwell
2013-09-27 14:51 ` Clayton Shotwell
2013-09-27 15:11 ` Thomas De Schampheleire
@ 2013-10-01 16:08 ` Arnout Vandecappelle
2013-10-01 18:51 ` Thomas Petazzoni
3 siblings, 0 replies; 10+ messages in thread
From: Arnout Vandecappelle @ 2013-10-01 16:08 UTC (permalink / raw)
To: buildroot
On 09/27/13 16:43, Clayton Shotwell wrote:
> Adding a check to remove a downloaded package if the version is HEAD. This causes the package to be re-downloaded with updated software. This feature is very useful during package development.
>
> Signed-off-by: Clayton Shotwell <clshotwe@rockwellcollins.com>
> ---
> package/pkg-generic.mk | 7 +++++++
> 1 files changed, 7 insertions(+), 0 deletions(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index bfc4dc1..859c4a9 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -27,6 +27,13 @@
> # Retrieve the archive
> $(BUILD_DIR)/%/.stamp_downloaded:
> ifeq ($(DL_MODE),DOWNLOAD)
> +# Remove the pre-downloaded tar file if the package is a HEAD version
> + $(Q)( \
> + if test $($(PKG)_VERSION) == HEAD -a -e $(DL_DIR)/$($(PKG)_SOURCE); then \
> + rm $(DL_DIR)/$($(PKG)_SOURCE); \
Use $(RM) instead of rm, then you don't need to check for existence.
> + $(call MESSAGE,"Removing $(PKG) HEAD source file"); \
I don't think this extra message is needed.
> + fi; \
> + )
> # Only show the download message if it isn't already downloaded
> $(Q)if test ! -e $(DL_DIR)/$($(PKG)_SOURCE); then \
> $(call MESSAGE,"Downloading") ; \
>
You'll also need to avoid touching the stamp file after the download,
otherwise the rule is not even evaluated.
Also I think I would prefer instead to have an explicit variable to
indicate that it should be re-downloaded, e.g.
ifneq ($($(PKG)_FORCE_DOWNLOAD),)
$(Q)$(RM) $(DL_DIR)/$($(PKG)_SOURCE)
endif
This variable can be set either from the package's .mk file or from the
local override file.
And finally, the documentation should be updated as well.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-09-27 14:43 [Buildroot] [RFC] package: Redownload HEAD packages every build Clayton Shotwell
` (2 preceding siblings ...)
2013-10-01 16:08 ` Arnout Vandecappelle
@ 2013-10-01 18:51 ` Thomas Petazzoni
2013-10-01 20:01 ` clshotwe at rockwellcollins.com
3 siblings, 1 reply; 10+ messages in thread
From: Thomas Petazzoni @ 2013-10-01 18:51 UTC (permalink / raw)
To: buildroot
Dear Clayton Shotwell,
On Fri, 27 Sep 2013 09:43:35 -0500, Clayton Shotwell wrote:
> Adding a check to remove a downloaded package if the version is HEAD. This causes the
> package to be re-downloaded with updated software. This feature is very
> useful during package development.
I haven't made up my mind yet on this, but in general, I'd like to
avoid having several "mechanisms" achieving the same goal. I believe
the <pkg>_OVERRIDE_SRCDIR mechanism is already here to easily allow to
rebuild the latest version of a component source code during its
development.
It seems like the mechanism you're proposing overlaps quite
significantly with that, while being a bit less flexible. Have you
tried the <pkg>_OVERRIDE_SRCDIR mechanism?
Regarding the implementation, I'm not sure to understand how
<pkg>_VERSION = HEAD is supposed to be used. For example, I have a
software, whose source code is handled in Git, in its 'master' branch.
How would I tell your mechanism that I want the latest source code to
be redownloaded each time?
Moreover, it's going to be redownloaded each time completely, which is
quite annoying during development. The <pkg>_OVERRIDE_SRCDIR allows you
to point to a local directory, which avoids re-downloading the entire
source code everytime.
Thanks!
Thomas
--
Thomas Petazzoni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 10+ messages in thread* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-10-01 18:51 ` Thomas Petazzoni
@ 2013-10-01 20:01 ` clshotwe at rockwellcollins.com
2013-10-01 21:37 ` Thomas Petazzoni
0 siblings, 1 reply; 10+ messages in thread
From: clshotwe at rockwellcollins.com @ 2013-10-01 20:01 UTC (permalink / raw)
To: buildroot
Thomas and Arnout,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote on 10/01/2013
01:51:54 PM:
> It seems like the mechanism you're proposing overlaps quite
> significantly with that, while being a bit less flexible. Have you
> tried the <pkg>_OVERRIDE_SRCDIR mechanism?
I did a little digging into the <pkg>_OVERRIDE_SRCDIR mechanism but I
could not find any documentation and the implementation in the make
structure is a little unclear. What is the full intent of that mechanism?
> Regarding the implementation, I'm not sure to understand how
> <pkg>_VERSION = HEAD is supposed to be used. For example, I have a
> software, whose source code is handled in Git, in its 'master' branch.
> How would I tell your mechanism that I want the latest source code to
> be redownloaded each time?
I failed to take git and mercurial into account with this patch.
Subversion uses "HEAD" for its latest source code.
> Moreover, it's going to be redownloaded each time completely, which is
> quite annoying during development. The <pkg>_OVERRIDE_SRCDIR allows you
> to point to a local directory, which avoids re-downloading the entire
> source code everytime.
Arnout Vandecappelle <arnout@mind.be> wrote on 10/01/2013 11:08:23 AM:
> Also I think I would prefer instead to have an explicit variable to
> indicate that it should be re-downloaded, e.g.
>
> ifneq ($($(PKG)_FORCE_DOWNLOAD),)
> $(Q)$(RM) $(DL_DIR)/$($(PKG)_SOURCE)
> endif
I like this approach much better. It would make this mechanism very
explicit and avoid having to handle all of the different repository
nuances. I would like to create a new patch that implements this method.
> This variable can be set either from the package's .mk file or from
the
> local override file.
>
>
> And finally, the documentation should be updated as well.
Agreed.
Thanks,
Clayton
Clayton Shotwell
Software Engineer
clshotwe at rockwellcollins.com
www.rockwellcollins.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131001/48284d2e/attachment.html>
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-10-01 20:01 ` clshotwe at rockwellcollins.com
@ 2013-10-01 21:37 ` Thomas Petazzoni
2013-10-01 21:54 ` Ryan Barnett
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Petazzoni @ 2013-10-01 21:37 UTC (permalink / raw)
To: buildroot
Hello,
On Tue, 1 Oct 2013 15:01:52 -0500, clshotwe at rockwellcollins.com wrote:
> > It seems like the mechanism you're proposing overlaps quite
> > significantly with that, while being a bit less flexible. Have you
> > tried the <pkg>_OVERRIDE_SRCDIR mechanism?
>
> I did a little digging into the <pkg>_OVERRIDE_SRCDIR mechanism but I
> could not find any documentation and the implementation in the make
> structure is a little unclear. What is the full intent of that mechanism?
Either in your package .mk file or in a local.mk file, you can add:
FOO_OVERRIDE_SRCDIR = /path/to/foo/sources/
If you do that, then Buildroot will no longer download/extract/patch
the sources for the foo package, it will directly assume the sources
are located in /path/to/foo/sources/. It will rsync them to the build
directory, and build from there. Whenever you do:
make foo-reconfigure
or
make foo-rebuild
Buildroot will rsync again the source code, and restart the build at
the configuration step, or the build step of the foo package.
Usually, /path/to/foo/sources/ will be the directory where you did your
Subversion checkout or Git clone of your "foo" source code.
Best regards,
Thomas
--
Thomas Petazzoni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-10-01 21:37 ` Thomas Petazzoni
@ 2013-10-01 21:54 ` Ryan Barnett
2013-10-02 7:13 ` Thomas Petazzoni
0 siblings, 1 reply; 10+ messages in thread
From: Ryan Barnett @ 2013-10-01 21:54 UTC (permalink / raw)
To: buildroot
Thomas,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote on
10/01/2013 04:37:39 PM:
> Hello,
>
> On Tue, 1 Oct 2013 15:01:52 -0500, clshotwe at rockwellcollins.com wrote:
>
> > > It seems like the mechanism you're proposing overlaps quite
> > > significantly with that, while being a bit less flexible. Have you
> > > tried the <pkg>_OVERRIDE_SRCDIR mechanism?
> >
> > I did a little digging into the <pkg>_OVERRIDE_SRCDIR mechanism but I
> > could not find any documentation and the implementation in the make
> > structure is a little unclear. What is the full intent of that
mechanism?
>
> Either in your package .mk file or in a local.mk file, you can add:
>
> FOO_OVERRIDE_SRCDIR = /path/to/foo/sources/
>
> If you do that, then Buildroot will no longer download/extract/patch
> the sources for the foo package, it will directly assume the sources
> are located in /path/to/foo/sources/. It will rsync them to the build
> directory, and build from there. Whenever you do:
>
> make foo-reconfigure
>
> or
>
> make foo-rebuild
>
> Buildroot will rsync again the source code, and restart the build at
> the configuration step, or the build step of the foo package.
>
> Usually, /path/to/foo/sources/ will be the directory where you did your
> Subversion checkout or Git clone of your "foo" source code.
Could you add this explanation to buildroot manual since it looks like
<pkg>_OVERRIDE_SRCDIR mechanism is your feature and it is currently
lacking in documentation in the manual? I too have tried to understand
how this feature worked and it was a bit unclear.
The only mention of OVERRIDE_SRCDIR in the documentation is under the
rebuild and reconfigure commands. Where is says that it that it only
makes sense to use these features with OVERRIDE_SRCDIR but doesn't
explain what you just explained above.
Thanks,
-Ryan
------------------------------------------------------------------------------------------
Ryan J Barnett / Software Engineer / Platform SW
MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US
rjbarnet at rockwellcollins.com
www.rockwellcollins.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-10-01 21:54 ` Ryan Barnett
@ 2013-10-02 7:13 ` Thomas Petazzoni
2013-10-02 18:34 ` clshotwe at rockwellcollins.com
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Petazzoni @ 2013-10-02 7:13 UTC (permalink / raw)
To: buildroot
Dear Ryan Barnett,
On Tue, 1 Oct 2013 16:54:04 -0500, Ryan Barnett wrote:
> > Buildroot will rsync again the source code, and restart the build at
> > the configuration step, or the build step of the foo package.
> >
> > Usually, /path/to/foo/sources/ will be the directory where you did your
> > Subversion checkout or Git clone of your "foo" source code.
>
> Could you add this explanation to buildroot manual since it looks like
> <pkg>_OVERRIDE_SRCDIR mechanism is your feature and it is currently
> lacking in documentation in the manual? I too have tried to understand
> how this feature worked and it was a bit unclear.
>
> The only mention of OVERRIDE_SRCDIR in the documentation is under the
> rebuild and reconfigure commands. Where is says that it that it only
> makes sense to use these features with OVERRIDE_SRCDIR but doesn't
> explain what you just explained above.
Right, I will submit a documentation patch.
Thanks!
Thomas
--
Thomas Petazzoni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Buildroot] [RFC] package: Redownload HEAD packages every build
2013-10-02 7:13 ` Thomas Petazzoni
@ 2013-10-02 18:34 ` clshotwe at rockwellcollins.com
0 siblings, 0 replies; 10+ messages in thread
From: clshotwe at rockwellcollins.com @ 2013-10-02 18:34 UTC (permalink / raw)
To: buildroot
Thomas,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote on 10/02/2013
02:13:49 AM:
> > > Buildroot will rsync again the source code, and restart the build at
> > > the configuration step, or the build step of the foo package.
> > >
> > > Usually, /path/to/foo/sources/ will be the directory where you did
your
> > > Subversion checkout or Git clone of your "foo" source code.
> >
> > Could you add this explanation to buildroot manual since it looks like
> > <pkg>_OVERRIDE_SRCDIR mechanism is your feature and it is currently
> > lacking in documentation in the manual? I too have tried to understand
> > how this feature worked and it was a bit unclear.
> >
> > The only mention of OVERRIDE_SRCDIR in the documentation is under the
> > rebuild and reconfigure commands. Where is says that it that it only
> > makes sense to use these features with OVERRIDE_SRCDIR but doesn't
> > explain what you just explained above.
>
> Right, I will submit a documentation patch.
This sounds like a much better method for my development. Thanks for
explaining it.
Thanks,
Clayton
Clayton Shotwell
Software Engineer
clshotwe at rockwellcollins.com
www.rockwellcollins.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131002/1353fa91/attachment.html>
^ permalink raw reply [flat|nested] 10+ messages in thread