* [Buildroot] cached downloads for scm branches @ 2016-10-20 17:27 Michael Walle 2016-10-20 19:38 ` Thomas Petazzoni 2016-10-20 19:45 ` Yann E. MORIN 0 siblings, 2 replies; 6+ messages in thread From: Michael Walle @ 2016-10-20 17:27 UTC (permalink / raw) To: buildroot Hi, If I have a package which downloads it's source from a SCM branch, the download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This makes sense if the version is a tag or a (git) hash. But if used with branches, the package will never be downloaded again until I remove it from the DL_DIR. This may be the intended behaviour, but as far as I can tell, I have no chance to change it. Do I miss something? Does it make sense to add a new package variable to force a download? -michael ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] cached downloads for scm branches 2016-10-20 17:27 [Buildroot] cached downloads for scm branches Michael Walle @ 2016-10-20 19:38 ` Thomas Petazzoni 2016-10-20 20:29 ` Michael Walle 2016-10-20 19:45 ` Yann E. MORIN 1 sibling, 1 reply; 6+ messages in thread From: Thomas Petazzoni @ 2016-10-20 19:38 UTC (permalink / raw) To: buildroot Hello, On Thu, 20 Oct 2016 19:27:19 +0200, Michael Walle wrote: > If I have a package which downloads it's source from a SCM branch, the > download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This > makes sense if the version is a tag or a (git) hash. But if used with > branches, the package will never be downloaded again until I remove it > from the DL_DIR. This may be the intended behaviour, but as far as I can > tell, I have no chance to change it. > > Do I miss something? Does it make sense to add a new package variable to > force a download? This behavior is intended and adding a new package variable to force a download is not what we want to do. Indeed, Buildroot aims at doing *reproducible* builds. If a package points to a branch, then you have no idea what you are building, this what you're building depends on the moment you start the build (as it decides when you will fetch, and therefore what you will fetch). *However*, we have a mechanism called <pkg>_OVERRIDE_SRCDIR, which allows you to tell Buildroot: "For this specific package, please skip the download/extract/patch steps, and instead take the source from <this> local directory". Thanks to that, you can checkout/clone your project source code in <this> folder, and ask Buildroot to fetch it from there. This is what we recommend during active development on a given component. See http://free-electrons.com/doc/training/buildroot/buildroot-slides.pdf slides 261 and following for more details. Best regards, Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] cached downloads for scm branches 2016-10-20 19:38 ` Thomas Petazzoni @ 2016-10-20 20:29 ` Michael Walle 2016-10-21 11:46 ` Jérôme Pouiller 2016-10-21 18:09 ` Arnout Vandecappelle 0 siblings, 2 replies; 6+ messages in thread From: Michael Walle @ 2016-10-20 20:29 UTC (permalink / raw) To: buildroot hi, thanks. yeah i thought so. Just to give you some background regarding my question. We have some kind of CI, actually it's rather a daily build. we build a root filesystem with some fixed packages, but also with multiple packages of our own and we want to track the tip of a branch for a build (because it is CI ;)). I don't think the src override method really suit the CI case. I guess one way could be that the CI defines the commit that should be build. Or simply delete the download. mhh ;) -michael sorry for the tofu. i'm on my mobile. Am 20. Oktober 2016 21:38:41 MESZ, schrieb Thomas Petazzoni <thomas.petazzoni@free-electrons.com>: >Hello, > >On Thu, 20 Oct 2016 19:27:19 +0200, Michael Walle wrote: > >> If I have a package which downloads it's source from a SCM branch, >the >> download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This > >> makes sense if the version is a tag or a (git) hash. But if used with > >> branches, the package will never be downloaded again until I remove >it >> from the DL_DIR. This may be the intended behaviour, but as far as I >can >> tell, I have no chance to change it. >> >> Do I miss something? Does it make sense to add a new package variable >to >> force a download? > >This behavior is intended and adding a new package variable to force a >download is not what we want to do. Indeed, Buildroot aims at doing >*reproducible* builds. If a package points to a branch, then you have >no idea what you are building, this what you're building depends on the >moment you start the build (as it decides when you will fetch, and >therefore what you will fetch). > >*However*, we have a mechanism called <pkg>_OVERRIDE_SRCDIR, which >allows you to tell Buildroot: "For this specific package, please skip >the download/extract/patch steps, and instead take the source from ><this> local directory". Thanks to that, you can checkout/clone your >project source code in <this> folder, and ask Buildroot to fetch it >from there. This is what we recommend during active development on a >given component. > >See >http://free-electrons.com/doc/training/buildroot/buildroot-slides.pdf >slides 261 and following for more details. > >Best regards, > >Thomas >-- >Thomas Petazzoni, CTO, Free Electrons >Embedded Linux and Kernel engineering >http://free-electrons.com >_______________________________________________ >buildroot mailing list >buildroot at busybox.net >http://lists.busybox.net/mailman/listinfo/buildroot -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20161020/41bb724b/attachment.html> ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] cached downloads for scm branches 2016-10-20 20:29 ` Michael Walle @ 2016-10-21 11:46 ` Jérôme Pouiller 2016-10-21 18:09 ` Arnout Vandecappelle 1 sibling, 0 replies; 6+ messages in thread From: Jérôme Pouiller @ 2016-10-21 11:46 UTC (permalink / raw) To: buildroot Hello Michael, On Thursday 20 October 2016 22:29:15 Michael Walle wrote: > hi, > > thanks. yeah i thought so. Just to give you some background regarding > my question. We have some kind of CI, actually it's rather a daily > build. we build a root filesystem with some fixed packages, but also > with multiple packages of our own and we want to track the tip of a > branch for a build (because it is CI ;)). I don't think the src > override method really suit the CI case. I guess one way could be > that the CI defines the commit that should be build. Or simply delete > the download. mhh ;) I suggest to pass package version to command line: make MY_PKG_VERSION=$(git ls-remote http://repo.git/path master) all You may also force versions using local.mk: override MY_PKG_VERSION = $(shell git ls-remote http://repo.git/path master) However, these ways are not officially supported. Consider them as hacks. BR, -- J?r?me Pouiller ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] cached downloads for scm branches 2016-10-20 20:29 ` Michael Walle 2016-10-21 11:46 ` Jérôme Pouiller @ 2016-10-21 18:09 ` Arnout Vandecappelle 1 sibling, 0 replies; 6+ messages in thread From: Arnout Vandecappelle @ 2016-10-21 18:09 UTC (permalink / raw) To: buildroot On 20-10-16 22:29, Michael Walle wrote: > thanks. yeah i thought so. Just to give you some background regarding my > question. We have some kind of CI, actually it's rather a daily build. we build > a root filesystem with some fixed packages, but also with multiple packages of > our own and we want to track the tip of a branch for a build (because it is CI > ;)). I don't think the src override method really suit the CI case. I guess one > way could be that the CI defines the commit that should be build. Or simply > delete the download. mhh ;) Actually, the src override method fits the CI case perfectly because CI can watch the repositories to be overridden and check them out. Regards, Arnout -- Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286500 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] cached downloads for scm branches 2016-10-20 17:27 [Buildroot] cached downloads for scm branches Michael Walle 2016-10-20 19:38 ` Thomas Petazzoni @ 2016-10-20 19:45 ` Yann E. MORIN 1 sibling, 0 replies; 6+ messages in thread From: Yann E. MORIN @ 2016-10-20 19:45 UTC (permalink / raw) To: buildroot Michael, All, On 2016-10-20 19:27 +0200, Michael Walle spake thusly: > Hi, > > If I have a package which downloads it's source from a SCM branch, the > download will be cached in DL_DIR/<pkgname>-<branchname>.tar.gz. This makes > sense if the version is a tag or a (git) hash. But if used with branches, > the package will never be downloaded again until I remove it from the > DL_DIR. This may be the intended behaviour, but as far as I can tell, I have > no chance to change it. Yes, this is the intended behaviour: you should not use a branch names or other unstable references Using a branch name makes the build non-reproducible, and that is not good. Only use stable references, like a tag or a sha1 or ... If however you are actively working on the package, then you should use the source-override mechanism instead: - create a local.mk file (in the directory where you ran 'make menuconfig') - in that file, add this line MY_PACKAGE_OVERRIDE_SRCDIR = /path/to/my-package Then you can manage that location "by hand" (like pulling new changes...) and have it automatically used by Buildroot: make my-package-rebuild (and other such commands, see "make help" and the manual for the complete set.) Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-10-21 18:09 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-10-20 17:27 [Buildroot] cached downloads for scm branches Michael Walle 2016-10-20 19:38 ` Thomas Petazzoni 2016-10-20 20:29 ` Michael Walle 2016-10-21 11:46 ` Jérôme Pouiller 2016-10-21 18:09 ` Arnout Vandecappelle 2016-10-20 19:45 ` Yann E. MORIN
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox