From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Petazzoni Date: Wed, 8 Apr 2020 08:39:03 +0200 Subject: [Buildroot] [PATCH] infra: add the transient download mechanism In-Reply-To: <20200115203753.12110-1-yann.morin.1998@free.fr> References: <20200115203753.12110-1-yann.morin.1998@free.fr> Message-ID: <20200408083903.547003f2@windsurf.home> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: buildroot@busybox.net Hello, On Wed, 15 Jan 2020 21:37:53 +0100 "Yann E. MORIN" wrote: > NOTE: NOT TO BE COMMITTED! > > This patch is a proof of concept for the transient download mechanism, > that would help using branches as versions, and keep using the head of > the branch each time a build started. So what is the status of this patch? I see Yann you marked it as to be ignored by patchwork, so we don't have it in patchwork. I think there's generally been some positive feedback, with Arnout saying: "" Bottom line: even though repo is the better tool to manage the trees both for development and release, the transient mechanism is a good quick win. "" > A package declares its download as transient with: > FOO_DOWNLOAD_TRANSIENT = YES > > this means that the download infra will not use any already downloaded > archive, and will instead always download it as if missing. > > Since the check is done in the download wrapper, we have no TOCTOU race > in case two bulds would attempt the same transient download: the archive > is only replaced ato,mically as usual. Typo: atomically > So, if the package uses a branch as version, the branch's HEAD at that > very moment will be redownloaded. > > This stil has the drawback that two builds in parallel may get slightly > different content for the same branch, and the first build could end up > using the download of the second build: > > build-1 build-2 > > download > | download > | | > save to dl-dir | > [...] save to dl-dir > extract That is not a drawback, that is exactly what you expect with this feature. > Furthermore, even with a single build, it might get what it expects: > > developer-1 developer-2 > > git push branch > trigger CI git push branch > [...] > download > > In that case the build of delopper-1 would get the code of developper-2 > who pushed on the same branch. Ditto here: this is exactly what we expect for a transient download feature. If developer-1 wants to test only his own feature, then he has to trigger CI on a branch that only he has control over. There is however one drawback: * You run a build (1), which downloads package A from a Git repo, and places the tarball in global DL_DIR. * You run a different build (2), which downloads the same package A from the Git repo. Due to the transient download mechanism being enabled for package A, it re-downloads it, and places an updated tarball in the global DL_DIR. * Back to build (1), you run "make legal-info", which picks up the updated tarball, which was generated by build (2). So, in build (1), your image uses the "previous" version of package A, while your legal-info contains the "new" version of package A. To solve this, one solution would be to use a per-build download directory for the transient packages, rather than the global DL_DIR. But I think this could be a separate improvement. Indeed, the transient download mechanism is clearly oriented towards development/CI workflow, where consistency of legal-info may not be a primary concern. For a final release build, you would not use TRANSIENT_DOWNLOAD = YES. For example, custom packages that make use of TRANSIENT_DOWNLOAD could be implemented like this: ifeq ($(BR2_DEVELOPER_MODE),y) _TRANSIENT_DOWNLOAD = YES _VERSION = some-branch else _VERSION = some-tag endif So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y, but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures we have a fully reproducible build, and consistent legal-info output. Yann, do you think you could resubmit your patch, for actual merging ? Of course, we don't need the busybox.mk change. Best regards, Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com