[Buildroot] [PATCH] infra: add the transient download mechanism

[Thomas Petazzoni <thomas.petazzoni@bootlin.com>]

Hello,

On Wed, 15 Jan 2020 21:37:53 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> NOTE: NOT TO BE COMMITTED!
> 
> This patch is a proof of concept for the transient download mechanism,
> that would help using branches as versions, and keep using the head of
> the branch each time a build started.

So what is the status of this patch? I see Yann you marked it as to be
ignored by patchwork, so we don't have it in patchwork.

I think there's generally been some positive feedback, with Arnout
saying:

""
 Bottom line: even though repo is the better tool to manage the trees both for
development and release, the transient mechanism is a good quick win.
""

> A package declares its download as transient with:
>   FOO_DOWNLOAD_TRANSIENT = YES
> 
> this means that the download infra will not use any already downloaded
> archive, and will instead always download it as if missing.
> 
> Since the check is done in the download wrapper, we have no TOCTOU race
> in case two bulds would attempt the same transient download: the archive
> is only replaced ato,mically as usual.

Typo: atomically

> So, if the package uses a branch as version, the branch's HEAD at that
> very moment will be redownloaded.
> 
> This stil has the drawback that two builds in parallel may get slightly
> different content for the same branch, and the first build could end up
> using the download of the second build:
> 
>     build-1             build-2
> 
>     download
>     |                   download
>     |                   |
>     save to dl-dir      |
>     [...]               save to dl-dir
>     extract

That is not a drawback, that is exactly what you expect with this
feature.

> Furthermore, even with a single build, it might get what it expects:
> 
>     developer-1         developer-2
> 
>     git push branch
>     trigger CI          git push branch
>     [...]
>     download
> 
> In that case the build of delopper-1 would get the code of developper-2
> who pushed on the same branch.

Ditto here: this is exactly what we expect for a transient download
feature. If developer-1 wants to test only his own feature, then he has
to trigger CI on a branch that only he has control over.

There is however one drawback:

 * You run a build (1), which downloads package A from a Git repo, and
   places the tarball in global DL_DIR.

 * You run a different build (2), which downloads the same package A
   from the Git repo. Due to the transient download mechanism being
   enabled for package A, it re-downloads it, and places an updated
   tarball in the global DL_DIR.

 * Back to build (1), you run "make legal-info", which picks up the
   updated tarball, which was generated by build (2).

So, in build (1), your image uses the "previous" version of package A,
while your legal-info contains the "new" version of package A.

To solve this, one solution would be to use a per-build download
directory for the transient packages, rather than the global DL_DIR.

But I think this could be a separate improvement. Indeed, the transient
download mechanism is clearly oriented towards development/CI workflow,
where consistency of legal-info may not be a primary concern. For a
final release build, you would not use TRANSIENT_DOWNLOAD = YES.

For example, custom packages that make use of TRANSIENT_DOWNLOAD could
be implemented like this:

ifeq ($(BR2_DEVELOPER_MODE),y)
<pkg>_TRANSIENT_DOWNLOAD = YES
<pkg>_VERSION = some-branch
else
<pkg>_VERSION = some-tag
endif

So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y,
but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures
we have a fully reproducible build, and consistent legal-info output.

Yann, do you think you could resubmit your patch, for actual merging ?
Of course, we don't need the busybox.mk change.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com