[Buildroot] [PATCH] infra: add the transient download mechanism

[Yann E. MORIN <yann.morin.1998@free.fr>]

Thomas, All,

On 2020-04-08 08:39 +0200, Thomas Petazzoni spake thusly:
> On Wed, 15 Jan 2020 21:37:53 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > This patch is a proof of concept for the transient download mechanism,
> > that would help using branches as versions, and keep using the head of
> > the branch each time a build started.
> 
> So what is the status of this patch? I see Yann you marked it as to be
> ignored by patchwork, so we don't have it in patchwork.
> 
> I think there's generally been some positive feedback, with Arnout
> saying:
> 
> ""
>  Bottom line: even though repo is the better tool to manage the trees both for
> development and release, the transient mechanism is a good quick win.
> ""

I will address all the questions and comments in the review, and will
respin it for proper consideration.

[--SNIP--]
> > This stil has the drawback that two builds in parallel may get slightly
> > different content for the same branch, and the first build could end up
> > using the download of the second build:
> > 
> >     build-1             build-2
> > 
> >     download
> >     |                   download
> >     |                   |
> >     save to dl-dir      |
> >     [...]               save to dl-dir
> >     extract
> 
> That is not a drawback, that is exactly what you expect with this
> feature.

Sorry, the two builds are supposed to be running in parallel... For
build 1, it would get the download from build 2, which is not what build
1 expects.

But drawback is not the correct word. This is a pitfall.

> > Furthermore, even with a single build, it might get what it expects:
> > 
> >     developer-1         developer-2
> > 
> >     git push branch
> >     trigger CI          git push branch
> >     [...]
> >     download
> > 
> > In that case the build of delopper-1 would get the code of developper-2
> > who pushed on the same branch.
> 
> Ditto here: this is exactly what we expect for a transient download
> feature. If developer-1 wants to test only his own feature, then he has
> to trigger CI on a branch that only he has control over.

Yes, this is not a drawback, but a pitfall.

> There is however one drawback:
> 
>  * You run a build (1), which downloads package A from a Git repo, and
>    places the tarball in global DL_DIR.
> 
>  * You run a different build (2), which downloads the same package A
>    from the Git repo. Due to the transient download mechanism being
>    enabled for package A, it re-downloads it, and places an updated
>    tarball in the global DL_DIR.
> 
>  * Back to build (1), you run "make legal-info", which picks up the
>    updated tarball, which was generated by build (2).

This is not a drawback; this is another pitfall.

> So, in build (1), your image uses the "previous" version of package A,
> while your legal-info contains the "new" version of package A.
> 
> To solve this, one solution would be to use a per-build download
> directory for the transient packages, rather than the global DL_DIR.

Meh... Or rather, make legal-info impossible for a package that is
transient, like when it is 'local'. This is the best solution.

> But I think this could be a separate improvement. Indeed, the transient
> download mechanism is clearly oriented towards development/CI workflow,
> where consistency of legal-info may not be a primary concern. For a
> final release build, you would not use TRANSIENT_DOWNLOAD = YES.
> 
> For example, custom packages that make use of TRANSIENT_DOWNLOAD could
> be implemented like this:
> 
> ifeq ($(BR2_DEVELOPER_MODE),y)
> <pkg>_TRANSIENT_DOWNLOAD = YES
> <pkg>_VERSION = some-branch
> else
> <pkg>_VERSION = some-tag
> endif

The more I think about it, the less proud I am to have come up with this
transient idea. Using unstable references is complete madness to begin
with and is so flawed by design... We should not even provide such a
feature. But we can't stop people from being... people... :-[

> So, while in development mode, users and CI use BR2_DEVELOPER_MODE=y,
> but for final releases, BR2_DEVELOPER_MODE is disabled, which ensures
> we have a fully reproducible build, and consistent legal-info output.
> 
> Yann, do you think you could resubmit your patch, for actual merging ?

Yes I will.

> Of course, we don't need the busybox.mk change.

We don't? How comne" ;-)

> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'