Re: improving display of build artifacts

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2016-05-25 at 09:28 +0100, Smith, Elliot wrote:
> Thanks for putting this together.
> 
> Some of these should be relatively easy (like 1.1, 1.2, 2.2).
> 
> (NB there is also 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8556, which is
> really the same bug: image_fstypes *is* being passed, but we don't
> detect the output vmdk file correctly. This should also be relatively
> easy to fix.)
> 
> However, the fundamental problem at the moment is that we don't get a
> precise notification about which files are built when, or about which
> targets they belong to. We rely on trying to guess and erring on the
> side of caution, which is why we don't show files we have previously
> associated with a build.
> 
> I am going to spend some time trying to figure out a way to provide
> this information, as anything we do without this is going to be prone
> to error and guesswork (which is why it works as it does at the
> moment).

I can give a braindump here which may or may not help.

In the cases where we need to track artefacts in general we use sstate.
sstate works by providing an output directory which the task writes
data to, then sstate itself moves the files into the final position.

We use sstate for deploy, package, package_write_*, packagedata and so
on but one thing we don't use it for are images. Why?

Images are large and we don't want the overhead of compressing the
image into an sstate tarball or the overhead of downloading such sstate
artefacts when its likely cheaper to build the image.

There is one task we take a "half way house" approach with which is the
kernel source population. Here, it behaves like an sstate task from a
dependency graph point of view but its generally cheaper to run the
build rather than compress and extract it.

I have wondered if we should make images behave a bit more like sstate,
in that they could write to an intermediate directory which is then
moved into position.

That way, toaster could know that a set of files come from a given task
for sure.

Its obviously a pretty invasive change in behaviour but also partly in
keeping with the rework of the image generation we've been pursusing.

Is it just the do_image_* output we have the issue with?

Cheers,

Richard