improving display of build artifacts

improving display of build artifacts

[Barros Pena, Belen]

We have a few issues in Bugzilla connected to the display of build
artifacts:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8563
https://bugzilla.yoctoproject.org/show_bug.cgi?id=9023
https://bugzilla.yoctoproject.org/show_bug.cgi?id=9594

I just want to list what the requirements are all together, so that it is
easier to discuss what's feasible and what is harder to do.

Build artifacts (excluding logs) are presented in the build dashboard in 2
mains areas:

1. an 'image files' list per image recipe built in a build
2. an 'other artifacts' section per build

So we can break down the requirements per area:

1. Requirements for the 'image files' area

1.1. All image files should show up (right now we are missing at least the
live image types), bearing in mind that custom image types can exist in
layers (e.g. sdcard).

1.2 Image files should be associated with the correct image recipe
(important in multi-target builds). Right now this is not quite there: to
see what happens, create a new project and build core-image-minimal and
core-image-minimal-dev in the same build.

1.3 When an image file is not generated for a certain image recipe
(because a valid image file already exists that was produced by a previous
build), the 'image files' area should show the valid image files produced
by the older build

1.4 The rootfs.manifest files should be associated to their image recipe
and display in the 'image files' area (see
https://bugzilla.yoctoproject.org/attachment.cgi?id=2957)

1.5 SDK-related artifacts should be associated with their image recipe and
display in the 'image files' area (right now they show in the 'other
artifacts' area). 

2. Requirements for the 'other artifacts' area

2.1. No image files, rootfs manifests or sdk-related files should ever
show in the 'other artifacts' area

2.2. The README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt file should
not show up in the 'other artifacts' area

2.3. The 'other artifacts' section should show the list of all files in
the correct 

/build-toaster-<project-di>/tmp/deploy/images/<machine>/

directory that are *not* image files, rootfs manifests or sdk-related
files.

So, from the above, what's relatively easy to do and what will cause us
trouble?

Thanks!

Belén

Re: improving display of build artifacts

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 3473 bytes --]

Thanks for putting this together.

Some of these should be relatively easy (like 1.1, 1.2, 2.2).

(NB there is also https://bugzilla.yoctoproject.org/show_bug.cgi?id=8556,
which is really the same bug: image_fstypes *is* being passed, but we don't
detect the output vmdk file correctly. This should also be relatively easy
to fix.)

However, the fundamental problem at the moment is that we don't get a
precise notification about which files are built when, or about which
targets they belong to. We rely on trying to guess and erring on the side
of caution, which is why we don't show files we have previously associated
with a build.

I am going to spend some time trying to figure out a way to provide this
information, as anything we do without this is going to be prone to error
and guesswork (which is why it works as it does at the moment).

Elliot

On 24 May 2016 at 18:07, Barros Pena, Belen <belen.barros.pena@intel.com>
wrote:

> We have a few issues in Bugzilla connected to the display of build
> artifacts:
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8563
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=9023
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=9594
>
> I just want to list what the requirements are all together, so that it is
> easier to discuss what's feasible and what is harder to do.
>
> Build artifacts (excluding logs) are presented in the build dashboard in 2
> mains areas:
>
> 1. an 'image files' list per image recipe built in a build
> 2. an 'other artifacts' section per build
>
> So we can break down the requirements per area:
>
> 1. Requirements for the 'image files' area
>
> 1.1. All image files should show up (right now we are missing at least the
> live image types), bearing in mind that custom image types can exist in
> layers (e.g. sdcard).
>
> 1.2 Image files should be associated with the correct image recipe
> (important in multi-target builds). Right now this is not quite there: to
> see what happens, create a new project and build core-image-minimal and
> core-image-minimal-dev in the same build.
>
> 1.3 When an image file is not generated for a certain image recipe
> (because a valid image file already exists that was produced by a previous
> build), the 'image files' area should show the valid image files produced
> by the older build
>
> 1.4 The rootfs.manifest files should be associated to their image recipe
> and display in the 'image files' area (see
> https://bugzilla.yoctoproject.org/attachment.cgi?id=2957)
>
> 1.5 SDK-related artifacts should be associated with their image recipe and
> display in the 'image files' area (right now they show in the 'other
> artifacts' area).
>
> 2. Requirements for the 'other artifacts' area
>
> 2.1. No image files, rootfs manifests or sdk-related files should ever
> show in the 'other artifacts' area
>
> 2.2. The README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt file should
> not show up in the 'other artifacts' area
>
> 2.3. The 'other artifacts' section should show the list of all files in
> the correct
>
> /build-toaster-<project-di>/tmp/deploy/images/<machine>/
>
> directory that are *not* image files, rootfs manifests or sdk-related
> files.
>
> So, from the above, what's relatively easy to do and what will cause us
> trouble?
>
> Thanks!
>
> Belén
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 4799 bytes --]

Re: improving display of build artifacts

[Richard Purdie]

On Wed, 2016-05-25 at 09:28 +0100, Smith, Elliot wrote:
> Thanks for putting this together.
> 
> Some of these should be relatively easy (like 1.1, 1.2, 2.2).
> 
> (NB there is also 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8556, which is
> really the same bug: image_fstypes *is* being passed, but we don't
> detect the output vmdk file correctly. This should also be relatively
> easy to fix.)
> 
> However, the fundamental problem at the moment is that we don't get a
> precise notification about which files are built when, or about which
> targets they belong to. We rely on trying to guess and erring on the
> side of caution, which is why we don't show files we have previously
> associated with a build.
> 
> I am going to spend some time trying to figure out a way to provide
> this information, as anything we do without this is going to be prone
> to error and guesswork (which is why it works as it does at the
> moment).

I can give a braindump here which may or may not help.

In the cases where we need to track artefacts in general we use sstate.
sstate works by providing an output directory which the task writes
data to, then sstate itself moves the files into the final position.

We use sstate for deploy, package, package_write_*, packagedata and so
on but one thing we don't use it for are images. Why?

Images are large and we don't want the overhead of compressing the
image into an sstate tarball or the overhead of downloading such sstate
artefacts when its likely cheaper to build the image.

There is one task we take a "half way house" approach with which is the
kernel source population. Here, it behaves like an sstate task from a
dependency graph point of view but its generally cheaper to run the
build rather than compress and extract it.

I have wondered if we should make images behave a bit more like sstate,
in that they could write to an intermediate directory which is then
moved into position.

That way, toaster could know that a set of files come from a given task
for sure.

Its obviously a pretty invasive change in behaviour but also partly in
keeping with the rework of the image generation we've been pursusing.

Is it just the do_image_* output we have the issue with?

Cheers,

Richard

Re: improving display of build artifacts

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

Thanks for the response, Richard.

On 25 May 2016 at 10:25, Richard Purdie <richard.purdie@linuxfoundation.org>
wrote:

> ...
>
> I have wondered if we should make images behave a bit more like sstate,
> in that they could write to an intermediate directory which is then
> moved into position.
>
> That way, toaster could know that a set of files come from a given task
> for sure.
>
> Its obviously a pretty invasive change in behaviour but also partly in
> keeping with the rework of the image generation we've been pursusing.
>
> Is it just the do_image_* output we have the issue with?
>

We need to know about all of the files in tmp/deploy/images/${MACHINE},
both the images and any other files which are produced.

We also need to know about SDK artifacts (I can't recall if they end up
there or somewhere else).

I have had a quick look at buildhistory.bbclass, and could use that to
write some useful stuff out to a file in the buildhistory directory, such
as the image name and image fstypes. I could then figure out from that
which image files were produced by the build, which I think would be an
improvement over what we have.

However, we still have the issue of associated artifact files and how we
get those. In particular, we need to be able to tell when a file would have
been produced if it wasn't already there. Could we use sstate for that?

Cheers,
Elliot


>
> Cheers,
>
> Richard
>
>
>


-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2330 bytes --]