[PATCH 0/2] toaster: Better SDK artifacts and fstypes V2

[PATCH 0/2] toaster: Better SDK artifacts and fstypes V2

[bavery]

This is part of a patch set that is split over the openembedded-core and
bitbake-devel mailing lists.  This is a second version of the patch.

These patches remove complexity from the toaster.bbclass in order to move those
functions to bitbake/lib/toaster.

The overall intent of this set is to improve Toaster's ability to locate artifacts,
specify fstypes correctly, and add some tests for these changes.

-brian avery
an Intel employee


The following changes since commit 0659b4f780120b1459b068c85794be4f04f943bd:

  toaster: fix typo in arguments for libtoaster.js function (2016-07-12 14:12:16 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib bavery/submit/elliot/toaster/8556-image_fstypesV2_foroe
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/elliot/toaster/8556-image_fstypesV2_foroe

Elliot Smith (2):
  toaster.bbclass: remove directory scan logic for detecting artifacts
  toaster.bbclass: only scan files-in-image.txt if it exists

 classes/toaster.bbclass | 99 ++++++++++++-------------------------------------
 1 file changed, 24 insertions(+), 75 deletions(-)

--
1.9.1

[PATCH 1/2] toaster.bbclass: remove directory scan logic for detecting artifacts

[bavery]

From: Elliot Smith <elliot.smith@intel.com>

toaster.bbclass does a scan of the image deploy and SDK directories
when a build finishes. However, this brings no benefit and could be
better managed and made easier to modify if moved to toasterui and
carried out when the BuildCompleted event occurs.

Remove the image scan code from toaster.bbclass, prior to moving it
to toasterui and buildinfohelper.

Also remove the license manifest update code, as this can also be
done from toasterui.

The postfuncs for do_populate_sdk are retained, but no longer
do the directory scan for SDK artifacts. Instead, they fire
an event with the value of the TOOLCHAIN_OUTPUTNAME variable,
as this is only accessible at the point when the do_populate_sdk
and do_populate_sdk_ext tasks are run. The value of this can then
be used by buildinfohelper to find the SDK artifacts produced by a
target.

[YOCTO #9002]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
---
 classes/toaster.bbclass | 74 ++++++-------------------------------------------
 1 file changed, 8 insertions(+), 66 deletions(-)

diff --git a/classes/toaster.bbclass b/classes/toaster.bbclass
index 1878fe0..90ea563 100644
--- a/classes/toaster.bbclass
+++ b/classes/toaster.bbclass
@@ -136,60 +136,16 @@ python toaster_package_dumpdata() {
 
 # 2. Dump output image files information
 
-python toaster_image_dumpdata() {
-    """
-    Image filename for output images is not standardized.
-    image_types.bbclass will spell out IMAGE_CMD_xxx variables that actually
-    have hardcoded ways to create image file names in them.
-    So we look for files starting with the set name.
-
-    We also look for other files in the images/ directory which don't
-    match IMAGE_NAME, such as the kernel bzImage, modules tarball etc.
-    """
-
-    dir_to_walk = d.getVar('DEPLOY_DIR_IMAGE', True);
-    image_name = d.getVar('IMAGE_NAME', True);
-    image_info_data = {}
-    artifact_info_data = {}
-
-    # collect all images and artifacts in the images directory
-    for dirpath, dirnames, filenames in os.walk(dir_to_walk):
-        for filename in filenames:
-            full_path = os.path.join(dirpath, filename)
-            try:
-                if filename.startswith(image_name):
-                    # image
-                    image_info_data[full_path] = os.stat(full_path).st_size
-                else:
-                    # other non-image artifact
-                    if not os.path.islink(full_path):
-                        artifact_info_data[full_path] = os.stat(full_path).st_size
-            except OSError as e:
-                bb.event.fire(bb.event.MetadataEvent("OSErrorException", e), d)
-
-    bb.event.fire(bb.event.MetadataEvent("ImageFileSize", image_info_data), d)
-    bb.event.fire(bb.event.MetadataEvent("ArtifactFileSize", artifact_info_data), d)
-}
-
 python toaster_artifact_dumpdata() {
     """
-    Dump data about artifacts in the SDK_DEPLOY directory
+    Dump data about SDK variables
     """
 
-    dir_to_walk = d.getVar("SDK_DEPLOY", True)
-    artifact_info_data = {}
-
-    # collect all artifacts in the sdk directory
-    for dirpath, dirnames, filenames in os.walk(dir_to_walk):
-        for filename in filenames:
-            full_path = os.path.join(dirpath, filename)
-            try:
-                if not os.path.islink(full_path):
-                    artifact_info_data[full_path] = os.stat(full_path).st_size
-            except OSError as e:
-                bb.event.fire(bb.event.MetadataEvent("OSErrorException", e), d)
+    event_data = {
+      "TOOLCHAIN_OUTPUTNAME": d.getVar("TOOLCHAIN_OUTPUTNAME", True)
+    }
 
-    bb.event.fire(bb.event.MetadataEvent("ArtifactFileSize", artifact_info_data), d)
+    bb.event.fire(bb.event.MetadataEvent("SDKArtifactInfo", event_data), d)
 }
 
 # collect list of buildstats files based on fired events; when the build completes, collect all stats and fire an event with collected data
@@ -361,17 +317,6 @@ python toaster_buildhistory_dump() {
 
 }
 
-# dump information related to license manifest path
-
-python toaster_licensemanifest_dump() {
-    deploy_dir = d.getVar('DEPLOY_DIR', True);
-    image_name = d.getVar('IMAGE_NAME', True);
-
-    data = { 'deploy_dir' : deploy_dir, 'image_name' : image_name }
-
-    bb.event.fire(bb.event.MetadataEvent("LicenseManifestPath", data), d)
-}
-
 # set event handlers
 addhandler toaster_layerinfo_dumpdata
 toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -388,11 +333,8 @@ do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
 
-do_image_complete[postfuncs] += "toaster_image_dumpdata "
-do_image_complete[vardepsexclude] += "toaster_image_dumpdata "
-
-do_rootfs[postfuncs] += "toaster_licensemanifest_dump "
-do_rootfs[vardepsexclude] += "toaster_licensemanifest_dump "
-
 do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
 do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+
+do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
+do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
\ No newline at end of file
-- 
1.9.1

[PATCH 2/2] toaster.bbclass: only scan files-in-image.txt if it exists

[bavery]

From: Elliot Smith <elliot.smith@intel.com>

We can reach the method in toaster.bbclass which tries to read from
the files-in-image.txt file via a build which doesn't create that
file (e.g. "bitbake core-image-minimal -c rootfs"). This causes
the build to fail with an exception.

Check that this file exists before trying to read from it.

[YOCTO #9784]

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: bavery <brian.avery@intel.com>
---
 classes/toaster.bbclass | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/classes/toaster.bbclass b/classes/toaster.bbclass
index 90ea563..972efb9 100644
--- a/classes/toaster.bbclass
+++ b/classes/toaster.bbclass
@@ -288,15 +288,22 @@ python toaster_buildhistory_dump() {
                             images[target][dependsname] = {'size': 0, 'depends' : []}
                         images[target][pname]['depends'].append((dependsname, deptype))
 
-            with open("%s/files-in-image.txt" % installed_img_path, "r") as fin:
-                for line in fin:
-                    lc = [ x for x in line.strip().split(" ") if len(x) > 0 ]
-                    if lc[0].startswith("l"):
-                        files[target]['syms'].append(lc)
-                    elif lc[0].startswith("d"):
-                        files[target]['dirs'].append(lc)
-                    else:
-                        files[target]['files'].append(lc)
+            # files-in-image.txt is only generated if an image file is created,
+            # so the file entries ('syms', 'dirs', 'files') for a target will be
+            # empty for rootfs builds and other "image" tasks which don't
+            # produce image files
+            # (e.g. "bitbake core-image-minimal -c populate_sdk")
+            files_in_image_path = "%s/files-in-image.txt" % installed_img_path
+            if os.path.exists(files_in_image_path):
+                with open(files_in_image_path, "r") as fin:
+                    for line in fin:
+                        lc = [ x for x in line.strip().split(" ") if len(x) > 0 ]
+                        if lc[0].startswith("l"):
+                            files[target]['syms'].append(lc)
+                        elif lc[0].startswith("d"):
+                            files[target]['dirs'].append(lc)
+                        else:
+                            files[target]['files'].append(lc)
 
             for pname in images[target]:
                 if not pname in allpkgs:
-- 
1.9.1

Re: [PATCH 0/2] toaster: Better SDK artifacts and fstypes V2

[Richard Purdie]

On Tue, 2016-07-12 at 15:54 -0700, bavery wrote:
> This is part of a patch set that is split over the openembedded-core
> and
> bitbake-devel mailing lists.  This is a second version of the patch.
> 
> These patches remove complexity from the toaster.bbclass in order to
> move those
> functions to bitbake/lib/toaster.
> 
> The overall intent of this set is to improve Toaster's ability to
> locate artifacts,
> specify fstypes correctly, and add some tests for these changes.

FWIW you stripped a directory level off these patches so they didn't
apply. Since I'd already merged the bitbake changes, I fixed these
patches by hand.

Cheers,

Richard