[PATCH 0/2] toaster: buildinfohelper fixes

[PATCH 0/2] toaster: buildinfohelper fixes

[Elliot Smith]

Two patches which fix regressions in the build -> toaster.

1. The first is where the recipe in the build history is linking to the
recipe that is Toaster's data rather than the build's data.

This fixes an issue identified by Belen:

"The recipes table in the build history is linking to the wrong recipe
id. The recipe details page appears empty but if you access the same
recipe from the package details page, you get to a recipe with the same
name and version, but different recipe id, that has all the information
from the build. To reproduce, build core-image-minimal. When the build
completes, go to the recipes page and search for busybox: click through to
the recipe details page. It's empty. Now click the image name on the left
nav to see the list of packages installed. Search for 'busybox'. Click
through to the package details page. Then, on the right hand side
information about the package, click the busybox recipe name. The page is
no longer empty: in fact, it's a different recipe id."

2. The second patch avoids the buildinfo helper bailing out when there are more packages reported by the ImagePkgs event data than have currently been provided by the all packages list. We get the meta data for a package - including its dependencies from this "all packages". Also improved the error logging to make it clearer what is actually happening.

The following changes since commit 18decad837a86b220bc549eec432f94f55510fcb:

  toaster: Modify "New build" button behaviour for cli builds project (2015-10-07 16:14:03 +0300)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib elliot/submit/michaelw/toaster/buildinfo-fixes
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/submit/michaelw/toaster/buildinfo-fixes

Michael Wood (2):
  toaster: buildinfohelper associate build data with built_recipe
  toaster: buildinfohelper Skip packages we have no build info about

 lib/bb/ui/buildinfohelper.py | 45 ++++++++++++++++++++++++++++++++------------
 1 file changed, 33 insertions(+), 12 deletions(-)

--
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 1/2] toaster: buildinfohelper associate build data with built_recipe

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

Make sure we associate build data with the built recipe rather than
toaster's configuration copy of the recipe.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 793418a..9195584 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -285,6 +285,7 @@ class ORMWrapper(object):
 
         update_recipe_obj(recipe)
 
+        built_recipe = None
         # Create a copy of the recipe for historical puposes and update it
         for built_layer in self.layer_version_built:
             if built_layer.layer == recipe_information['layer_version'].layer:
@@ -300,7 +301,7 @@ class ORMWrapper(object):
         if created and must_exist:
             raise NotExisting("Recipe object created when expected to exist", recipe_information)
 
-        return recipe
+        return built_recipe
 
     def get_update_layer_version_object(self, build_obj, layer_obj, layer_version_information):
         if isinstance(layer_obj, Layer_Version):
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 2/2] toaster: buildinfohelper Skip packages we have no build info about

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

If there are more packages listed as installed than we know about from
bitbake, and therefore have insufficient information to be able to
create a Toaster Package object then skip it. Also handle the case where
a dependency references such a package.

Also clarify the error logging.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py | 42 +++++++++++++++++++++++++++++++-----------
 1 file changed, 31 insertions(+), 11 deletions(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 9195584..67c239e 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -319,7 +319,7 @@ class ORMWrapper(object):
                             commit=layer_version_information['commit'],
                             local_path = layer_version_information['local_path'],
                             )
-            logger.warning("created new historical layer version %d", layer_copy.pk)
+            logger.info("created new historical layer version %d", layer_copy.pk)
 
             self.layer_version_built.append(layer_copy)
 
@@ -510,6 +510,12 @@ class ORMWrapper(object):
         errormsg = ""
         for p in packagedict:
             searchname = p
+            if p not in pkgpnmap:
+                logger.warning("Image packages list contains %p, but is"
+                               " missing from all packages list where the"
+                               " metadata comes from. Skipping...", p)
+                continue
+
             if 'OPKGN' in pkgpnmap[p].keys():
                 searchname = pkgpnmap[p]['OPKGN']
 
@@ -553,13 +559,20 @@ class ORMWrapper(object):
                 elif deptype == 'recommends':
                     tdeptype = Package_Dependency.TYPE_TRECOMMENDS
 
-                packagedeps_objs.append(Package_Dependency( package = packagedict[p]['object'],
-                                        depends_on = packagedict[px]['object'],
-                                        dep_type = tdeptype,
-                                        target = target_obj))
+                try:
+                    packagedeps_objs.append(Package_Dependency(
+                        package = packagedict[p]['object'],
+                        depends_on = packagedict[px]['object'],
+                        dep_type = tdeptype,
+                        target = target_obj))
+                except KeyError as e:
+                    logger.warn("Could not add dependency to the package %s "
+                                "because %s is an unknown package", p, px)
 
         if len(packagedeps_objs) > 0:
             Package_Dependency.objects.bulk_create(packagedeps_objs)
+        else:
+            logger.info("No package dependencies created")
 
         if len(errormsg) > 0:
             logger.warn("buildinfohelper: target_package_info could not identify recipes: \n%s", errormsg)
@@ -1160,15 +1173,22 @@ class BuildInfoHelper(object):
         # for all image targets
         for target in self.internal_state['targets']:
             if target.is_image:
+                pkgdata = BuildInfoHelper._get_data_from_event(event)['pkgdata']
+                imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
+                filedata = BuildInfoHelper._get_data_from_event(event)['filedata'][target.target]
+
                 try:
-                    pkgdata = BuildInfoHelper._get_data_from_event(event)['pkgdata']
-                    imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
                     self.orm_wrapper.save_target_package_information(self.internal_state['build'], target, imgdata, pkgdata, self.internal_state['recipes'])
-                    filedata = BuildInfoHelper._get_data_from_event(event)['filedata'][target.target]
+                except KeyError as e:
+                    logger.warn("KeyError in save_target_package_information"
+                                "%s ", e)
+
+                try:
                     self.orm_wrapper.save_target_file_information(self.internal_state['build'], target, filedata)
-                except KeyError:
-                    # we must have not got the data for this image, nothing to save
-                    pass
+                except KeyError as e:
+                    logger.warn("KeyError in save_target_file_information"
+                                "%s ", e)
+
 
 
 
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.