[PATCH][1.26] Backport toaster fix from master

[PATCH][1.26] Backport toaster fix from master

[Paul Barker]

This is a backport of commit b4ce793685f70cab3f28cb4329aaaf3878cd62e8 from
master, re-written as `git cherry-pick` couldn't figure it out. I'm unsure if
this is the right way to send it as I'm not the auther of the original patch. I
can edit the authorship to credit Michael Wood and resubmit if desired - as I've
altered the patch I didn't want to put his name on it initially.

This fixes issues I've been seeing in toaster with packages missing from the
"Packages included" list and an empty "Directory structure" page for my images
when using fido and bitbake 1.26.

Paul Barker (1):
  toaster: buildinfohelper Skip packages we have no build info about

 lib/bb/ui/buildinfohelper.py | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)

-- 
1.8.3.1

[PATCH][1.26] toaster: buildinfohelper Skip packages we have no build info about

[Paul Barker]

If there are more packages listed as installed than we know about from
bitbake, and therefore have insufficient information to be able to
create a Toaster Package object then skip it. Also handle the case where
a dependency references such a package.

Also clarify the error logging.

Signed-off-by: Paul Barker <paul.barker@commagility.com>
Cc: Michael Wood <michael.g.wood@intel.com>
Cc: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 64bd94e..f7e0bfb 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -419,6 +419,11 @@ class ORMWrapper(object):
         errormsg = ""
         for p in packagedict:
             searchname = p
+            if p not in pkgpnmap:
+                logger.warning("Image packages list contains %p, but is"
+                               " missing from all packages list where the"
+                               " metadata comes from.  Skipping...", p)
+                continue
             if 'OPKGN' in pkgpnmap[p].keys():
                 searchname = pkgpnmap[p]['OPKGN']
 
@@ -462,13 +467,20 @@ class ORMWrapper(object):
                 elif deptype == 'recommends':
                     tdeptype = Package_Dependency.TYPE_TRECOMMENDS
 
-                packagedeps_objs.append(Package_Dependency( package = packagedict[p]['object'],
-                                        depends_on = packagedict[px]['object'],
-                                        dep_type = tdeptype,
-                                        target = target_obj))
+                try:
+                    packagedeps_objs.append(Package_Dependency(
+                        package = packagedict[p]['object'],
+                        depends_on = packagedict[px]['object'],
+                        dep_type = tdeptype,
+                        target = target_obj))
+                except KeyError as e:
+                    logger.warn("Could not add dependency to the package %s "
+                                "because %s is an unknown package", p, px)
 
         if len(packagedeps_objs) > 0:
             Package_Dependency.objects.bulk_create(packagedeps_objs)
+        else:
+            logger.info("No package dependencies created")
 
         if (len(errormsg) > 0):
             logger.warn("buildinfohelper: target_package_info could not identify recipes: \n%s" % errormsg)
@@ -1015,17 +1027,21 @@ class BuildInfoHelper(object):
         # for all image targets
         for target in self.internal_state['targets']:
             if target.is_image:
+                pkgdata = BuildInfoHelper._get_data_from_event(event)['pkgdata']
+                imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
+                filedata = BuildInfoHelper._get_data_from_event(event)['filedata'][target.target]
+
                 try:
-                    pkgdata = BuildInfoHelper._get_data_from_event(event)['pkgdata']
-                    imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
                     self.orm_wrapper.save_target_package_information(self.internal_state['build'], target, imgdata, pkgdata, self.internal_state['recipes'])
-                    filedata = BuildInfoHelper._get_data_from_event(event)['filedata'][target.target]
-                    self.orm_wrapper.save_target_file_information(self.internal_state['build'], target, filedata)
-                except KeyError:
-                    # we must have not got the data for this image, nothing to save
-                    pass
-
+                except KeyError as e:
+                    logger.warn("KeyError in save_target_package_information"
+                                "%s ", e)
 
+                try:
+                    self.orm_wrapper.save_target_file_information(self.internal_state['build'], target, filedata)
+                except KeyError as e:
+                    logger.warn("KeyError in save_target_file_information"
+                                "%s ", e)
 
     def store_dependency_information(self, event):
         assert '_depgraph' in vars(event)
-- 
1.8.3.1

Re: [PATCH][1.26] Backport toaster fix from master

[Michael Wood]

On 22/10/15 15:21, Paul Barker wrote:
> This is a backport of commit b4ce793685f70cab3f28cb4329aaaf3878cd62e8 from
> master, re-written as `git cherry-pick` couldn't figure it out. I'm unsure if
> this is the right way to send it as I'm not the auther of the original patch. I
> can edit the authorship to credit Michael Wood and resubmit if desired - as I've
> altered the patch I didn't want to put his name on it initially.
>
> This fixes issues I've been seeing in toaster with packages missing from the
> "Packages included" list and an empty "Directory structure" page for my images
> when using fido and bitbake 1.26.
>
> Paul Barker (1):
>    toaster: buildinfohelper Skip packages we have no build info about
>
>   lib/bb/ui/buildinfohelper.py | 40 ++++++++++++++++++++++++++++------------
>   1 file changed, 28 insertions(+), 12 deletions(-)
>

All fine by me.

Thanks,

Michael