[PATCH 0/2] michaelw/toaster/buildinfo-fixes

[PATCH 0/2] michaelw/toaster/buildinfo-fixes

[Michael Wood]

Two patches which fix regressions in the build -> toaster.

The first is where the recipe in the build history is linking to the recipe that is Toaster's data rather than the build's data. 

Fixes this issue identified by Belen:
"The recipes table in the build history is linking to the wrong recipe
id. The recipe details page appears empty but if you access the same
recipe from the package details page, you get to a recipe with the same
name and version, but different recipe id, that has all the information
from the build. To reproduce, build core-image-minimal. When the build
completes, go to the recipes page and search for busybox: click through to
the recipe details page. It's empty. Now click the image name on the left
nav to see the list of packages installed. Search for 'busybox'. Click
through to the package details page. Then, on the right hand side
information about the package, click the busybox recipe name. The page is
no longer empty: in fact, it's a different recipe id."

The second patch avoids the buildinfo helper bailing out when there are more packages reported by the ImagePkgs event data than have currently been provided by the all packages list. We get the meta data for a package - including it's dependencies from this "all packages". Also improved the error logging to make it clearer what is actually happening.

Branch on poky-contrib michaelw/toaster/buildinfo-fixes

Michael Wood (2):
  toaster: buildinfohelper associate build data with built_recipe
  toaster: buildinfohelper Skip packages we have no build info about

 bitbake/lib/bb/ui/buildinfohelper.py | 45 ++++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 12 deletions(-)

-- 
2.1.4

[PATCH 1/2] toaster: buildinfohelper associate build data with built_recipe

[Michael Wood]

Make sure we associate build data with the built recipe rather than
toaster's configuration copy of the recipe.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 793418a..9195584 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -285,6 +285,7 @@ class ORMWrapper(object):
 
         update_recipe_obj(recipe)
 
+        built_recipe = None
         # Create a copy of the recipe for historical puposes and update it
         for built_layer in self.layer_version_built:
             if built_layer.layer == recipe_information['layer_version'].layer:
@@ -300,7 +301,7 @@ class ORMWrapper(object):
         if created and must_exist:
             raise NotExisting("Recipe object created when expected to exist", recipe_information)
 
-        return recipe
+        return built_recipe
 
     def get_update_layer_version_object(self, build_obj, layer_obj, layer_version_information):
         if isinstance(layer_obj, Layer_Version):
-- 
2.1.4

[PATCH 2/2] toaster: buildinfohelper Skip packages we have no build info about

[Michael Wood]

If there are more packages listed as installed than we know about from
bitbake, and therefore have insufficient information to be able to
create a Toaster Package object then skip it. Also handle the case where
a dependency references such a package.

Also clarify the error logging.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
---
 bitbake/lib/bb/ui/buildinfohelper.py | 42 ++++++++++++++++++++++++++----------
 1 file changed, 31 insertions(+), 11 deletions(-)

diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
index 9195584..67c239e 100644
--- a/bitbake/lib/bb/ui/buildinfohelper.py
+++ b/bitbake/lib/bb/ui/buildinfohelper.py
@@ -319,7 +319,7 @@ class ORMWrapper(object):
                             commit=layer_version_information['commit'],
                             local_path = layer_version_information['local_path'],
                             )
-            logger.warning("created new historical layer version %d", layer_copy.pk)
+            logger.info("created new historical layer version %d", layer_copy.pk)
 
             self.layer_version_built.append(layer_copy)
 
@@ -510,6 +510,12 @@ class ORMWrapper(object):
         errormsg = ""
         for p in packagedict:
             searchname = p
+            if p not in pkgpnmap:
+                logger.warning("Image packages list contains %p, but is"
+                               " missing from all packages list where the"
+                               " metadata comes from. Skipping...", p)
+                continue
+
             if 'OPKGN' in pkgpnmap[p].keys():
                 searchname = pkgpnmap[p]['OPKGN']
 
@@ -553,13 +559,20 @@ class ORMWrapper(object):
                 elif deptype == 'recommends':
                     tdeptype = Package_Dependency.TYPE_TRECOMMENDS
 
-                packagedeps_objs.append(Package_Dependency( package = packagedict[p]['object'],
-                                        depends_on = packagedict[px]['object'],
-                                        dep_type = tdeptype,
-                                        target = target_obj))
+                try:
+                    packagedeps_objs.append(Package_Dependency(
+                        package = packagedict[p]['object'],
+                        depends_on = packagedict[px]['object'],
+                        dep_type = tdeptype,
+                        target = target_obj))
+                except KeyError as e:
+                    logger.warn("Could not add dependency to the package %s "
+                                "because %s is an unknown package", p, px)
 
         if len(packagedeps_objs) > 0:
             Package_Dependency.objects.bulk_create(packagedeps_objs)
+        else:
+            logger.info("No package dependencies created")
 
         if len(errormsg) > 0:
             logger.warn("buildinfohelper: target_package_info could not identify recipes: \n%s", errormsg)
@@ -1160,15 +1173,22 @@ class BuildInfoHelper(object):
         # for all image targets
         for target in self.internal_state['targets']:
             if target.is_image:
+                pkgdata = BuildInfoHelper._get_data_from_event(event)['pkgdata']
+                imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
+                filedata = BuildInfoHelper._get_data_from_event(event)['filedata'][target.target]
+
                 try:
-                    pkgdata = BuildInfoHelper._get_data_from_event(event)['pkgdata']
-                    imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target]
                     self.orm_wrapper.save_target_package_information(self.internal_state['build'], target, imgdata, pkgdata, self.internal_state['recipes'])
-                    filedata = BuildInfoHelper._get_data_from_event(event)['filedata'][target.target]
+                except KeyError as e:
+                    logger.warn("KeyError in save_target_package_information"
+                                "%s ", e)
+
+                try:
                     self.orm_wrapper.save_target_file_information(self.internal_state['build'], target, filedata)
-                except KeyError:
-                    # we must have not got the data for this image, nothing to save
-                    pass
+                except KeyError as e:
+                    logger.warn("KeyError in save_target_file_information"
+                                "%s ", e)
+
 
 
 
-- 
2.1.4

Re: [PATCH 1/2] toaster: buildinfohelper associate build data with built_recipe

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 1738 bytes --]

On 6 October 2015 at 19:03, Michael Wood <michael.g.wood@intel.com> wrote:

> Make sure we associate build data with the built recipe rather than
> toaster's configuration copy of the recipe.
>

What difference does this make to Toaster's behaviour? (I'm just asking so
I can check the patch.)

Thanks.
Elliot


>
> Signed-off-by: Michael Wood <michael.g.wood@intel.com>
> ---
>  bitbake/lib/bb/ui/buildinfohelper.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/bitbake/lib/bb/ui/buildinfohelper.py
> b/bitbake/lib/bb/ui/buildinfohelper.py
> index 793418a..9195584 100644
> --- a/bitbake/lib/bb/ui/buildinfohelper.py
> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
> @@ -285,6 +285,7 @@ class ORMWrapper(object):
>
>          update_recipe_obj(recipe)
>
> +        built_recipe = None
>          # Create a copy of the recipe for historical puposes and update it
>          for built_layer in self.layer_version_built:
>              if built_layer.layer ==
> recipe_information['layer_version'].layer:
> @@ -300,7 +301,7 @@ class ORMWrapper(object):
>          if created and must_exist:
>              raise NotExisting("Recipe object created when expected to
> exist", recipe_information)
>
> -        return recipe
> +        return built_recipe
>
>      def get_update_layer_version_object(self, build_obj, layer_obj,
> layer_version_information):
>          if isinstance(layer_obj, Layer_Version):
> --
> 2.1.4
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 2706 bytes --]

Re: [PATCH 1/2] toaster: buildinfohelper associate build data with built_recipe

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On 7 October 2015 at 11:01, Smith, Elliot <elliot.smith@intel.com> wrote:

> On 6 October 2015 at 19:03, Michael Wood <michael.g.wood@intel.com> wrote:
>
>> Make sure we associate build data with the built recipe rather than
>> toaster's configuration copy of the recipe.
>>
>
> What difference does this make to Toaster's behaviour? (I'm just asking so
> I can check the patch.)
>

Apologies, I just realised it's explained in the cover letter.

Elliot
-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 1283 bytes --]

Re: [PATCH 0/2] michaelw/toaster/buildinfo-fixes

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

On 6 October 2015 at 19:03, Michael Wood <michael.g.wood@intel.com> wrote:

> Two patches which fix regressions in the build -> toaster.
>

I'm currently reviewing this, but having some issues getting a build
running which I can test against. I'll get back to you once I've sorted it.

Elliot


>
> The first is where the recipe in the build history is linking to the
> recipe that is Toaster's data rather than the build's data.
>
> Fixes this issue identified by Belen:
> "The recipes table in the build history is linking to the wrong recipe
> id. The recipe details page appears empty but if you access the same
> recipe from the package details page, you get to a recipe with the same
> name and version, but different recipe id, that has all the information
> from the build. To reproduce, build core-image-minimal. When the build
> completes, go to the recipes page and search for busybox: click through to
> the recipe details page. It's empty. Now click the image name on the left
> nav to see the list of packages installed. Search for 'busybox'. Click
> through to the package details page. Then, on the right hand side
> information about the package, click the busybox recipe name. The page is
> no longer empty: in fact, it's a different recipe id."
>
> The second patch avoids the buildinfo helper bailing out when there are
> more packages reported by the ImagePkgs event data than have currently been
> provided by the all packages list. We get the meta data for a package -
> including it's dependencies from this "all packages". Also improved the
> error logging to make it clearer what is actually happening.
>
> Branch on poky-contrib michaelw/toaster/buildinfo-fixes
>
> Michael Wood (2):
>   toaster: buildinfohelper associate build data with built_recipe
>   toaster: buildinfohelper Skip packages we have no build info about
>
>  bitbake/lib/bb/ui/buildinfohelper.py | 45
> ++++++++++++++++++++++++++----------
>  1 file changed, 33 insertions(+), 12 deletions(-)
>
> --
> 2.1.4
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 3294 bytes --]

Re: [PATCH 0/2] michaelw/toaster/buildinfo-fixes

[Barros Pena, Belen]

On 07/10/2015 11:33, "toaster-bounces@yoctoproject.org on behalf of Smith,
Elliot" <toaster-bounces@yoctoproject.org on behalf of
elliot.smith@intel.com> wrote:

>On 6 October 2015 at 19:03, Michael Wood
><michael.g.wood@intel.com> wrote:
>
>Two patches which fix regressions in the build -> toaster.
>
>I'm currently reviewing this, but having some issues getting a build
>running which I can test against. I'll get back to you once I've sorted
>it.

Pending Elliot's review, this works for me. I've tested building
core-image-base on both master and fido (reusing sstate but clean tmp,
then rebuilding again). Recipe information is populated, the number of
packages installed (110 in master, 109 in fido) sounds about right
(although I no longer know what to compare against, seems the .manifest
file might not be 100% correct), and the package dependency information is
displayed. 

Thanks!

Belén

>
>
>Elliot
>
> 
>
>
>The first is where the recipe in the build history is linking to the
>recipe that is Toaster's data rather than the build's data.
>
>Fixes this issue identified by Belen:
>"The recipes table in the build history is linking to the wrong recipe
>id. The recipe details page appears empty but if you access the same
>recipe from the package details page, you get to a recipe with the same
>name and version, but different recipe id, that has all the information
>from the build. To reproduce, build core-image-minimal. When the build
>completes, go to the recipes page and search for busybox: click through to
>the recipe details page. It's empty. Now click the image name on the left
>nav to see the list of packages installed. Search for 'busybox'. Click
>through to the package details page. Then, on the right hand side
>information about the package, click the busybox recipe name. The page is
>no longer empty: in fact, it's a different recipe id."
>
>The second patch avoids the buildinfo helper bailing out when there are
>more packages reported by the ImagePkgs event data than have currently
>been provided by the all packages list. We get the meta data for a
>package - including it's dependencies from this
> "all packages". Also improved the error logging to make it clearer what
>is actually happening.
>
>Branch on poky-contrib michaelw/toaster/buildinfo-fixes
>
>Michael Wood (2):
>  toaster: buildinfohelper associate build data with built_recipe
>  toaster: buildinfohelper Skip packages we have no build info about
>
> bitbake/lib/bb/ui/buildinfohelper.py | 45
>++++++++++++++++++++++++++----------
> 1 file changed, 33 insertions(+), 12 deletions(-)
>
>--
>2.1.4
>
>--
>_______________________________________________
>toaster mailing list
>toaster@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/toaster
>
>
>
>
>
>
>
>-- 
>Elliot Smith
>Software Engineer
>Intel Open Source Technology Centre
>
>
>
>

[PATCH 1/2] toaster: buildinfohelper associate build data with built_recipe

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

Make sure we associate build data with the built recipe rather than
toaster's configuration copy of the recipe.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 793418a..9195584 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -285,6 +285,7 @@ class ORMWrapper(object):
 
         update_recipe_obj(recipe)
 
+        built_recipe = None
         # Create a copy of the recipe for historical puposes and update it
         for built_layer in self.layer_version_built:
             if built_layer.layer == recipe_information['layer_version'].layer:
@@ -300,7 +301,7 @@ class ORMWrapper(object):
         if created and must_exist:
             raise NotExisting("Recipe object created when expected to exist", recipe_information)
 
-        return recipe
+        return built_recipe
 
     def get_update_layer_version_object(self, build_obj, layer_obj, layer_version_information):
         if isinstance(layer_obj, Layer_Version):
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: [PATCH 0/2] michaelw/toaster/buildinfo-fixes

[Smith, Elliot]

[-- Attachment #1: Type: text/plain, Size: 2211 bytes --]

On 6 October 2015 at 19:03, Michael Wood <michael.g.wood@intel.com> wrote:

> Two patches which fix regressions in the build -> toaster.
>

Submitted upstream to bitbake-devel, added to toaster-next.

Elliot


>
> The first is where the recipe in the build history is linking to the
> recipe that is Toaster's data rather than the build's data.
>
> Fixes this issue identified by Belen:
> "The recipes table in the build history is linking to the wrong recipe
> id. The recipe details page appears empty but if you access the same
> recipe from the package details page, you get to a recipe with the same
> name and version, but different recipe id, that has all the information
> from the build. To reproduce, build core-image-minimal. When the build
> completes, go to the recipes page and search for busybox: click through to
> the recipe details page. It's empty. Now click the image name on the left
> nav to see the list of packages installed. Search for 'busybox'. Click
> through to the package details page. Then, on the right hand side
> information about the package, click the busybox recipe name. The page is
> no longer empty: in fact, it's a different recipe id."
>
> The second patch avoids the buildinfo helper bailing out when there are
> more packages reported by the ImagePkgs event data than have currently been
> provided by the all packages list. We get the meta data for a package -
> including it's dependencies from this "all packages". Also improved the
> error logging to make it clearer what is actually happening.
>
> Branch on poky-contrib michaelw/toaster/buildinfo-fixes
>
> Michael Wood (2):
>   toaster: buildinfohelper associate build data with built_recipe
>   toaster: buildinfohelper Skip packages we have no build info about
>
>  bitbake/lib/bb/ui/buildinfohelper.py | 45
> ++++++++++++++++++++++++++----------
>  1 file changed, 33 insertions(+), 12 deletions(-)
>
> --
> 2.1.4
>
> --
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster
>



-- 
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

[-- Attachment #2: Type: text/html, Size: 3081 bytes --]

Re: [PATCH 1/2] toaster: buildinfohelper associate build data with built_recipe

[Ed Bartosh]

Hi Michael,

This change caused below 2 tracebacks in my setup(ed/toaster/8279-v2, toaster
run from scratch: rm toaster.sqlite; . ../bitbake/bin/toaster

ERROR: Cannot assign None: "Task.recipe" does not allow null values.
Traceback (most recent call last):
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/ui/toasterui.py", line
248, in main
    buildinfohelper.store_started_task(event)
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/ui/buildinfohelper.py",
line 1043, in store_started_task
    self.orm_wrapper.get_update_task_object(task_information)
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/ui/buildinfohelper.py",
line 225, in get_update_task_object
    task_name=task_information['task_name']
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/ui/buildinfohelper.py",
line 99, in _cached_get_or_create
    clazz.objects.get_or_create(**kwargs)
  File "/usr/lib/python2.7/site-packages/django/db/models/manager.py",
line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/usr/lib/python2.7/site-packages/django/db/models/query.py",
line 381, in get_or_create
    obj = self.model(**params)
  File "/home/ed/git/yocto/poky/bitbake/lib/toaster/orm/models.py", line
500, in __init__
    super(Task, self).__init__(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/db/models/base.py", line
405, in __init__
    setattr(self, field.name, rel_obj)
  File
"/usr/lib/python2.7/site-packages/django/db/models/fields/related.py",
line 335, in __set__
    (instance._meta.object_name, self.field.name))
ValueError: Cannot assign None: "Task.recipe" does not allow null
values.


Traceback (most recent call last):
  File "/home/ed/git/yocto/poky/bitbake/bin/bitbake", line 45, in
<module>
    cookerdata.CookerConfiguration()))
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/main.py", line 419, in
bitbake_main
    return ui_module.main(server_connection.connection,
server_connection.events, configParams)
  File "/home/ed/git/yocto/poky/bitbake/lib/bb/ui/toasterui.py", line
368, in main
    logger.error("Error data dump %s\n%s\n" ,
traceback.format_tb(curr,1), pformat(curr.tb_frame.f_locals))
  File "/usr/lib64/python2.7/pprint.py", line 63, in pformat
    return PrettyPrinter(indent=indent, width=width,
depth=depth).pformat(object)
  File "/usr/lib64/python2.7/pprint.py", line 122, in pformat
    self._format(object, sio, 0, 0, {}, 0)
  File "/usr/lib64/python2.7/pprint.py", line 140, in _format
    rep = self._repr(object, context, level - 1)
  File "/usr/lib64/python2.7/pprint.py", line 226, in _repr
    self._depth, level)
  File "/usr/lib64/python2.7/pprint.py", line 238, in format
    return _safe_repr(object, context, maxlevels, level)
  File "/usr/lib64/python2.7/pprint.py", line 282, in _safe_repr
    vrepr, vreadable, vrecur = saferepr(v, context, maxlevels, level)
  File "/usr/lib64/python2.7/pprint.py", line 323, in _safe_repr
    rep = repr(object)
  File "/usr/lib/python2.7/site-packages/django/db/models/base.py", line
423, in __repr__
    u = six.text_type(self)
  File "/home/ed/git/yocto/poky/bitbake/lib/toaster/orm/models.py", line
552, in __unicode__
    return "%d(%d) %s:%s" % (self.pk, self.build.pk, self.recipe.name,
self.task_name)
  File
"/usr/lib/python2.7/site-packages/django/db/models/fields/related.py",
line 305, in __get__
    val = self.field.get_local_related_value(instance)
  File
"/usr/lib/python2.7/site-packages/django/db/models/fields/related.py",
line 997, in get_local_related_value
    return self.get_instance_value_for_fields(instance,
self.local_related_fields)
  File
"/usr/lib/python2.7/site-packages/django/db/models/fields/related.py",
line 1012, in get_instance_value_for_fields
    ret.append(getattr(instance, field.attname))
AttributeError: 'Task' object has no attribute 'recipe_id'

I also noticed that progress bar for commandline builds is not shown.
Probably it's caused by above tracebacks.

I removed this change from my branch. If you want to reproduce this
you can get my branch and re-apply it.

Regards,
Ed

On Tue, Oct 06, 2015 at 07:03:40PM +0100, Michael Wood wrote:
> Make sure we associate build data with the built recipe rather than
> toaster's configuration copy of the recipe.
> 
> Signed-off-by: Michael Wood <michael.g.wood@intel.com>
> ---
>  bitbake/lib/bb/ui/buildinfohelper.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py
> index 793418a..9195584 100644
> --- a/bitbake/lib/bb/ui/buildinfohelper.py
> +++ b/bitbake/lib/bb/ui/buildinfohelper.py
> @@ -285,6 +285,7 @@ class ORMWrapper(object):
>  
>          update_recipe_obj(recipe)
>  
> +        built_recipe = None
>          # Create a copy of the recipe for historical puposes and update it
>          for built_layer in self.layer_version_built:
>              if built_layer.layer == recipe_information['layer_version'].layer:
> @@ -300,7 +301,7 @@ class ORMWrapper(object):
>          if created and must_exist:
>              raise NotExisting("Recipe object created when expected to exist", recipe_information)
>  
> -        return recipe
> +        return built_recipe
>  
>      def get_update_layer_version_object(self, build_obj, layer_obj, layer_version_information):
>          if isinstance(layer_obj, Layer_Version):
> -- 
> 2.1.4
> 
> -- 
> _______________________________________________
> toaster mailing list
> toaster@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/toaster

-- 
--
Regards,
Ed