[PATCH 0/8] Add support for PROVIDES to Toaster

[PATCH 0/8] Add support for PROVIDES to Toaster

[Elliot Smith]

Add support of PROVIDES to Toaster backend and frontend.

Please see https://bugzilla.yoctoproject.org/show_bug.cgi?id=6169 for
more details.

The following changes since commit 3e0c0fb8eaf9b764c19f2c25fe50c4eccbd4a9bd:

  toaster-manual: describe the way to start Toaster (2016-01-08 11:09:50 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib elliot/submit/ed/toaster/provides-6169
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/submit/ed/toaster/provides-6169

Ed Bartosh (8):
  taskdata: refactor get_providermap
  cooker: add providermap to dep_tree
  buildinfohelper: use providermap
  toaster: add Provider model
  buildinfohelper: add provides info to the db
  toaster: show list of provides for the recipe
  toaster: show 'satisfied via' text for build deps
  toaster: show 'satisfied via' text for reverse deps

 lib/bb/cooker.py                                   |  7 +++++
 lib/bb/runqueue.py                                 |  2 +-
 lib/bb/taskdata.py                                 | 17 +++++-------
 lib/bb/ui/buildinfohelper.py                       | 31 +++++++++++++++-------
 .../orm/migrations/0002_auto_20151223_1528.py      | 27 +++++++++++++++++++
 lib/toaster/orm/models.py                          |  5 ++++
 lib/toaster/toastergui/templates/recipe.html       | 28 +++++++++++++++++--
 7 files changed, 95 insertions(+), 22 deletions(-)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto_20151223_1528.py

--
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 1/8] taskdata: refactor get_providermap

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Added optional parameter 'prefix' to filter out names that
don't start with specified prefix. Changed existing call
of get_providermap according to changed API.

Optimized the code: got rid of extra loop and temporary
list variable virts.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/runqueue.py |  2 +-
 lib/bb/taskdata.py | 17 +++++++----------
 2 files changed, 8 insertions(+), 11 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index ee06f0e..7fd0044 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -808,7 +808,7 @@ class RunQueueData:
                     invalidate_task(fn, st, True)
 
         # Create and print to the logs a virtual/xxxx -> PN (fn) table
-        virtmap = taskData.get_providermap()
+        virtmap = taskData.get_providermap(prefix="virtual/")
         virtpnmap = {}
         for v in virtmap:
             virtpnmap[v] = self.dataCache.pkg_fn[virtmap[v]]
diff --git a/lib/bb/taskdata.py b/lib/bb/taskdata.py
index 4d12b33..b083870 100644
--- a/lib/bb/taskdata.py
+++ b/lib/bb/taskdata.py
@@ -612,17 +612,14 @@ class TaskData:
                 break
         # self.dump_data()
 
-    def get_providermap(self):
-        virts = []
-        virtmap = {}
-
+    def get_providermap(self, prefix=None):
+        provmap = {}
         for name in self.build_names_index:
-            if name.startswith("virtual/"):
-                virts.append(name)
-        for v in virts:
-            if self.have_build_target(v):
-                virtmap[v] = self.fn_index[self.get_provider(v)[0]]
-        return virtmap
+            if prefix and not name.startswith(prefix):
+                continue
+            if self.have_build_target(name):
+                provmap[name] = self.fn_index[self.get_provider(name)[0]]
+        return provmap
 
     def dump_data(self):
         """
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 2/8] cooker: add providermap to dep_tree

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Added providermap information to the result of buildDependTree API.

This will be used by Toaster to map virtual dependencies to recipes.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/cooker.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index edceca0..e957169 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -725,8 +725,15 @@ class BBCooker:
         depend_tree["packages"] = {}
         depend_tree["rdepends-pkg"] = {}
         depend_tree["rrecs-pkg"] = {}
+        depend_tree['providermap'] = {}
         depend_tree["layer-priorities"] = self.recipecache.bbfile_config_priorities
 
+        for name, fn in taskdata.get_providermap().iteritems():
+            pn = self.recipecache.pkg_fn[fn]
+            if name != pn:
+                version = "%s:%s-%s" % self.recipecache.pkg_pepvpr[fn]
+                depend_tree['providermap'][name] = (pn, version)
+
         for task in xrange(len(rq.rqdata.runq_fnid)):
             taskname = rq.rqdata.runq_task[task]
             fnid = rq.rqdata.runq_fnid[task]
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 3/8] buildinfohelper: use providermap

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Used providermap in store_dependency_information function
to find virtual dependencies. This should fix annoying
warnings "stpd: KeyError saving recipe dependency"

[YOCTO #6169]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 279c5c7..cba0165 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -1186,6 +1186,7 @@ class BuildInfoHelper(object):
         assert 'layer-priorities' in event._depgraph
         assert 'pn' in event._depgraph
         assert 'tdepends' in event._depgraph
+        assert 'providermap' in event._depgraph
 
         errormsg = ""
 
@@ -1263,15 +1264,22 @@ class BuildInfoHelper(object):
         # buildtime
         recipedeps_objects = []
         for recipe in event._depgraph['depends']:
-            try:
-                target = self.internal_state['recipes'][recipe]
-                for dep in event._depgraph['depends'][recipe]:
+           target = self.internal_state['recipes'][recipe]
+           for dep in event._depgraph['depends'][recipe]:
+                if dep in assume_provided:
+                    continue
+                if dep in event._depgraph['providermap']:
+                    dep = event._depgraph['providermap'][dep][0]
+                if dep in self.internal_state['recipes']:
                     dependency = self.internal_state['recipes'][dep]
-                    recipedeps_objects.append(Recipe_Dependency( recipe = target,
-                            depends_on = dependency, dep_type = Recipe_Dependency.TYPE_DEPENDS))
-            except KeyError as e:
-                if e not in assume_provided and not str(e).startswith("virtual/"):
-                    errormsg += "  stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, e)
+                else:
+                    errormsg += "  stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, dep)
+                    continue
+                recipe_dep = Recipe_Dependency(recipe=target,
+                                               depends_on=dependency,
+                                               dep_type=Recipe_Dependency.TYPE_DEPENDS)
+                recipedeps_objects.append(recipe_dep)
+
         Recipe_Dependency.objects.bulk_create(recipedeps_objects)
 
         # save all task information
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 4/8] toaster: add Provider model

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Added new model Provider and a foreign key 'via' to link
Recipe_Dependency to it.

[YOCTO #6169]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 .../orm/migrations/0002_auto_20151223_1528.py      | 27 ++++++++++++++++++++++
 lib/toaster/orm/models.py                          |  5 ++++
 2 files changed, 32 insertions(+)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0002_auto_20151223_1528.py

diff --git a/lib/toaster/orm/migrations/0002_auto_20151223_1528.py b/lib/toaster/orm/migrations/0002_auto_20151223_1528.py
new file mode 100644
index 0000000..194c897
--- /dev/null
+++ b/lib/toaster/orm/migrations/0002_auto_20151223_1528.py
@@ -0,0 +1,27 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('orm', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Provides',
+            fields=[
+                ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
+                ('name', models.CharField(max_length=100)),
+                ('recipe', models.ForeignKey(to='orm.Recipe')),
+            ],
+        ),
+        migrations.AddField(
+            model_name='recipe_dependency',
+            name='via',
+            field=models.ForeignKey(null=True, default=None, to='orm.Provides'),
+        ),
+    ]
diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py
index 6e87c54..e4ab0bb 100644
--- a/lib/toaster/orm/models.py
+++ b/lib/toaster/orm/models.py
@@ -733,6 +733,10 @@ class Recipe_DependencyManager(models.Manager):
     def get_queryset(self):
         return super(Recipe_DependencyManager, self).get_queryset().exclude(recipe_id = F('depends_on__id'))
 
+class Provides(models.Model):
+    name = models.CharField(max_length=100)
+    recipe = models.ForeignKey(Recipe)
+
 class Recipe_Dependency(models.Model):
     TYPE_DEPENDS = 0
     TYPE_RDEPENDS = 1
@@ -743,6 +747,7 @@ class Recipe_Dependency(models.Model):
     )
     recipe = models.ForeignKey(Recipe, related_name='r_dependencies_recipe')
     depends_on = models.ForeignKey(Recipe, related_name='r_dependencies_depends')
+    via = models.ForeignKey(Provides, null=True, default=None)
     dep_type = models.IntegerField(choices=DEPENDS_TYPE)
     objects = Recipe_DependencyManager()
 
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 5/8] buildinfohelper: add provides info to the db

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Added new entries to Provides model and link them to
Recipe_Dependency using 'via' field.

This data will be used by Toaster UI to show 'Provides:'
information for the recipes.

[YOCTO #6169]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index cba0165..0cb6f68 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -41,7 +41,7 @@ from orm.models import Target_Image_File, BuildArtifact
 from orm.models import Variable, VariableHistory
 from orm.models import Package, Package_File, Target_Installed_Package, Target_File
 from orm.models import Task_Dependency, Package_Dependency
-from orm.models import Recipe_Dependency
+from orm.models import Recipe_Dependency, Provides
 
 from orm.models import Project
 from bldcontrol.models import BuildEnvironment, BuildRequest
@@ -1268,15 +1268,20 @@ class BuildInfoHelper(object):
            for dep in event._depgraph['depends'][recipe]:
                 if dep in assume_provided:
                     continue
+                via = None
                 if dep in event._depgraph['providermap']:
-                    dep = event._depgraph['providermap'][dep][0]
-                if dep in self.internal_state['recipes']:
+                    deprecipe = event._depgraph['providermap'][dep][0]
+                    dependency = self.internal_state['recipes'][deprecipe]
+                    via = Provides.objects.get_or_create(name=dep,
+                                                         recipe=dependency)[0]
+                elif dep in self.internal_state['recipes']:
                     dependency = self.internal_state['recipes'][dep]
                 else:
                     errormsg += "  stpd: KeyError saving recipe dependency for %s, %s \n" % (recipe, dep)
                     continue
                 recipe_dep = Recipe_Dependency(recipe=target,
                                                depends_on=dependency,
+                                               via=via,
                                                dep_type=Recipe_Dependency.TYPE_DEPENDS)
                 recipedeps_objects.append(recipe_dep)
 
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 6/8] toaster: show list of provides for the recipe

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Showed list of names that recipe provides.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/toastergui/templates/recipe.html | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lib/toaster/toastergui/templates/recipe.html b/lib/toaster/toastergui/templates/recipe.html
index c6ae2f3..bbe7475 100644
--- a/lib/toaster/toastergui/templates/recipe.html
+++ b/lib/toaster/toastergui/templates/recipe.html
@@ -71,6 +71,16 @@
                     Layer commit
                 </dt>
                 <dd class="iscommit">{{layer_version.commit}}</dd>
+                {% if object.provides_set.all %}
+                <dt>
+                    <i class="icon-question-sign get-help"
+                     title="A list of aliases by which a particular recipe can be known. The additional aliases are
+                           synonyms for the recipe and can be useful satisfying dependencies of other recipes during
+                           the build"></i>
+                    PROVIDES
+                </dt>
+                 <dd><code>{% for provider in object.provides_set.all %}{{ provider.name }}&nbsp;{% endfor %}</code></dd>
+                {% endif %}
             </dl>
 
             <h2 class="details">Tasks</h2>
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 7/8] toaster: show 'satisfied via' text for build deps

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Showed '<dependency> satisfied via <provider>' text and
help tooltip for the build dependencies provided through
'PROVIDES' in the 'Build dependencies' tab.

[YOCTO #6169]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/toastergui/templates/recipe.html | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/toaster/toastergui/templates/recipe.html b/lib/toaster/toastergui/templates/recipe.html
index bbe7475..d6b464b 100644
--- a/lib/toaster/toastergui/templates/recipe.html
+++ b/lib/toaster/toastergui/templates/recipe.html
@@ -172,7 +172,14 @@
 
                     {% for rr in object.r_dependencies_recipe.all|dictsort:"depends_on.name" %}
                     <tr>
-                        <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.name}}</a></td>
+                        <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.name}}</a>
+                            {% if rr.via %}
+                                <span class="muted">satisfied via {{rr.via.name}}</span>
+                                <i class="icon-question-sign get-help hover-help"
+                                 title="This dependency is satisfied by the PROVIDES value
+                                       {{rr.via.name}} in the {{rr.depends_on.name}} recipe"></i>
+                            {% endif %}
+                        </td>
                         <td><a href="{% url "recipe" build.pk rr.depends_on.pk %}">{{rr.depends_on.version}}</a></td>
                     </tr>
                     {% endfor %}
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 8/8] toaster: show 'satisfied via' text for reverse deps

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Showed '<dependency> satisfied via <provider>' text and
help tooltip for the reverse build dependencies provided
through 'PROVIDES' in the 'Reverse build dependencies' tab.

[YOCTO #6169]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/toastergui/templates/recipe.html | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/toaster/toastergui/templates/recipe.html b/lib/toaster/toastergui/templates/recipe.html
index d6b464b..1d6d64e 100644
--- a/lib/toaster/toastergui/templates/recipe.html
+++ b/lib/toaster/toastergui/templates/recipe.html
@@ -211,7 +211,14 @@
 
                     {% for rr in object.r_dependencies_depends.all|dictsort:"recipe.name" %}
                     <tr>
-                        <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.name}}</a></td>
+                        <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.name}}</a>
+                        {% if rr.via %}
+                            <span class="muted"> satisfied via {{rr.via.name}}</span>
+                            <i class="icon-question-sign get-help hover-help"
+                             title="This dependency is satisfied by the PROVIDES value
+                            {{rr.via.name}} in the {{rr.depends_on.name}} recipe"></i>
+                        {% endif %}
+                        </td>
                         <td><a href="{% url "recipe" build.pk rr.recipe.pk %}">{{rr.recipe.version}}</a></td>
                     </tr>
                     {% endfor %}
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.