[PATCH 0/5] localhostbecontroller improvements

[PATCH 0/5] localhostbecontroller improvements

[Elliot Smith]

Support file:/// URIs for git repos and make various small improvements
to handling of custom image recipes and layers.

The following changes since commit 5bcf5bd6b498d17aa42738ee8af5542f5c9a2248:

  toasterui: remove ParseStarted from the event list (2016-03-17 11:58:35 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib toaster-next
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=toaster-next

Michael Wood (5):
  toaster: localhostbecontroller Don't clear out toaster custom layer
    dir
  toaster: orm Add a constant for the CustomImageRecipe's layer name
  toaster: localhostbecontroller Allow file:/// uri type for git repo
  toaster: localhostbecontroller put generated layer in the builddir
  toaster: orm generate CustomImageRecipe contents try secondary path

 lib/bb/ui/buildinfohelper.py                    |  4 ++--
 lib/toaster/bldcontrol/localhostbecontroller.py | 10 +++++-----
 lib/toaster/orm/models.py                       | 15 +++++++++++----
 lib/toaster/toastergui/views.py                 |  2 +-
 4 files changed, 19 insertions(+), 12 deletions(-)

--
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 1/5] toaster: localhostbecontroller Don't clear out toaster custom layer dir

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

We may have a recipe which is based on a custom image recipe that has
already been built so keep the recipe file around so that it can be read
by the generate recipe function.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/bldcontrol/localhostbecontroller.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/toaster/bldcontrol/localhostbecontroller.py b/lib/toaster/bldcontrol/localhostbecontroller.py
index efd82c3..38e97a2 100644
--- a/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -217,9 +217,9 @@ class LocalhostBEController(BuildEnvironmentController):
             raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
 
         # 5. create custom layer and add custom recipes to it
-        layerpath = os.path.join(self.be.sourcedir, "_meta-toaster-custom")
-        if os.path.isdir(layerpath):
-            shutil.rmtree(layerpath) # remove leftovers from previous builds
+        layerpath = os.path.join(self.be.sourcedir,
+                                 "_meta-toaster-custom_",
+                                 bitbake.req.project.name)
         for target in targets:
             try:
                 customrecipe = CustomImageRecipe.objects.get(name=target.target,
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 2/5] toaster: orm Add a constant for the CustomImageRecipe's layer name

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

Use a constant to define the name for the toaster custom images layer;
this constant is then used to identify this layer in various places.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py    | 4 ++--
 lib/toaster/orm/models.py       | 4 ++++
 lib/toaster/toastergui/views.py | 2 +-
 3 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index 28c7353..7b8b1a5 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -42,7 +42,7 @@ from orm.models import Variable, VariableHistory
 from orm.models import Package, Package_File, Target_Installed_Package, Target_File
 from orm.models import Task_Dependency, Package_Dependency
 from orm.models import Recipe_Dependency, Provides
-from orm.models import Project, CustomImagePackage
+from orm.models import Project, CustomImagePackage, CustomImageRecipe
 
 from bldcontrol.models import BuildEnvironment, BuildRequest
 
@@ -344,7 +344,7 @@ class ORMWrapper(object):
             # Special case the toaster-custom-images layer which is created
             # on the fly so don't update the values which may cause the layer
             # to be duplicated on a future get_or_create
-            if layer_obj.layer.name == "toaster-custom-images":
+            if layer_obj.layer.name == CustomImageRecipe.LAYER_NAME:
                 return layer_obj
             # We already found our layer version for this build so just
             # update it with the new build information
diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py
index 3c98c0b..a8b0408 100644
--- a/lib/toaster/orm/models.py
+++ b/lib/toaster/orm/models.py
@@ -1448,6 +1448,10 @@ class ProjectLayer(models.Model):
         unique_together = (("project", "layercommit"),)
 
 class CustomImageRecipe(Recipe):
+
+    # CustomImageRecipe's belong to layers called:
+    LAYER_NAME = "toaster-custom-images"
+
     search_allowed_fields = ['name']
     base_recipe = models.ForeignKey(Recipe, related_name='based_on_recipe')
     project = models.ForeignKey(Project)
diff --git a/lib/toaster/toastergui/views.py b/lib/toaster/toastergui/views.py
index 85ca9be..15760b3 100755
--- a/lib/toaster/toastergui/views.py
+++ b/lib/toaster/toastergui/views.py
@@ -2407,7 +2407,7 @@ if True:
 
             # create layer 'Custom layer' and verion if needed
             layer = Layer.objects.get_or_create(
-                name="toaster-custom-images",
+                name=CustomImageRecipe.LAYER_NAME,
                 summary="Layer for custom recipes",
                 vcs_url="file:///toaster_created_layer")[0]
 
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 3/5] toaster: localhostbecontroller Allow file:/// uri type for git repo

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

We don't need to skip file:/// uri type locations for git repositories.
If you're using a file:/// uri you should know that it has to be a local
path.

[YOCTO #9200]

igned-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/bldcontrol/localhostbecontroller.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/toaster/bldcontrol/localhostbecontroller.py b/lib/toaster/bldcontrol/localhostbecontroller.py
index 38e97a2..9deb572 100644
--- a/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -128,8 +128,9 @@ class LocalhostBEController(BuildEnvironmentController):
         gitrepos[(bitbake.giturl, bitbake.commit)].append( ("bitbake", bitbake.dirpath) )
 
         for layer in layers:
-            # we don't process local URLs
-            if layer.giturl.startswith("file://"):
+            # We don't need to git clone the layer for the CustomImageRecipe
+            # as it's generated by us layer on if needed
+            if CustomImageRecipe.LAYER_NAME in layer.name:
                 continue
             if not (layer.giturl, layer.commit) in gitrepos:
                 gitrepos[(layer.giturl, layer.commit)] = []
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 4/5] toaster: localhostbecontroller put generated layer in the builddir

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

Move the generated layer for custom recipes into the build directory.
The build directory makes more sense as this layer/recipe is generated
for a particular build/project.

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/bldcontrol/localhostbecontroller.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/toaster/bldcontrol/localhostbecontroller.py b/lib/toaster/bldcontrol/localhostbecontroller.py
index 9deb572..08419f2 100644
--- a/lib/toaster/bldcontrol/localhostbecontroller.py
+++ b/lib/toaster/bldcontrol/localhostbecontroller.py
@@ -218,9 +218,8 @@ class LocalhostBEController(BuildEnvironmentController):
             raise BuildSetupException("BE is not consistent: bblayers.conf file missing at %s" % bblayerconf)
 
         # 5. create custom layer and add custom recipes to it
-        layerpath = os.path.join(self.be.sourcedir,
-                                 "_meta-toaster-custom_",
-                                 bitbake.req.project.name)
+        layerpath = os.path.join(self.be.builddir,
+                                 CustomImageRecipe.LAYER_NAME)
         for target in targets:
             try:
                 customrecipe = CustomImageRecipe.objects.get(name=target.target,
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[PATCH 5/5] toaster: orm generate CustomImageRecipe contents try secondary path

[Elliot Smith]

From: Michael Wood <michael.g.wood@intel.com>

Try a secondary file path if the first does not exist. When we get the
recipe paths and layer information from the layer index it is not a
complete path but we are usually able to reconstruct it. If the complete
real path has been discovered by building then use this instead.

[YOCTO #9206]

Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/toaster/orm/models.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/lib/toaster/orm/models.py b/lib/toaster/orm/models.py
index a8b0408..6ae6316 100644
--- a/lib/toaster/orm/models.py
+++ b/lib/toaster/orm/models.py
@@ -1533,10 +1533,13 @@ class CustomImageRecipe(Recipe):
                 packages_conf += pkg.name+' '
 
         packages_conf += "\""
-
-        base_recipe = open("%s/%s" %
-                           (self.base_recipe.layer_version.dirpath,
-                            self.base_recipe.file_path), 'r').read()
+        try:
+            base_recipe = open("%s/%s" %
+                               (self.base_recipe.layer_version.dirpath,
+                                self.base_recipe.file_path), 'r').read()
+        except IOError:
+            # The path may now be the full path if the recipe has been built
+            base_recipe = open(self.base_recipe.file_path, 'r').read()
 
         # Add a special case for when the recipe we have based a custom image
         # recipe on requires another recipe.
-- 
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.