[layerindex-web][PATCH 0/3] Layer index fixes

[layerindex-web][PATCH 0/3] Layer index fixes

[Paul Eggleton]

Some minor fixes for the recent python3 work in the layer index, but more
importantly fix parsing in master for the multiconfig changes that recently
went into bitbake.

NOTE: this depends on the bitbake patchset I just sent in order to work.


The following changes since commit 1d76228675e93add50dd5216c43e00f977e1aaa8:

  Use functools.reduce instead of reduce (2016-07-04 09:51:21 +1200)

are available in the git repository at:

  git://git.yoctoproject.org/layerindex-web paule/mc-fixes
  http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/mc-fixes

Paul Eggleton (3):
  Use python3 commands in README
  Fix output decoding/reporting for bulk change patch download
  update_layer.py: fix up for bitbake API change

 README                             |  6 +++---
 layerindex/tools/import_classic.py | 11 +++++++----
 layerindex/update_layer.py         | 17 ++++++++++-------
 layerindex/views.py                |  3 +++
 4 files changed, 23 insertions(+), 14 deletions(-)

-- 
2.5.5

[layerindex-web][PATCH 1/3] Use python3 commands in README

[Paul Eggleton]

If you're in a virtualenv it doesn't make a difference, but outside of
one it will.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 README | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/README b/README
index fa5c3b2..8437089 100644
--- a/README
+++ b/README
@@ -47,15 +47,15 @@ Setup instructions:
 2. Run the following commands within the layerindex-web directory to
    initialise the database:
 
-   python manage.py syncdb
-   python manage.py migrate
+   python3 manage.py syncdb
+   python3 manage.py migrate
 
    You should answer "yes" when asked to create an admin account.
 
 3. You can test the web application locally by setting DEBUG = True in
    settings.py and running the following:
 
-   python manage.py runserver
+   python3 manage.py runserver
 
    Then visit http://127.0.0.1:8000/layerindex/ with your browser. As
    with all Django applications there is an admin interface available
-- 
2.5.5

[layerindex-web][PATCH 2/3] Fix output decoding/reporting for bulk change patch download

[Paul Eggleton]

With Python 3 we need to take care of decoding the output so we can
treat it as a string. At the same time, it's more useful to see the
output string rather than the exception if there is some output.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 layerindex/views.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/layerindex/views.py b/layerindex/views.py
index dfe4453..62f5268 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -237,8 +237,11 @@ def bulk_change_patch_view(request, pk):
     except Exception as e:
         output = getattr(e, 'output', None)
         if output:
+            output = output.decode('utf-8', errors="ignore")
             if 'timeout' in output:
                 return HttpResponse('Failed to generate patches: timed out waiting for lock. Please try again shortly.', content_type='text/plain')
+            else:
+                return HttpResponse('Failed to generate patches: %s' % output, content_type='text/plain')
         return HttpResponse('Failed to generate patches: %s' % e, content_type='text/plain')
     # FIXME better error handling
 
-- 
2.5.5

[layerindex-web][PATCH 3/3] update_layer.py: fix up for bitbake API change

[Paul Eggleton]

The multiconfig changes broke the calls here to loadDataFull(). To avoid
this being an issue in future, make use of tinfoil's new parse_recipe_file()
function (if available) to isolate the code here from any future changes to
bitbake's internal code.

Part of the fix for [YOCTO #10192].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 layerindex/tools/import_classic.py | 11 +++++++----
 layerindex/update_layer.py         | 17 ++++++++++-------
 2 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/layerindex/tools/import_classic.py b/layerindex/tools/import_classic.py
index ee99246..45ccaa9 100755
--- a/layerindex/tools/import_classic.py
+++ b/layerindex/tools/import_classic.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # Import OE-Classic recipe data into the layer index database
 #
@@ -27,11 +27,14 @@ import recipeparse
 logger = utils.logger_create('LayerIndexUpdate')
 
 
-def update_recipe_file(data, path, recipe, layerdir_start, repodir):
+def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir):
     fn = str(os.path.join(path, recipe.filename))
     try:
         logger.debug('Updating recipe %s' % fn)
-        envdata = bb.cache.Cache.loadDataFull(fn, [], data)
+        if hasattr(tinfoil, 'parse_recipe_file'):
+            envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)
+        else:
+            envdata = bb.cache.Cache.loadDataFull(fn, [], data)
         envdata.setVar('SRCPV', 'X')
         envdata.setVar('SRCDATE', 'X')
         envdata.setVar('SRCREV', 'X')
@@ -182,7 +185,7 @@ def main():
                     recipe.layerbranch = layerbranch
                     recipe.filename = filename
                     recipe.filepath = filepath
-                    update_recipe_file(config_data_copy, root, recipe, layerdir_start, oeclassicpath)
+                    update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, oeclassicpath)
                     recipe.save()
 
         layerbranch.vcs_last_fetch = datetime.now()
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index fd8d3d4..13b508f 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -54,11 +54,14 @@ def split_recipe_fn(path):
         pv = "1.0"
     return (pn, pv)
 
-def update_recipe_file(data, path, recipe, layerdir_start, repodir):
+def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir):
     fn = str(os.path.join(path, recipe.filename))
     try:
         logger.debug('Updating recipe %s' % fn)
-        envdata = bb.cache.Cache.loadDataFull(fn, [], data)
+        if hasattr(tinfoil, 'parse_recipe_file'):
+            envdata = tinfoil.parse_recipe_file(fn, appends=False, config_data=data)
+        else:
+            envdata = bb.cache.Cache.loadDataFull(fn, [], data)
         envdata.setVar('SRCPV', 'X')
         recipe.pn = envdata.getVar("PN", True)
         recipe.pv = envdata.getVar("PV", True)
@@ -356,7 +359,7 @@ def main():
                                     recipe.filepath = newfilepath
                                     recipe.filename = newfilename
                                     recipe.save()
-                                    update_recipe_file(config_data_copy, os.path.join(layerdir, newfilepath), recipe, layerdir_start, repodir)
+                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, newfilepath), recipe, layerdir_start, repodir)
                                     updatedrecipes.add(os.path.join(oldfilepath, oldfilename))
                                     updatedrecipes.add(os.path.join(newfilepath, newfilename))
                                 else:
@@ -471,7 +474,7 @@ def main():
                                 results = layerrecipes.filter(filepath=filepath).filter(filename=filename)[:1]
                                 if results:
                                     recipe = results[0]
-                                    update_recipe_file(config_data_copy, os.path.join(layerdir, filepath), recipe, layerdir_start, repodir)
+                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, filepath), recipe, layerdir_start, repodir)
                                     recipe.save()
                                     updatedrecipes.add(recipe.full_path())
                             elif typename == 'machine':
@@ -487,7 +490,7 @@ def main():
 
                     for recipe in dirtyrecipes:
                         if not recipe.full_path() in updatedrecipes:
-                            update_recipe_file(config_data_copy, os.path.join(layerdir, recipe.filepath), recipe, layerdir_start, repodir)
+                            update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, recipe.filepath), recipe, layerdir_start, repodir)
                 else:
                     # Collect recipe data from scratch
 
@@ -513,7 +516,7 @@ def main():
                                 # Recipe still exists, update it
                                 results = layerrecipes.filter(id=v['id'])[:1]
                                 recipe = results[0]
-                                update_recipe_file(config_data_copy, root, recipe, layerdir_start, repodir)
+                                update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir)
                             else:
                                 # Recipe no longer exists, mark it for later on
                                 layerrecipes_delete.append(v)
@@ -575,7 +578,7 @@ def main():
                     recipe.filename = os.path.basename(added)
                     root = os.path.dirname(added)
                     recipe.filepath = os.path.relpath(root, layerdir)
-                    update_recipe_file(config_data_copy, root, recipe, layerdir_start, repodir)
+                    update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir)
                     recipe.save()
 
                 for deleted in layerrecipes_delete:
-- 
2.5.5

Re: [layerindex-web][PATCH 0/3] Layer index fixes

[Andreas Müller]

On Tue, Aug 30, 2016 at 11:03 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> Some minor fixes for the recent python3 work in the layer index, but more
> importantly fix parsing in master for the multiconfig changes that recently
> went into bitbake.
>
> NOTE: this depends on the bitbake patchset I just sent in order to work.
>
>
> The following changes since commit 1d76228675e93add50dd5216c43e00f977e1aaa8:
>
Thanks

Andreas