[layerindex-web][PATCH 0/3] Minor layer index fixes

[layerindex-web][PATCH 0/3] Minor layer index fixes

[Paul Eggleton]

A couple of minor fixes for the recent RRS and other distro integration,
plus one additional debugging feature for the update script.


The following changes since commit b3e9cb05d22e0961341d87dce3128eeb273544f6:

  Include errors/warnings in main log in error/warning counts for updates (2018-08-14 15:09:36 +0200)

are available in the Git repository at:

  git://git.yoctoproject.org/layerindex-web paule/fixes6
  http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=paule/fixes6

Paul Eggleton (3):
  rrs: fix recipe/maintainer count showing at top
  Hide navbar on comparison select page
  update: add option to stop immediately on parse errors

 layerindex/update.py                          |  8 ++++
 layerindex/update_layer.py                    | 39 ++++++++++++-------
 templates/base_toplevel.html                  |  2 +
 .../layerindex/comparisonrecipeselect.html    |  3 ++
 templates/rrs/maintainers.html                |  6 +--
 templates/rrs/recipes.html                    |  6 +--
 6 files changed, 43 insertions(+), 21 deletions(-)

-- 
2.17.1

[layerindex-web][PATCH 1/3] rrs: fix recipe/maintainer count showing at top

[Paul Eggleton]

We were using the main badge class as a selector here to show the recipe
or maintainer count, and that meant that it also showed up at the top
next to the login button if there were some layers to review. Use a
proper id to stop that from happening.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 templates/rrs/maintainers.html | 6 +++---
 templates/rrs/recipes.html     | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/templates/rrs/maintainers.html b/templates/rrs/maintainers.html
index c237000f..cee02006 100644
--- a/templates/rrs/maintainers.html
+++ b/templates/rrs/maintainers.html
@@ -23,7 +23,7 @@
     <div class="navbar-inner table-controls">
         <ul class="nav">
             <li class="dropdown">
-            <span class="badge" style="margin-top:11px;"></span>
+            <span class="badge" id="maintainer-count" style="margin-top:11px;"></span>
             </li>
         </ul>    		
         <form id="form-search" class="pull-right input-append">
@@ -188,9 +188,9 @@ $(document).ready(function() {
         }
 
         if (count == 2) {
-            $('.badge').html("1 maintainer");
+            $('#maintainer-count').html("1 maintainer");
         } else {
-            $('.badge').html((count - 1) + " maintainers")
+            $('#maintainer-count').html((count - 1) + " maintainers")
         }
         updateTotals()
     }
diff --git a/templates/rrs/recipes.html b/templates/rrs/recipes.html
index af3c08d6..d917f19d 100644
--- a/templates/rrs/recipes.html
+++ b/templates/rrs/recipes.html
@@ -23,7 +23,7 @@
     <div class="navbar-inner table-controls">
         <ul class="nav">
             <li class="dropdown">
-                <span class="badge"></span>
+                <span class="badge" id="recipe-count"></span>
             </li>
             <li class="dropdown">
                 <a data-toggle="dropdown" href="#" class="dropdown-toggle" id="selected-status">
@@ -203,9 +203,9 @@ $(document).ready(function() {
         }
 
         if (count == 2) {
-            $('.badge').html("1 recipe");
+            $('#recipe-count').html("1 recipe");
         } else {
-            $('.badge').html((count - 1) + " recipes");
+            $('#recipe-count').html((count - 1) + " recipes");
         }
     }
 
-- 
2.17.1

[layerindex-web][PATCH 2/3] Hide navbar on comparison select page

[Paul Eggleton]

We don't want to see the navbar on this page because otherwise it lets
you select branches and click on Stats etc. which doesn't make any sense
in this context.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 templates/base_toplevel.html                     | 2 ++
 templates/layerindex/comparisonrecipeselect.html | 3 +++
 2 files changed, 5 insertions(+)

diff --git a/templates/base_toplevel.html b/templates/base_toplevel.html
index 1dcac775..afcc19bf 100644
--- a/templates/base_toplevel.html
+++ b/templates/base_toplevel.html
@@ -15,6 +15,7 @@
     <div class="container-fluid">
         <div class="row-fluid">
 
+            {% block navbar %}
             <div class="navbar">
                 <div class="navbar-inner">
                     <ul class="nav">
@@ -53,6 +54,7 @@
                     {% block navs_extra %}{% endblock %}
                 </div>
             </div>
+            {% endblock %}
 
             {% block content_inner %}{% endblock %}
 
diff --git a/templates/layerindex/comparisonrecipeselect.html b/templates/layerindex/comparisonrecipeselect.html
index 1646ae3d..47b299be 100644
--- a/templates/layerindex/comparisonrecipeselect.html
+++ b/templates/layerindex/comparisonrecipeselect.html
@@ -14,6 +14,9 @@
 {% block title_append %} - comparison - select match for {{ select_for.name }}{% endblock %}
 -->
 
+{% block navbar %}
+{% endblock %}
+
                 {% block page_heading %}
                 <h2>Select match for {{ select_for.pn }} in {% if branch.comparison %}{{ branch }}{% else %}OpenEmbedded{% endif %}</h2>
                 <div class="pull-right">
-- 
2.17.1

[layerindex-web][PATCH 3/3] update: add option to stop immediately on parse errors

[Paul Eggleton]

During debugging a parsing issue, we don't really want to continue if a
parsing error occurs, and in that situation I usually end up using
Ctrl+C to exit early. Add an option to exit immediately upon error to
avoid having to do that.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 layerindex/update.py       |  8 ++++++++
 layerindex/update_layer.py | 39 +++++++++++++++++++++++---------------
 2 files changed, 32 insertions(+), 15 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index aea7cd09..1b2bb153 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -57,6 +57,8 @@ def prepare_update_layer_command(options, branch, layer, initial=False):
         cmd += ' -q'
     if options.keep_temp:
         cmd += ' --keep-temp'
+    if options.stop_on_error:
+        cmd += ' --stop-on-error'
     return cmd
 
 def update_actual_branch(layerquery, fetchdir, branch, options, update_bitbake, bitbakepath):
@@ -159,6 +161,9 @@ def main():
     parser.add_option("", "--nocheckout",
             help = "Don't check out branches",
             action="store_true", dest="nocheckout")
+    parser.add_option("", "--stop-on-error",
+            help = "Stop on first parsing error",
+            action="store_true", default=False, dest="stop_on_error")
     parser.add_option("-a", "--actual-branch",
             help = "Update actual branch for layer and bitbake",
             action="store", dest="actual_branch", default='')
@@ -528,6 +533,9 @@ def main():
                         # Interrupted by user, break out of loop
                         logger.info('Update interrupted, exiting')
                         sys.exit(254)
+                    if options.stop_on_error and ret != 0:
+                        logger.info('Layer update failed with --stop-on-error, stopping')
+                        sys.exit(1)
             if failed_layers:
                 for branch, err_msg_list in failed_layers.items():
                     if err_msg_list:
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index d9c93360..eb04fc44 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -57,7 +57,7 @@ def split_recipe_fn(path):
 
 patch_status_re = re.compile(r"^[\t ]*(Upstream[-_ ]Status:?)[\t ]*(\w+)([\t ]+.*)?", re.IGNORECASE | re.MULTILINE)
 
-def collect_patch(recipe, patchfn, layerdir_start):
+def collect_patch(recipe, patchfn, layerdir_start, stop_on_error):
     from django.db import DatabaseError
     from layerindex.models import Patch
 
@@ -93,10 +93,13 @@ def collect_patch(recipe, patchfn, layerdir_start):
     except DatabaseError:
         raise
     except Exception as e:
-        logger.error("Unable to read patch %s: %s", patchfn, str(e))
-        patchrec.save()
+        if stop_on_error:
+            raise
+        else:
+            logger.error("Unable to read patch %s: %s", patchfn, str(e))
+            patchrec.save()
 
-def collect_patches(recipe, envdata, layerdir_start):
+def collect_patches(recipe, envdata, layerdir_start, stop_on_error):
     from layerindex.models import Patch
 
     try:
@@ -111,9 +114,9 @@ def collect_patches(recipe, envdata, layerdir_start):
         if not patch.startswith(layerdir_start):
             # Likely a remote patch, skip it
             continue
-        collect_patch(recipe, patch, layerdir_start)
+        collect_patch(recipe, patch, layerdir_start, stop_on_error)
 
-def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir, skip_patches=False):
+def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir, stop_on_error, skip_patches=False):
     from django.db import DatabaseError
 
     fn = str(os.path.join(path, recipe.filename))
@@ -197,7 +200,7 @@ def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir, ski
 
         if not skip_patches:
             # Handle patches
-            collect_patches(recipe, envdata, layerdir_start)
+            collect_patches(recipe, envdata, layerdir_start, stop_on_error)
 
         # Get file dependencies within this layer
         deps = envdata.getVar('__depends', True)
@@ -235,9 +238,12 @@ def update_recipe_file(tinfoil, data, path, recipe, layerdir_start, repodir, ski
     except DatabaseError:
         raise
     except BaseException as e:
-        if not recipe.pn:
-            recipe.pn = recipe.filename[:-3].split('_')[0]
-        logger.error("Unable to read %s: %s", fn, str(e))
+        if stop_on_error:
+            raise
+        else:
+            if not recipe.pn:
+                recipe.pn = recipe.filename[:-3].split('_')[0]
+            logger.error("Unable to read %s: %s", fn, str(e))
 
 def update_machine_conf_file(path, machine):
     logger.debug('Updating machine %s' % path)
@@ -304,6 +310,9 @@ def main():
     parser.add_option("", "--nocheckout",
             help = "Don't check out branches",
             action="store_true", dest="nocheckout")
+    parser.add_option("", "--stop-on-error",
+            help = "Stop on first parsing error",
+            action="store_true", default=False, dest="stop_on_error")
     parser.add_option("-i", "--initial",
             help = "Print initial values parsed from layer.conf only",
             action="store_true")
@@ -523,7 +532,7 @@ def main():
                                     recipe.filepath = newfilepath
                                     recipe.filename = newfilename
                                     recipe.save()
-                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, newfilepath), recipe, layerdir_start, repodir, skip_patches)
+                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, newfilepath), recipe, layerdir_start, repodir, options.stop_on_error, skip_patches)
                                     updatedrecipes.add(os.path.join(oldfilepath, oldfilename))
                                     updatedrecipes.add(os.path.join(newfilepath, newfilename))
                                 else:
@@ -655,7 +664,7 @@ def main():
                                 results = layerrecipes.filter(filepath=filepath).filter(filename=filename)[:1]
                                 if results:
                                     recipe = results[0]
-                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, filepath), recipe, layerdir_start, repodir, skip_patches)
+                                    update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, filepath), recipe, layerdir_start, repodir, options.stop_on_error, skip_patches)
                                     recipe.save()
                                     updatedrecipes.add(recipe.full_path())
                             elif typename == 'machine':
@@ -677,7 +686,7 @@ def main():
 
                     for recipe in dirtyrecipes:
                         if not recipe.full_path() in updatedrecipes:
-                            update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, recipe.filepath), recipe, layerdir_start, repodir, skip_patches)
+                            update_recipe_file(tinfoil, config_data_copy, os.path.join(layerdir, recipe.filepath), recipe, layerdir_start, repodir, options.stop_on_error, skip_patches)
                 else:
                     # Collect recipe data from scratch
 
@@ -708,7 +717,7 @@ def main():
                                 # Recipe still exists, update it
                                 results = layerrecipes.filter(id=v['id'])[:1]
                                 recipe = results[0]
-                                update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir, skip_patches)
+                                update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir, options.stop_on_error, skip_patches)
                             else:
                                 # Recipe no longer exists, mark it for later on
                                 layerrecipes_delete.append(v)
@@ -777,7 +786,7 @@ def main():
                     recipe.filename = os.path.basename(added)
                     root = os.path.dirname(added)
                     recipe.filepath = os.path.relpath(root, layerdir)
-                    update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir, skip_patches)
+                    update_recipe_file(tinfoil, config_data_copy, root, recipe, layerdir_start, repodir, options.stop_on_error, skip_patches)
                     recipe.save()
 
                 for deleted in layerrecipes_delete:
-- 
2.17.1