[layerindex-web][PATCH 1/6] update.py: fix Ctrl+C behaviour

[layerindex-web][PATCH 1/6] update.py: fix Ctrl+C behaviour

[Paul Eggleton]

If the user hit Ctrl+C during the initial info gathering then it didn't
break out of the loop in update.py, so you had to hit Ctrl+C for as many
layers as were involved in the update. Look for exit code 254 from
update_layer.py and stop if it is returned since that indicates Ctrl+C
has been used.

Additionally, ensure we return exit code 254 and print a message from
the main update script when it is interrupted in this way.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 layerindex/update.py | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index 5c83b00..f60b943 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -337,7 +337,11 @@ def main():
                     logger.debug('Running layer update command: %s' % cmd)
                     ret, output = run_command_interruptible(cmd)
                     logger.debug('output: %s' % output)
-                    if ret != 0:
+                    if ret == 254:
+                        # Interrupted by user, break out of loop
+                        logger.info('Update interrupted, exiting')
+                        sys.exit(254)
+                    elif ret != 0:
                         continue
                     col = re.search("^BBFILE_COLLECTIONS = \"(.*)\"", output, re.M).group(1) or ''
                     ver = re.search("^LAYERVERSION = \"(.*)\"", output, re.M).group(1) or ''
@@ -418,10 +422,14 @@ def main():
 
                     if ret == 254:
                         # Interrupted by user, break out of loop
-                        break
+                        logger.info('Update interrupted, exiting')
+                        sys.exit(254)
         finally:
             utils.unlock_file(lockfile)
 
+    except KeyboardInterrupt:
+        logger.info('Update interrupted, exiting')
+        sys.exit(254)
     finally:
         update.log = ''.join(listhandler.read())
         update.finished = datetime.now()
-- 
2.14.3

[layerindex-web][PATCH 2/6] Add statistics page

[Paul Eggleton]

Add a page with basic statistics for the index - number of layers,
recipes, classes, machines and distros on an overall basis (distinct
names) and per branch, since I've been asked a few times for this kind
of information. It's currently only linked from the Tools menu for
logged-in users, but the URL will work for anyone.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 layerindex/urls.py              |  6 ++++-
 layerindex/views.py             | 17 ++++++++++++
 templates/base.html             |  1 +
 templates/layerindex/stats.html | 60 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 83 insertions(+), 1 deletion(-)
 create mode 100644 templates/layerindex/stats.html

diff --git a/layerindex/urls.py b/layerindex/urls.py
index 7deaf23..d2f9eab 100644
--- a/layerindex/urls.py
+++ b/layerindex/urls.py
@@ -8,7 +8,7 @@ from django.conf.urls import *
 from django.views.generic import TemplateView, DetailView, ListView, RedirectView
 from django.views.defaults import page_not_found
 from django.core.urlresolvers import reverse_lazy
-from layerindex.views import LayerListView, LayerReviewListView, LayerReviewDetailView, RecipeSearchView, MachineSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, HistoryListView, EditProfileFormView, AdvancedRecipeSearchView, BulkChangeView, BulkChangeSearchView, bulk_change_edit_view, bulk_change_patch_view, BulkChangeDeleteView, RecipeDetailView, RedirectParamsView, ClassicRecipeSearchView, ClassicRecipeDetailView, ClassicRecipeStatsView, LayerUpdateDetailView, UpdateListView, UpdateDetailView
+from layerindex.views import LayerListView, LayerReviewListView, LayerReviewDetailView, RecipeSearchView, MachineSearchView, PlainTextListView, LayerDetailView, edit_layer_view, delete_layer_view, edit_layernote_view, delete_layernote_view, HistoryListView, EditProfileFormView, AdvancedRecipeSearchView, BulkChangeView, BulkChangeSearchView, bulk_change_edit_view, bulk_change_patch_view, BulkChangeDeleteView, RecipeDetailView, RedirectParamsView, ClassicRecipeSearchView, ClassicRecipeDetailView, ClassicRecipeStatsView, LayerUpdateDetailView, UpdateListView, UpdateDetailView, StatsView
 from layerindex.models import LayerItem, Recipe, RecipeChangeset
 from rest_framework import routers
 from . import restviews
@@ -127,6 +127,10 @@ urlpatterns = patterns('',
         TemplateView.as_view(
             template_name='layerindex/about.html'),
             name="about"),
+    url(r'^stats/$',
+        StatsView.as_view(
+            template_name='layerindex/stats.html'),
+            name='stats'),
     url(r'^oe-classic/$',
         RedirectView.as_view(url=reverse_lazy('classic_recipe_search'), permanent=False),
             name='classic'),
diff --git a/layerindex/views.py b/layerindex/views.py
index 4f6c2c9..95812ca 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -1007,3 +1007,20 @@ class ClassicRecipeStatsView(TemplateView):
             'jquery_on_ready': False,
         }
         return context
+
+
+class StatsView(TemplateView):
+    def get_context_data(self, **kwargs):
+        context = super(StatsView, self).get_context_data(**kwargs)
+        context['layercount'] = LayerItem.objects.count()
+        context['recipe_count_distinct'] = Recipe.objects.values('pn').distinct().count()
+        context['class_count_distinct'] = BBClass.objects.values('name').distinct().count()
+        context['machine_count_distinct'] = Machine.objects.values('name').distinct().count()
+        context['distro_count_distinct'] = Distro.objects.values('name').distinct().count()
+        context['perbranch'] = Branch.objects.order_by('sort_priority').annotate(
+                layer_count=Count('layerbranch', distinct=True),
+                recipe_count=Count('layerbranch__recipe', distinct=True),
+                class_count=Count('layerbranch__bbclass', distinct=True),
+                machine_count=Count('layerbranch__machine', distinct=True),
+                distro_count=Count('layerbranch__distro', distinct=True))
+        return context
diff --git a/templates/base.html b/templates/base.html
index 8a3b8fe..a9d7c83 100644
--- a/templates/base.html
+++ b/templates/base.html
@@ -73,6 +73,7 @@
                             <li><a href="{% url 'bulk_change' %}">Bulk Change</a></li>
                             <li><a href="{% url 'duplicates' 'master' %}">Duplicates</a></li>
                             <li><a href="{% url 'update_list' %}">Updates</a></li>
+                            <li><a href="{% url 'stats' %}">Statistics</a></li>
                         </ul>
                     </li>
                     {% endif %}
diff --git a/templates/layerindex/stats.html b/templates/layerindex/stats.html
new file mode 100644
index 0000000..3c4971b
--- /dev/null
+++ b/templates/layerindex/stats.html
@@ -0,0 +1,60 @@
+{% extends "base.html" %}
+{% load i18n %}
+{% load static %}
+
+{% comment %}
+
+  layerindex-web - statistics page template
+
+  Copyright (C) 2018 Intel Corporation
+  Licensed under the MIT license, see COPYING.MIT for details
+
+{% endcomment %}
+
+
+<!--
+{% block title_append %} - statistics{% endblock %}
+-->
+
+{% block content %}
+{% autoescape on %}
+
+<h2>Statistics</h2>
+
+<h3>Overall</h3>
+<dl class="dl-horizontal">
+    <dt>Layers</dt><dd>{{ layercount }}</dd>
+    <dt>Recipes</dt><dd>{{ recipe_count_distinct }} (distinct names)</dd>
+    <dt>Machines</dt><dd>{{ machine_count_distinct }} (distinct names)</dd>
+    <dt>Classes</dt><dd>{{ class_count_distinct }} (distinct names)</dd>
+    <dt>Distros</dt><dd>{{ distro_count_distinct }} (distinct names)</dd>
+</dl>
+
+<h3>Per branch</h3>
+
+<table class="table">
+    <thead>
+        <th>Branch</th>
+        <th style="text-align: right">Layers</th>
+        <th style="text-align: right">Recipes</th>
+        <th style="text-align: right">Machines</th>
+        <th style="text-align: right">Classes</th>
+        <th style="text-align: right">Distros</th>
+    </thead>
+    <tbody>
+        {% for branch in perbranch %}
+        <tr {% if not branch.updates_enabled %}class="muted"{% endif %}>
+            <td>{{ branch.name }}</td>
+            <td style="text-align: right">{{ branch.layer_count }}</td>
+            <td style="text-align: right">{{ branch.recipe_count }}</td>
+            <td style="text-align: right">{{ branch.machine_count }}</td>
+            <td style="text-align: right">{{ branch.class_count }}</td>
+            <td style="text-align: right">{{ branch.distro_count }}</td>
+        </tr>
+        {% endfor %}
+    </tbody>
+</table>
+
+{% endautoescape %}
+{% endblock %}
+
-- 
2.14.3

[layerindex-web][PATCH 3/6] Show layer description with newlines in layer detail

[Paul Eggleton]

A lot of people enter line breaks in their layer descriptions hoping
that these will show up in the layer description, but of course since
they are being displayed as part of an HTML document, they don't by
default. Use a style in the description paragraph to ensure that they
do.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 templates/layerindex/detail.html | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/templates/layerindex/detail.html b/templates/layerindex/detail.html
index 0747421..3081ef3 100644
--- a/templates/layerindex/detail.html
+++ b/templates/layerindex/detail.html
@@ -76,9 +76,7 @@
 
             <div class="row-fluid">
                 <div class="description span7">
-                    <p>
-                       {{ layeritem.description }}
-                    </p>
+                    <p style="white-space: pre">{{ layeritem.description }}</p>
                     <p>
                         {% if layeritem.usage_url %}
                             <span class="label label-info">
-- 
2.14.3

[layerindex-web][PATCH 4/6] requirements.txt: add missing dependencies

[Paul Eggleton]

These are dependencies of items already listed in requirements.txt, so
nothing new.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 requirements.txt | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/requirements.txt b/requirements.txt
index 7cd741b..af6ab5a 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,3 +1,6 @@
+amqp==1.4.9
+anyjson==0.3.3
+billiard==3.3.0.23
 celery==3.1.25
 Django==1.8.17
 django-cors-headers==1.1.0
@@ -10,11 +13,13 @@ djangorestframework==3.2.5
 gitdb==0.6.4
 GitPython==2.0.5
 Jinja2==2.8
+kombu==3.0.37
 MarkupSafe==0.23
 mysqlclient==1.3.7
 Pillow==3.3.0
 python-nvd3==0.14.2
 python-slugify==1.1.4
+pytz==2018.3
 six==1.10.0
 smmap==0.9.0
 Unidecode==0.4.19
-- 
2.14.3

[layerindex-web][PATCH 5/6] requirements.txt: update some dependency versions

[Paul Eggleton]

Use latest tested versions (though we pin djangorestframework at 3.6.4
since that is the last version that supports Django 1.8), and add new
resulting dependencies.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 requirements.txt | 34 +++++++++++++++++++---------------
 1 file changed, 19 insertions(+), 15 deletions(-)

diff --git a/requirements.txt b/requirements.txt
index af6ab5a..58100a9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,25 +1,29 @@
-amqp==1.4.9
+amqp==2.2.2
 anyjson==0.3.3
-billiard==3.3.0.23
-celery==3.1.25
-Django==1.8.17
+billiard==3.5.0.3
+celery==4.1.0
+confusable-homoglyphs==2.0.2
+Django==1.8.18
 django-cors-headers==1.1.0
 django-nvd3==0.9.7
-django-registration==2.1
+django-ranged-response==0.2.0
+django-registration==2.4.1
 django-reversion==1.9.3
 django-reversion-compare==0.5.6
-django-simple-captcha==0.4.6
-djangorestframework==3.2.5
+django-simple-captcha==0.5.6
+djangorestframework==3.6.4
 gitdb==0.6.4
-GitPython==2.0.5
-Jinja2==2.8
-kombu==3.0.37
-MarkupSafe==0.23
-mysqlclient==1.3.7
-Pillow==3.3.0
+GitPython==2.1.8
+Jinja2==2.10
+kombu==4.1.0
+MarkupSafe==1.0
+mysqlclient==1.3.12
+Pillow==5.0.0
 python-nvd3==0.14.2
-python-slugify==1.1.4
+python-slugify==1.2.4
 pytz==2018.3
-six==1.10.0
+six==1.11.0
 smmap==0.9.0
+smmap2==2.0.3
 Unidecode==0.4.19
+vine==1.1.4
-- 
2.14.3

[layerindex-web][PATCH 6/6] README: update dependency versions

[Paul Eggleton]

Update the minor version of Django, and replace the list of dependencies
with a pointer to requirements.txt since there's not much point
maintaining that in two places (and the README wasn't complete anyway).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 README | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/README b/README
index c7f7409..970c267 100644
--- a/README
+++ b/README
@@ -12,7 +12,7 @@ Setup
 In order to make use of this application you will need:
 
 * Python 3.4+
-* Django 1.8.x - tested with 1.8.17; newer versions may work, but
+* Django 1.8.x - tested with 1.8.18; newer versions may work, but
   the application has not been tested with 1.9 or newer.
 * RabbitMQ 3.6.x - tested with 3.6.10.
 * For production usage, a web server set up to host Django applications
@@ -20,14 +20,9 @@ In order to make use of this application you will need:
 * A database supported by Django (SQLite, MySQL, etc.). Django takes
   care of creating the database itself, you just need to ensure that the
   database server (if not using SQLite) is configured and running.
-* The following third-party Django modules (tested versions listed):
-  * django-registration (2.1)
-  * django-reversion (1.9.3)
-  * django-reversion-compare (0.5.6)
-  * django-simple-captcha (0.4.6)
-  * django-nvd3 (0.14.2)
-  * djangorestframework (3.2.5)
-  * django-cors-headers (1.1.0)
+* Third-party Django/python modules as detailed by requirements.txt.
+  If you're using pip, you can install these by running:
+    pip install -r requirements.txt
 * On the machine that will run the backend update script (which does not
   have to be the same machine as the web server, however it does still
   have to have Django installed, have the same or similar configuration
-- 
2.14.3