[layerindex-web][PATCH 0/6] Patches from mailing list and fixes

[layerindex-web][PATCH 0/6] Patches from mailing list and fixes

[Tim Orling]

This series catches up on some patches that had already been sent to
the mailing list, just re-iterating them here exactly as they will be
merged, as well as a couple new patches.

A big "Thank You!" to Robert Yang for sharing fixes that Wind River was carrying
internally.

One new patch, sent on Michael Halstead's behalf, represents the current
production instance changes in order to meet the Python 3.8 minimum version
requirement for current bitbake.

One additional new patch replaces deprecated distutils usage (which was only
used for LooseVersion), with pkg_resources.parse_version instead.

The following changes since commit d8ab1beebeb5b49693b522d8618393620495bd74:

  layerindex/utils.py: ignore 'core' in BBFILES_COLLECTIONS (2022-04-28 08:15:22 -0700)

are available in the Git repository at:

  https://git.yoctoproject.org/layerindex-web timo/mailing-list-patches
  http://git.yoctoproject.org/cgit.cgi/layerindex-web/log/?h=timo/mailing-list-patches

Michael Halstead (2):
  Dockerfile: add captcha dependencies
  Dockerfile: switch to ubuntu:jammy for >python3.8

Robert Yang (3):
  utils.py: Use get_layer_var() to get BBFILE_COLLECTIONS
  admin.py: Remove has_delete_permission()
  utils.py: Fix for is_commit_ancestor()

Tim Orling (1):
  global: replace deprecated distutils

 Dockerfile                             | 23 ++++++++++++-----------
 layerindex/admin.py                    | 17 -----------------
 layerindex/recipedesc.py               |  1 -
 layerindex/tools/import_classic.py     |  1 -
 layerindex/tools/import_otherdistro.py |  1 -
 layerindex/update.py                   |  4 ++--
 layerindex/update_layer.py             |  6 +++---
 layerindex/utils.py                    | 20 +++++++++++---------
 layerindex/views.py                    |  6 ++----
 rrs/tools/upgrade_history_internal.py  |  4 ++--
 10 files changed, 32 insertions(+), 51 deletions(-)

-- 
2.34.1

[layerindex-web][PATCH 1/6] Dockerfile: add captcha dependencies

[Tim Orling]

From: Michael Halstead <mhalstead@linuxfoundation.org>

The current version of django-simple-captcha requires libz and freetype
development libraries.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Refactor to apply. Fix typo. Fix previous mixed use of space vs tab.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 Dockerfile | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Dockerfile b/Dockerfile
index 4b2870a..a6c7ad2 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -31,7 +31,9 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
 	python3-mysqldb \
 	python3-dev \
 	python3-pil \
-        python3-wheel \
+	python3-wheel \
+	libz-dev \
+	libfreetype6-dev \
 	libjpeg-dev \
 	libmariadbclient-dev \
 	locales \
-- 
2.34.1

[layerindex-web][PATCH 2/6] utils.py: Use get_layer_var() to get BBFILE_COLLECTIONS

[Tim Orling]

From: Robert Yang <liezhi.yang@windriver.com>

The commit 38e6288c7dad579518b0de2b6664f23be58889b6 has added oe-core to
BBLAYERS by default, and then there will be multiple BBFILE_COLLECTIONS which
causes update.py failure to update layers (most of the dependencies or
recommends are missing), use get_layer_var() to get BBFILE_COLLECTIONS to fix
the problem since it can ignore the extra oe-core from BBFILE_COLLECTIONS.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Fix minor typos. Minor rewording.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 layerindex/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/layerindex/utils.py b/layerindex/utils.py
index efd18e4..ed7fb47 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -188,7 +188,7 @@ def _add_dependency(var, name, layerbranch, config_data, logger=None, required=T
             logger.warn('Either set REMOVE_LAYER_DEPENDENCIES to remove them from the database, or fix conf/layer.conf')
 
 def set_layerbranch_collection_version(layerbranch, config_data, logger=None):
-    layerbranch.collection = config_data.getVar('BBFILE_COLLECTIONS', True)
+    layerbranch.collection = get_layer_var(config_data, 'BBFILE_COLLECTIONS', logger)
     ver_str = "LAYERVERSION_"
     if layerbranch.collection:
         layerbranch.collection = layerbranch.collection.strip()
-- 
2.34.1

[layerindex-web][PATCH 3/6] admin.py: Remove has_delete_permission()

[Tim Orling]

From: Robert Yang <liezhi.yang@windriver.com>

The following commit has added on_delete=models.CASCADE to the ForeignKey:

===
commit 2d526f9b0d363f3e442abc97ef9bd9fab37ee5e4
Author: Amber Elliot <amber.n.elliot@intel.com>
Date:   Thu Jun 25 15:57:34 2020 -0700

    Updating models and imports for Django 2.2 upgrade.
===

The on_delete=models.CASCADE will remove the objects which reference the
removed objects, so there will be errors when removing a LayerBranch
with admin permissions:

===
Cannot delete layer branch
Deleting the selected layer branch would result in deleting related objects, but your account doesn't have permission to delete the following types of objects:

Append
distro
===

This is because has_delete_permission() always return False, remove it to use
Django's implementation to fix the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Minor rewording.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 layerindex/admin.py | 17 -----------------
 1 file changed, 17 deletions(-)

diff --git a/layerindex/admin.py b/layerindex/admin.py
index 9ed6102..f8b2f6e 100644
--- a/layerindex/admin.py
+++ b/layerindex/admin.py
@@ -102,8 +102,6 @@ class RecipeAdmin(admin.ModelAdmin):
         return rofields
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class PackageConfigAdmin(admin.ModelAdmin):
     search_fields = ['feature', 'recipe__pn']
@@ -132,8 +130,6 @@ class ClassicRecipeAdmin(admin.ModelAdmin):
         return rofields
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class MachineAdmin(admin.ModelAdmin):
     search_fields = ['name']
@@ -141,8 +137,6 @@ class MachineAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in Machine._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class DistroAdmin(admin.ModelAdmin):
     search_fields = ['name']
@@ -150,9 +144,6 @@ class DistroAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in Distro._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
-
 
 class BBAppendAdmin(admin.ModelAdmin):
     search_fields = ['filename']
@@ -160,8 +151,6 @@ class BBAppendAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in BBAppend._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class BBClassAdmin(admin.ModelAdmin):
     search_fields = ['name']
@@ -169,8 +158,6 @@ class BBClassAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in BBClass._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class PatchAdmin(admin.ModelAdmin):
     search_fields = ['path']
@@ -178,8 +165,6 @@ class PatchAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in Patch._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class PatchDispositionAdmin(admin.ModelAdmin):
     fields = ['patch', 'user', 'disposition', 'comment']
@@ -199,8 +184,6 @@ class IncFileAdmin(admin.ModelAdmin):
     readonly_fields = [f.name for f in IncFile._meta.get_fields()]
     def has_add_permission(self, request, obj=None):
         return False
-    def has_delete_permission(self, request, obj=None):
-        return False
 
 class RecipeChangeInline(admin.StackedInline):
     model = RecipeChange
-- 
2.34.1

[layerindex-web][PATCH 4/6] utils.py: Fix for is_commit_ancestor()

[Tim Orling]

From: Robert Yang <liezhi.yang@windriver.com>

The runcmd() would print an "ERROR" on failure which causes confusion since
the failure is expected on old branches, so subprocess.getstatusoutput to fix
the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Minor rewording.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 layerindex/utils.py | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/layerindex/utils.py b/layerindex/utils.py
index ed7fb47..6566710 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -243,16 +243,18 @@ def is_commit_ancestor(repodir, commit, logger):
         # check if commit is a sha1 hash
         if re.match('[0-9a-f]{40}', commit):
             # check if the commit is an ancestor
-            contained = runcmd(['git', 'merge-base', '--is-ancestor', '%s' % commit, 'HEAD'], repodir, logger=logger)
-            return True
-        else:
-            raise Exception('is_commit_ancestor: "commit" must be a SHA1 hash')
-    except subprocess.CalledProcessError as e:
-            if e.returncode == 1:
-                # commit is not an ancestor
+            cmd = "GIT_DIR=%s/.git git merge-base --is-ancestor %s HEAD" % (repodir, commit)
+            logger.debug('Running "%s"' % cmd)
+            ret, output = subprocess.getstatusoutput(cmd)
+            if ret == 0:
+                return True
+            elif ret == 1:
+                logger.debug('output: %s' % output)
                 return False
             else:
-                raise e
+                raise Exception('Failed to run command: %s: ret: %s, output: %s' % (cmd, ret, output))
+        else:
+            raise Exception('is_commit_ancestor: "commit" must be a SHA1 hash')
     except Exception as esc:
         logger.warn(esc)
 
-- 
2.34.1

[layerindex-web][PATCH 5/6] Dockerfile: switch to ubuntu:jammy for >python3.8

[Tim Orling]

From: Michael Halstead <mhalstead@linuxfoundation.org>

ubuntu:jammy provides python 3.10.6 which supports current bitbake.
Dropped unneeded python2 packages.
Moved locale generation high to reduce error messages during image building.

Signed-off-by: Michael Halstead <mhalstead@linuxfoundation.org>
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 Dockerfile | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index a6c7ad2..d8c38e1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,6 +1,6 @@
 # See README for how to use this.
 
-FROM debian:buster
+FROM ubuntu:jammy
 LABEL maintainer="Michael Halstead <mhalstead@linuxfoundation.org>"
 
 ENV PYTHONUNBUFFERED=1 \
@@ -17,30 +17,29 @@ ENV PYTHONUNBUFFERED=1 \
 
 COPY requirements.txt /
 RUN DEBIAN_FRONTEND=noninteractive apt-get update \
+    && apt-get install -y locales \
+    && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \
+        && locale-gen en_US.UTF-8 \
+        && update-locale \
     && apt-get install -y --no-install-recommends \
 	autoconf \
 	g++ \
 	gcc \
 	make \
-	python-pip \
-	python-mysqldb \
-	python-dev \
-	python-pil \
-	python-wheel \
+	python2 \
 	python3-pip \
 	python3-mysqldb \
 	python3-dev \
 	python3-pil \
 	python3-wheel \
-	libz-dev \
+	zlib1g-dev \
 	libfreetype6-dev \
 	libjpeg-dev \
-	libmariadbclient-dev \
-	locales \
+	libmariadb-dev-compat \
 	netcat-openbsd \
 	curl \
 	wget \
-	git-core \
+	git \
 	vim \
 	rpm2cpio \
 	rpm \
@@ -51,7 +50,7 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update \
     && pip3 install gunicorn \
     && pip3 install setuptools \
     && pip3 install -r /requirements.txt \
-    && apt-get purge -y autoconf g++ make python3-dev libjpeg-dev libmariadbclient-dev \
+    && apt-get purge -y autoconf g++ make python3-dev libjpeg-dev \
 	&& apt-get autoremove -y \
 	&& rm -rf /var/lib/apt/lists/* \
 	&& apt-get clean
-- 
2.34.1

[layerindex-web][PATCH 6/6] global: replace deprecated distutils

[Tim Orling]

Replace distutils.version.LooseVersion with pkg_resource.parse_version

https://majornetwork.net/2021/05/comparing-version-numbers-in-python/

[YOCTO #14990]

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 layerindex/recipedesc.py               | 1 -
 layerindex/tools/import_classic.py     | 1 -
 layerindex/tools/import_otherdistro.py | 1 -
 layerindex/update.py                   | 4 ++--
 layerindex/update_layer.py             | 6 +++---
 layerindex/views.py                    | 6 ++----
 rrs/tools/upgrade_history_internal.py  | 4 ++--
 7 files changed, 9 insertions(+), 14 deletions(-)

diff --git a/layerindex/recipedesc.py b/layerindex/recipedesc.py
index a844186..33aef31 100644
--- a/layerindex/recipedesc.py
+++ b/layerindex/recipedesc.py
@@ -13,7 +13,6 @@ import logging
 import subprocess
 from datetime import datetime
 import fnmatch
-from distutils.version import LooseVersion
 
 
 logger = None
diff --git a/layerindex/tools/import_classic.py b/layerindex/tools/import_classic.py
index 0216221..44f8f88 100755
--- a/layerindex/tools/import_classic.py
+++ b/layerindex/tools/import_classic.py
@@ -22,7 +22,6 @@ import fnmatch
 import re
 import tempfile
 import shutil
-from distutils.version import LooseVersion
 import utils
 import recipeparse
 
diff --git a/layerindex/tools/import_otherdistro.py b/layerindex/tools/import_otherdistro.py
index 0c6640f..5c00115 100755
--- a/layerindex/tools/import_otherdistro.py
+++ b/layerindex/tools/import_otherdistro.py
@@ -23,7 +23,6 @@ import subprocess
 import string
 import shlex
 import codecs
-from distutils.version import LooseVersion
 
 sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), '..')))
 sys.path.insert(0, os.path.realpath(os.path.join(os.path.dirname(__file__), 'lib')))
diff --git a/layerindex/update.py b/layerindex/update.py
index 5a2642b..7de9c0d 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -17,7 +17,7 @@ import codecs
 import logging
 import subprocess
 from datetime import datetime, timedelta
-from distutils.version import LooseVersion
+from pkg_resources import parse_version
 import utils
 import operator
 import re
@@ -130,7 +130,7 @@ def extract_value(valuename, output):
         return ''
 
 def main():
-    if LooseVersion(git.__version__) < '0.3.1':
+    if parse_version(git.__version__) < parse_version('0.3.1'):
         logger.error("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
         sys.exit(1)
 
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index e8b210e..e0ecc94 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -19,7 +19,7 @@ import re
 import tempfile
 import shutil
 import errno
-from distutils.version import LooseVersion
+from pkg_resources import parse_version
 import itertools
 import utils
 import recipeparse
@@ -239,7 +239,7 @@ def update_distro_conf_file(path, distro, d):
         distro.description = desc
 
 def main():
-    if LooseVersion(git.__version__) < '0.3.1':
+    if parse_version(git.__version__) < parse_version('0.3.1'):
         logger.error("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
         sys.exit(1)
 
@@ -808,7 +808,7 @@ def main():
         logger.error(traceback.format_exc().rstrip())
         sys.exit(1)
     finally:
-        if tinfoil and (LooseVersion(bb.__version__) > LooseVersion("1.27")):
+        if tinfoil and (parse_version(bb.__version__) > parse_version("1.27")):
             tinfoil.shutdown()
 
     if tempdir:
diff --git a/layerindex/views.py b/layerindex/views.py
index 6f22c0c..e660578 100644
--- a/layerindex/views.py
+++ b/layerindex/views.py
@@ -41,7 +41,6 @@ from django.views.generic.base import RedirectView
 from django.views.generic.edit import (CreateView, DeleteView, FormView,
                                        UpdateView)
 from django_registration.backends.activation.views import RegistrationView
-from pkg_resources import parse_version
 from reversion.models import Revision
 
 import settings
@@ -1736,7 +1735,6 @@ class BranchCompareView(FormView):
     form_class = BranchComparisonForm
 
     def get_recipes(self, from_branch, to_branch, layer_ids):
-        from distutils.version import LooseVersion
         class BranchComparisonResult:
             def __init__(self, pn, short_desc):
                 self.pn = pn
@@ -1791,9 +1789,9 @@ class BranchCompareView(FormView):
         changed = []
         removed = []
         for _, recipe in sorted(recipes.items(), key=lambda item: item[0]):
-            recipe.from_versions = sorted(recipe.from_versions, key=lambda item: LooseVersion(item.pv))
+            recipe.from_versions = sorted(recipe.from_versions, key=lambda item: parse_version(item.pv))
             from_version_exprs = [x.version_expr() for x in recipe.from_versions]
-            recipe.to_versions = sorted(recipe.to_versions, key=lambda item: LooseVersion(item.pv))
+            recipe.to_versions = sorted(recipe.to_versions, key=lambda item: parser_version(item.pv))
             to_version_exprs = [x.version_expr() for x in recipe.to_versions]
             if not from_version_exprs:
                 added.append(recipe)
diff --git a/rrs/tools/upgrade_history_internal.py b/rrs/tools/upgrade_history_internal.py
index e3a96d9..2ba96b1 100644
--- a/rrs/tools/upgrade_history_internal.py
+++ b/rrs/tools/upgrade_history_internal.py
@@ -16,7 +16,7 @@ import os
 import optparse
 import logging
 import re
-from distutils.version import LooseVersion
+from pkg_resources import parse_version
 import git
 from datetime import datetime
 import calendar
@@ -561,7 +561,7 @@ def generate_history(options, layerbranch_id, commit, logger):
             pass
 
     finally:
-        if tinfoil and hasattr(tinfoil, 'shutdown') and (LooseVersion(bb.__version__) > LooseVersion("1.27")):
+        if tinfoil and hasattr(tinfoil, 'shutdown') and (parse_version(bb.__version__) > parse_version("1.27")):
             tinfoil.shutdown()
         utils.rmtree_force(tempdir)
 
-- 
2.34.1