[layerindex-web][PATCH 0/6] Recipe Reporting System fixes

[layerindex-web][PATCH 0/6] Recipe Reporting System fixes

[Tim Orling]

This series fixes some syntax changes and other issues which have made
the Recipe Reporting System non-functional synce May 2020.

For now, the rrs/tools/rrs_distros.py script is commented out of the
rrs/tools/daily_run.sh since it requires changes in oe-core that may need
some discussion.

The other scripts which are part of rrs/tools/rrs_distros.py are now all
functional and have been live tested on layers.openembedded.org:
* rrs_maintainer_history.py
* rrs_upgrade_history.py
* rrs_upstream_history.py

The results can now be seen at https://layers.openembedded.org/rrs/

This series is applied on top of the previous "docker-fixes" series.

The following changes since commit 662ca9a4be52a21ed03e7421b51e5c1e1ef9afb1:

  docker-compose.yml: drop version '3' (2024-01-12 14:16:22 -0800)

are available in the Git repository at:

  https://github.com/moto-timo/layerindex-web rrs-fixes
  https://github.com/moto-timo/layerindex-web/tree/rrs-fixes

Tim Orling (6):
  rrs_maintainer_history: new override syntax
  rrs_maintainer_history: fix syntax
  rrs_upgrade_history: remap new srcrev fetcher API
  rrs_upstream_history: allow skip update_data
  rrs_upstream_history: fix get_recipe_pv_without_srcpv
  rrs_upstream_history: fix get_upstream_info

 rrs/tools/rrs_maintainer_history.py |  6 +++---
 rrs/tools/rrs_upgrade_history.py    |  1 +
 rrs/tools/rrs_upstream_history.py   | 30 ++++++++++++++++++++---------
 3 files changed, 25 insertions(+), 12 deletions(-)

-- 
2.34.1

[layerindex-web][PATCH 1/6] rrs_maintainer_history: new override syntax

[Tim Orling]

* Allow the regex for RECIPE_MAINTAINER to use either _ or : override
  syntax to allow processing maintainer changes across the change
  boundary.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 rrs/tools/rrs_maintainer_history.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rrs/tools/rrs_maintainer_history.py b/rrs/tools/rrs_maintainer_history.py
index 15e1d17..d2dd94a 100755
--- a/rrs/tools/rrs_maintainer_history.py
+++ b/rrs/tools/rrs_maintainer_history.py
@@ -37,7 +37,7 @@ MAINTAINERS_INCLUDE_PATH = 'conf/distro/include/maintainers.inc'
 """
 def get_recipe_maintainer(line, logger):
     import re
-    regex = re.compile('^RECIPE_MAINTAINER_pn-(?P<pn>.*)\s=\s"(?P<name>.+) <(?P<email>.*)>"$')
+    regex = re.compile('^RECIPE_MAINTAINER[:_]pn-(?P<pn>.*)\s=\s"(?P<name>.+) <(?P<email>.*)>"$')
 
     match = regex.search(line)
     if match:
-- 
2.34.1

[layerindex-web][PATCH 2/6] rrs_maintainer_history: fix syntax

[Tim Orling]

* Fix the link_maintainer.recipesymbol.pn syntax for debug logging

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 rrs/tools/rrs_maintainer_history.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/rrs/tools/rrs_maintainer_history.py b/rrs/tools/rrs_maintainer_history.py
index d2dd94a..a6da861 100755
--- a/rrs/tools/rrs_maintainer_history.py
+++ b/rrs/tools/rrs_maintainer_history.py
@@ -136,7 +136,7 @@ def maintainers_inc_history(options, logger, maintplan, layerbranch, repodir, la
                         rm.history = rms
                         rm.save()
                         if link_maintainer:
-                            logger.debug("%s: linked to maintainer for %s" % (recipe.pn, link_maintainer.recipe.pn))
+                            logger.debug("%s: linked to maintainer for %s" % (recipe.pn, link_maintainer.recipesymbol.pn))
                         else:
                             logger.debug("%s: Not found maintainer in commit %s set to 'No maintainer'." % \
                                             (recipe.pn, rms.sha1))
@@ -155,7 +155,7 @@ def maintainers_inc_history(options, logger, maintplan, layerbranch, repodir, la
                     rm.history = rms
                     rm.save()
                     if link_maintainer:
-                        logger.debug("%s: New recipe linked to maintainer for %s" % (recipe.pn, link_maintainer.recipe.pn))
+                        logger.debug("%s: New recipe linked to maintainer for %s" % (recipe.pn, link_maintainer.recipesymbol.pn))
                     else:
                         logger.debug("%s: New recipe not found maintainer set to 'No maintainer'." % \
                                     (recipe.pn))
-- 
2.34.1

[layerindex-web][PATCH 3/6] rrs_upgrade_history: remap new srcrev fetcher API

[Tim Orling]

When performing rrs_upgrade_history for commit date 8/24/2023,
the 'fetch2: Add new srcrev fetcher API' bitbake commit is not
automatically added.

Fix this by remapping 'cooker: Fix error message' (8/22/2023) to
the tag on 8/24/2023 'fetch2: git: Check if clone directory is
a git repo'.

This allows upgrade history for oe-core to continue across the
8/24/2023 date.

Fixes:
2024-01-08 23:59:07,114: DEBUG: run cmd '['git', 'show', '-s', '--format=%ci']' in /opt/workdir/git___git_openembedded_org_openembedded-core
2024-01-08 23:59:07,215: DEBUG: output: 2023-08-24 13:34:35 +0100
2024-01-08 23:59:07,227: DEBUG: run cmd '['git', 'rev-parse', 'HEAD']' in /opt/workdir/bitbake
2024-01-08 23:59:07,329: DEBUG: output: a06619951a43acb80b80d92e0caac560657ca249
2024-01-08 23:59:07,448: DEBUG: Running "GIT_DIR=/opt/workdir/bitbake/.git git merge-base --is-ancestor 87104b6a167188921da157c7dba45938849fb22a HEAD"
NOTE: Starting bitbake server...
ERROR: Failure expanding variable fetcher_hashes_dummyfunc[vardepvalue], expression was ${@bb.fetch.get_hashvalue(d)} which triggered exception AttributeError: module 'bb.fetch2' has no attribute 'get_hashvalue'
The variable dependency chain for the failure is: fetcher_hashes_dummyfunc[vardepvalue]

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 rrs/tools/rrs_upgrade_history.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/rrs/tools/rrs_upgrade_history.py b/rrs/tools/rrs_upgrade_history.py
index 9b74a05..42ba9b8 100755
--- a/rrs/tools/rrs_upgrade_history.py
+++ b/rrs/tools/rrs_upgrade_history.py
@@ -182,6 +182,7 @@ def upgrade_history(options, logger):
                 # Filter out some bad commits
                 remap_range('fef18b445c0cb6b266cd939b9c78d7cbce38663f', '39780b1ccbd76579db0fc6fb9369c848a3bafa9d')
                 remap_range('5796ed550d127853808f38257f8dcc8c1cf59342', '547128731e62b36d2271c4390b3fee2b16c535dc')
+                remap_range('a06619951a43acb80b80d92e0caac560657ca249', '2117db3146ce38bb4a6e2df40b6cd2ab11b514d5')
 
                 if options.stop_commit and (options.stop_commit not in [x.split()[0] for x in commit_list]):
                     logger.error('Stop commit %s is not in repository %s' % (options.stop_commit, repodir))
-- 
2.34.1

[layerindex-web][PATCH 4/6] rrs_upstream_history: allow skip update_data

[Tim Orling]

Fixes:
Traceback (most recent call last):
  File "/opt/layerindex/./rrs/tools/rrs_upstream_history.py", line 210, in <module>
    set_regexes(recipe_data)
  File "/opt/layerindex/./rrs/tools/rrs_upstream_history.py", line 73, in set_regexes
    bb.data.update_data(localdata)
AttributeError: module 'bb.data' has no attribute 'update_data'

bb.data.update_data was dropped in:
584989ed 'data/data_smart/build: Clean up datastore finalize/update_data references'

While we are here, drop some trailing whitespace.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 rrs/tools/rrs_upstream_history.py | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/rrs/tools/rrs_upstream_history.py b/rrs/tools/rrs_upstream_history.py
index 7dbf64c..25f4b20 100755
--- a/rrs/tools/rrs_upstream_history.py
+++ b/rrs/tools/rrs_upstream_history.py
@@ -70,7 +70,10 @@ def set_regexes(d):
 
             localdata.setVar('OVERRIDES', "pn-" + pnstripped + ":" +
                     d.getVar('OVERRIDES', True))
-            bb.data.update_data(localdata)
+            try:
+                bb.data.update_data(localdata)
+            except AttributeError:
+                pass
 
             for var in variables:
                 new_value = localdata.getVar(var, True)
@@ -107,7 +110,7 @@ def get_upstream_info(layerbranch, recipe_data, result):
         ru.date = ru_info['datetime']
 
         pv, _, _ = get_recipe_pv_without_srcpv(recipe_pv,
-                get_pv_type(recipe_pv)) 
+                get_pv_type(recipe_pv))
         upv, _, _ = get_recipe_pv_without_srcpv(ru_info['version'],
                 get_pv_type(ru_info['version']))
 
@@ -143,7 +146,7 @@ if __name__=="__main__":
     parser.add_option("-d", "--debug",
             help = "Enable debug output",
             action="store_const", const=logging.DEBUG, dest="loglevel", default=logging.INFO)
-    
+
     parser.add_option("--dry-run",
             help = "Do not write any data back to the database",
             action="store_true", dest="dry_run", default=False)
-- 
2.34.1

[layerindex-web][PATCH 5/6] rrs_upstream_history: fix get_recipe_pv_without_srcpv

[Tim Orling]

The get_recipe_pv_without_srcpv function was renamed to
get_recipe_pv_with_pfx_sfx in:

84794b59 lib/oe/recipeutils.py: accommodate SRCPV being optional and deprecated in version check regex

Try to import/call the old method and fail over to the new method.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 rrs/tools/rrs_upstream_history.py | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/rrs/tools/rrs_upstream_history.py b/rrs/tools/rrs_upstream_history.py
index 25f4b20..bf32c5d 100755
--- a/rrs/tools/rrs_upstream_history.py
+++ b/rrs/tools/rrs_upstream_history.py
@@ -87,8 +87,11 @@ def set_regexes(d):
 
 def get_upstream_info(layerbranch, recipe_data, result):
     from bb.utils import vercmp_string
-    from oe.recipeutils import get_recipe_upstream_version, \
-            get_recipe_pv_without_srcpv
+    from oe.recipeutils import get_recipe_upstream_version
+    try:
+        from oe.recipeutils import get_recipe_pv_without_srcpv
+    except ImportError:
+        from oe.recipeutils import get_recipe_pv_with_pfx_sfx
 
     pn = recipe_data.getVar('PN', True)
 
@@ -109,10 +112,16 @@ def get_upstream_info(layerbranch, recipe_data, result):
         ru.type = ru_info['type']
         ru.date = ru_info['datetime']
 
-        pv, _, _ = get_recipe_pv_without_srcpv(recipe_pv,
-                get_pv_type(recipe_pv))
-        upv, _, _ = get_recipe_pv_without_srcpv(ru_info['version'],
-                get_pv_type(ru_info['version']))
+        try:
+            pv, _, _ = get_recipe_pv_without_srcpv(recipe_pv,
+                       get_pv_type(recipe_pv))
+            upv, _, _ = get_recipe_pv_without_srcpv(ru_info['version'],
+                        get_pv_type(ru_info['version']))
+        except NameError:
+            pv, _, _ = get_recipe_pv_with_pfx_sfx(recipe_pv,
+                       get_pv_type(recipe_pv))
+            upv, _, _ = get_recipe_pv_with_pfx_sfx(ru_info['version'],
+                        get_pv_type(ru_info['version']))
 
         if pv and upv:
             cmp_ver = vercmp_string(pv, upv)
-- 
2.34.1

[layerindex-web][PATCH 6/6] rrs_upstream_history: fix get_upstream_info

[Tim Orling]

'result' is a list of RecipeUpstream objects... drop legacy 'recipe, ru'
syntax. In the current code base, result.append((recipe, ru)) was
appending a tuple of the last value of 'recipe' (in 'for recipe_qry:' under
'for maintplan in maintplans:') and the RecipeUpstream object.

Fixes:
AttributeError: 'tuple' object has no attribute 'history'

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 rrs/tools/rrs_upstream_history.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/rrs/tools/rrs_upstream_history.py b/rrs/tools/rrs_upstream_history.py
index bf32c5d..73b6d91 100755
--- a/rrs/tools/rrs_upstream_history.py
+++ b/rrs/tools/rrs_upstream_history.py
@@ -143,7 +143,7 @@ def get_upstream_info(layerbranch, recipe_data, result):
     ru.no_update_reason = recipe_data.getVar('RECIPE_NO_UPDATE_REASON',
             True) or ''
 
-    result.append((recipe, ru))
+    result.append(ru)
 
 if __name__=="__main__":
     parser = optparse.OptionParser(usage = """%prog [options]""")
-- 
2.34.1