[PATCH 1/4] lib/oe/recipeutils: return a dict in get_recipe_upgrade_status() instead of a tuple

[PATCH 1/4] lib/oe/recipeutils: return a dict in get_recipe_upgrade_status() instead of a tuple

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

Putting various things in a tuple is an anti-pattern of sorts, as the consumers
have to unpack it into local variables for readability, or access items directly
with indexes, which makes code pretty much unreadable.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oe/recipeutils.py                 |  2 +-
 meta/lib/oeqa/selftest/cases/distrodata.py |  4 ++--
 scripts/lib/devtool/upgrade.py             | 14 +++++++-------
 3 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 2d69a331132..f9d7dfe253a 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -1112,7 +1112,7 @@ def _get_recipe_upgrade_status(data):
     maintainer = data.getVar('RECIPE_MAINTAINER')
     no_upgrade_reason = data.getVar('RECIPE_NO_UPDATE_REASON')
 
-    return (pn, status, cur_ver, next_ver, maintainer, revision, no_upgrade_reason)
+    return {'pn':pn, 'status':status, 'cur_ver':cur_ver, 'next_ver':next_ver, 'maintainer':maintainer, 'revision':revision, 'no_upgrade_reason':no_upgrade_reason}
 
 def get_recipe_upgrade_status(recipes=None):
     pkgs_list = []
diff --git a/meta/lib/oeqa/selftest/cases/distrodata.py b/meta/lib/oeqa/selftest/cases/distrodata.py
index ad952c004b5..b60913dbca4 100644
--- a/meta/lib/oeqa/selftest/cases/distrodata.py
+++ b/meta/lib/oeqa/selftest/cases/distrodata.py
@@ -22,8 +22,8 @@ class Distrodata(OESelftestTestCase):
 
         pkgs = oe.recipeutils.get_recipe_upgrade_status()
 
-        regressed_failures = [pkg[0] for pkg in pkgs if pkg[1] == 'UNKNOWN_BROKEN']
-        regressed_successes = [pkg[0] for pkg in pkgs if pkg[1] == 'KNOWN_BROKEN']
+        regressed_failures = [pkg['pn'] for pkg in pkgs if pkg['status'] == 'UNKNOWN_BROKEN']
+        regressed_successes = [pkg['pn'] for pkg in pkgs if pkg['status'] == 'KNOWN_BROKEN']
         msg = ""
         if len(regressed_failures) > 0:
             msg = msg + """
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 8e13833b51c..10b4f8b5ee5 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -659,13 +659,13 @@ def check_upgrade_status(args, config, basepath, workspace):
     results = oe.recipeutils.get_recipe_upgrade_status(args.recipe)
     for result in results:
         # pn, update_status, current, latest, maintainer, latest_commit, no_update_reason
-        if args.all or result[1] != 'MATCH':
-            print("{:25} {:15} {:15} {} {} {}".format(   result[0],
-                                                               result[2],
-                                                               result[1] if result[1] != 'UPDATE' else (result[3] if not result[3].endswith("new-commits-available") else "new commits"),
-                                                               result[4],
-                                                               result[5] if result[5] != 'N/A' else "",
-                                                               "cannot be updated due to: %s" %(result[6]) if result[6] else ""))
+        if args.all or result['status'] != 'MATCH':
+            print("{:25} {:15} {:15} {} {} {}".format(   result['pn'],
+                                                               result['cur_ver'],
+                                                               result['status'] if result['status'] != 'UPDATE' else (result['next_ver'] if not result['next_ver'].endswith("new-commits-available") else "new commits"),
+                                                               result['maintainer'],
+                                                               result['revision'] if result['revision'] != 'N/A' else "",
+                                                               "cannot be updated due to: %s" %(result['no_upgrade_reason']) if result['no_upgrade_reason'] else ""))
 
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
-- 
2.39.2

[PATCH 2/4] lib/recipeutils: add a function to determine recipes with shared include files

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

This functionality is needed for 'lockstep version upgrades' where several
recipes need to be upgraded at the same time to produce a buildable
outcome.

The function itself obtains BBINCLUDED for each recipe and then massages
the data until it takes the form of a list of sets:

[{'cmake','cmake-native'},
 {'qemu','qemu-native','qemu-system-native'},
... ]

There's also a selftest that checks for the above.

Unfortunately this won't detect mutually exclusive recipes like mesa and mesa-gl
as they're chosen with PREFERRED_PROVIDER and can't be enabled in the same build
at the same time. ('devtool upgrade' will also accept just one of them but not the other)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oe/recipeutils.py                 | 37 ++++++++++++++++++++++
 meta/lib/oeqa/selftest/cases/distrodata.py | 12 +++++++
 2 files changed, 49 insertions(+)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index f9d7dfe253a..7586332fe0c 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -1183,3 +1183,40 @@ def get_recipe_upgrade_status(recipes=None):
         pkgs_list = executor.map(_get_recipe_upgrade_status, data_copy_list)
 
     return pkgs_list
+
+def get_common_include_recipes():
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=False)
+
+        recipes = tinfoil.all_recipe_files(variants=False)
+
+        recipeincludes = {}
+        for fn in recipes:
+            data = tinfoil.parse_recipe_file(fn)
+            recipeincludes[fn] = {'bbincluded':data.getVar('BBINCLUDED').split(),'pn':data.getVar('PN')}
+        return _get_common_include_recipes(recipeincludes)
+
+def _get_common_include_recipes(recipeincludes_all):
+        recipeincludes = {}
+        for fn,data in recipeincludes_all.items():
+            bbincluded_filtered = [i for i in data['bbincluded'] if os.path.dirname(i) == os.path.dirname(fn) and i != fn]
+            if bbincluded_filtered:
+                recipeincludes[data['pn']] = bbincluded_filtered
+
+        recipeincludes_inverted = {}
+        for k,v in recipeincludes.items():
+            for i in v:
+                recipeincludes_inverted.setdefault(i,set()).add(k)
+
+        recipeincludes_inverted_filtered = {k:v for k,v in recipeincludes_inverted.items() if len(v) > 1}
+
+        recipes_with_shared_includes = list()
+        for v in recipeincludes_inverted_filtered.values():
+            recipeset = v
+            for v1 in recipeincludes_inverted_filtered.values():
+                if recipeset.intersection(v1):
+                    recipeset.update(v1)
+            if recipeset not in recipes_with_shared_includes:
+                recipes_with_shared_includes.append(recipeset)
+
+        return recipes_with_shared_includes
diff --git a/meta/lib/oeqa/selftest/cases/distrodata.py b/meta/lib/oeqa/selftest/cases/distrodata.py
index b60913dbca4..bc561605220 100644
--- a/meta/lib/oeqa/selftest/cases/distrodata.py
+++ b/meta/lib/oeqa/selftest/cases/distrodata.py
@@ -115,3 +115,15 @@ The list of oe-core recipes with maintainers is empty. This may indicate that th
                 self.fail("""
 Unable to find recipes for the following entries in maintainers.inc:
 """ + "\n".join(['%s' % i for i in missing_recipes]))
+
+    def test_common_include_recipes(self):
+        """
+        Summary:     Test that obtaining recipes that share includes between them returns a sane result
+        Expected:    At least cmake and qemu entries are present in the output
+        Product:     oe-core
+        Author:      Alexander Kanavin <alex.kanavin@gmail.com>
+        """
+        recipes = oe.recipeutils.get_common_include_recipes()
+
+        self.assertIn({'qemu-system-native', 'qemu', 'qemu-native'}, recipes)
+        self.assertIn({'cmake-native', 'cmake'}, recipes)
-- 
2.39.2

[PATCH 3/4] recipeutils/get_recipe_upgrade_status: group recipes when they need to be upgraded together

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

This will allow 'lockstep upgrades' of such recipes, improving success
rates in automated version updating process.

devtool check-upgrade-status now prints:

These recipes need to be upgraded together {
glib-2.0                  2.80.2          2.80.4          Anuj Mittal <anuj.mittal@intel.com>
glib-2.0-initial          2.80.2          2.80.4          Anuj Mittal <anuj.mittal@intel.com>
}
These recipes need to be upgraded together {
util-linux                2.39.3          2.40.2          Chen Qi <Qi.Chen@windriver.com>
util-linux-libuuid        2.39.3          2.40.2          Chen Qi <Qi.Chen@windriver.com>
}
These recipes need to be upgraded together {
cmake                     3.29.3          3.30.0          Unassigned <unassigned@yoctoproject.org>
cmake-native              3.29.3          3.30.0          Unassigned <unassigned@yoctoproject.org>
}

etc.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oe/recipeutils.py                 | 23 +++++++++++++++++-
 meta/lib/oeqa/selftest/cases/distrodata.py |  6 ++---
 scripts/lib/devtool/upgrade.py             | 28 +++++++++++++++-------
 3 files changed, 44 insertions(+), 13 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 7586332fe0c..56be75dc9c9 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -1154,6 +1154,7 @@ def get_recipe_upgrade_status(recipes=None):
         if not recipes:
             recipes = tinfoil.all_recipe_files(variants=False)
 
+        recipeincludes = {}
         for fn in recipes:
             try:
                 if fn.startswith("/"):
@@ -1178,11 +1179,13 @@ def get_recipe_upgrade_status(recipes=None):
 
             data_copy_list.append(data_copy)
 
+            recipeincludes[data.getVar('FILE')] = {'bbincluded':data.getVar('BBINCLUDED').split(),'pn':data.getVar('PN')}
+
     from concurrent.futures import ProcessPoolExecutor
     with ProcessPoolExecutor(max_workers=utils.cpu_count()) as executor:
         pkgs_list = executor.map(_get_recipe_upgrade_status, data_copy_list)
 
-    return pkgs_list
+    return _group_recipes(pkgs_list, _get_common_include_recipes(recipeincludes))
 
 def get_common_include_recipes():
     with bb.tinfoil.Tinfoil() as tinfoil:
@@ -1220,3 +1223,21 @@ def _get_common_include_recipes(recipeincludes_all):
                 recipes_with_shared_includes.append(recipeset)
 
         return recipes_with_shared_includes
+
+def _group_recipes(recipes, groups):
+    recipedict = {}
+    for r in recipes:
+        recipedict[r['pn']] = r
+
+    recipegroups = []
+    for g in groups:
+        recipeset = []
+        for r in g:
+            if r in recipedict.keys():
+                recipeset.append(recipedict[r])
+                del recipedict[r]
+        recipegroups.append(recipeset)
+
+    for r in recipedict.values():
+        recipegroups.append([r])
+    return recipegroups
diff --git a/meta/lib/oeqa/selftest/cases/distrodata.py b/meta/lib/oeqa/selftest/cases/distrodata.py
index bc561605220..bd375523649 100644
--- a/meta/lib/oeqa/selftest/cases/distrodata.py
+++ b/meta/lib/oeqa/selftest/cases/distrodata.py
@@ -20,10 +20,10 @@ class Distrodata(OESelftestTestCase):
         feature = 'LICENSE_FLAGS_ACCEPTED += " commercial"\n'
         self.write_config(feature)
 
-        pkgs = oe.recipeutils.get_recipe_upgrade_status()
+        pkggroups = oe.recipeutils.get_recipe_upgrade_status()
 
-        regressed_failures = [pkg['pn'] for pkg in pkgs if pkg['status'] == 'UNKNOWN_BROKEN']
-        regressed_successes = [pkg['pn'] for pkg in pkgs if pkg['status'] == 'KNOWN_BROKEN']
+        regressed_failures = [pkg['pn'] for pkgs in pkggroups for pkg in pkgs if pkg['status'] == 'UNKNOWN_BROKEN']
+        regressed_successes = [pkg['pn'] for pkgs in pkggroups for pkg in pkgs if pkg['status'] == 'KNOWN_BROKEN']
         msg = ""
         if len(regressed_failures) > 0:
             msg = msg + """
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 10b4f8b5ee5..4c268af3a7c 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -654,18 +654,28 @@ def latest_version(args, config, basepath, workspace):
     return 0
 
 def check_upgrade_status(args, config, basepath, workspace):
+    def _print_status(recipe):
+        print("{:25} {:15} {:15} {} {} {}".format(   recipe['pn'],
+                                                               recipe['cur_ver'],
+                                                               recipe['status'] if recipe['status'] != 'UPDATE' else (recipe['next_ver'] if not recipe['next_ver'].endswith("new-commits-available") else "new commits"),
+                                                               recipe['maintainer'],
+                                                               recipe['revision'] if recipe['revision'] != 'N/A' else "",
+                                                               "cannot be updated due to: %s" %(recipe['no_upgrade_reason']) if recipe['no_upgrade_reason'] else ""))
     if not args.recipe:
         logger.info("Checking the upstream status for all recipes may take a few minutes")
     results = oe.recipeutils.get_recipe_upgrade_status(args.recipe)
-    for result in results:
-        # pn, update_status, current, latest, maintainer, latest_commit, no_update_reason
-        if args.all or result['status'] != 'MATCH':
-            print("{:25} {:15} {:15} {} {} {}".format(   result['pn'],
-                                                               result['cur_ver'],
-                                                               result['status'] if result['status'] != 'UPDATE' else (result['next_ver'] if not result['next_ver'].endswith("new-commits-available") else "new commits"),
-                                                               result['maintainer'],
-                                                               result['revision'] if result['revision'] != 'N/A' else "",
-                                                               "cannot be updated due to: %s" %(result['no_upgrade_reason']) if result['no_upgrade_reason'] else ""))
+    for recipegroup in results:
+        upgrades = [r for r in recipegroup if r['status'] != 'MATCH']
+        currents = [r for r in recipegroup if r['status'] == 'MATCH']
+        if len(upgrades) > 1:
+            print("These recipes need to be upgraded together {")
+        for r in upgrades:
+            _print_status(r)
+        if len(upgrades) > 1:
+            print("}")
+        for r in currents:
+            if args.all:
+                _print_status(r)
 
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
-- 
2.39.2

[PATCH 4/4] devtool/upgrade: use PN instead of BPN for naming newly created upgraded recipes

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

BPN isn't correct, as it is set to 'cmake' when 'cmake-native' is being upgraded
(or libva for libva-initial etc.)

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 scripts/lib/devtool/upgrade.py | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 4c268af3a7c..9ec66488516 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -76,19 +76,19 @@ def _rename_recipe_dirs(oldpv, newpv, path):
                         bb.utils.rename(os.path.join(path, oldfile),
                               os.path.join(path, newfile))
 
-def _rename_recipe_file(oldrecipe, bpn, oldpv, newpv, path):
+def _rename_recipe_file(oldrecipe, pn, oldpv, newpv, path):
     oldrecipe = os.path.basename(oldrecipe)
     if oldrecipe.endswith('_%s.bb' % oldpv):
-        newrecipe = '%s_%s.bb' % (bpn, newpv)
+        newrecipe = '%s_%s.bb' % (pn, newpv)
         if oldrecipe != newrecipe:
             shutil.move(os.path.join(path, oldrecipe), os.path.join(path, newrecipe))
     else:
         newrecipe = oldrecipe
     return os.path.join(path, newrecipe)
 
-def _rename_recipe_files(oldrecipe, bpn, oldpv, newpv, path):
+def _rename_recipe_files(oldrecipe, pn, oldpv, newpv, path):
     _rename_recipe_dirs(oldpv, newpv, path)
-    return _rename_recipe_file(oldrecipe, bpn, oldpv, newpv, path)
+    return _rename_recipe_file(oldrecipe, pn, oldpv, newpv, path)
 
 def _write_append(rc, srctreebase, srctree, same_dir, no_same_dir, revs, copied, workspace, d):
     """Writes an append file"""
@@ -335,19 +335,19 @@ def _add_license_diff_to_recipe(path, diff):
 def _create_new_recipe(newpv, checksums, srcrev, srcbranch, srcsubdir_old, srcsubdir_new, workspace, tinfoil, rd, license_diff, new_licenses, srctree, keep_failure):
     """Creates the new recipe under workspace"""
 
-    bpn = rd.getVar('BPN')
-    path = os.path.join(workspace, 'recipes', bpn)
+    pn = rd.getVar('PN')
+    path = os.path.join(workspace, 'recipes', pn)
     bb.utils.mkdirhier(path)
     copied, _ = oe.recipeutils.copy_recipe_files(rd, path, all_variants=True)
     if not copied:
-        raise DevtoolError('Internal error - no files were copied for recipe %s' % bpn)
+        raise DevtoolError('Internal error - no files were copied for recipe %s' % pn)
     logger.debug('Copied %s to %s' % (copied, path))
 
     oldpv = rd.getVar('PV')
     if not newpv:
         newpv = oldpv
     origpath = rd.getVar('FILE')
-    fullpath = _rename_recipe_files(origpath, bpn, oldpv, newpv, path)
+    fullpath = _rename_recipe_files(origpath, pn, oldpv, newpv, path)
     logger.debug('Upgraded %s => %s' % (origpath, fullpath))
 
     newvalues = {}
@@ -610,7 +610,7 @@ def upgrade(args, config, basepath, workspace):
             license_diff = _generate_license_diff(old_licenses, new_licenses)
             rf, copied = _create_new_recipe(args.version, checksums, args.srcrev, srcbranch, srcsubdir1, srcsubdir2, config.workspace_path, tinfoil, rd, license_diff, new_licenses, srctree, args.keep_failure)
         except (bb.process.CmdError, DevtoolError) as e:
-            recipedir = os.path.join(config.workspace_path, 'recipes', rd.getVar('BPN'))
+            recipedir = os.path.join(config.workspace_path, 'recipes', rd.getVar('PN'))
             _upgrade_error(e, recipedir, srctree, args.keep_failure)
         standard._add_md5(config, pn, os.path.dirname(rf))
 
-- 
2.39.2

Patchtest results for [PATCH 1/4] lib/oe/recipeutils: return a dict in get_recipe_upgrade_status() instead of a tuple

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3089 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/1-4-lib-oe-recipeutils-return-a-dict-in-get_recipe_upgrade_status-instead-of-a-tuple.patch

FAIL: test max line length: Patch line too long (current length 217, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

Patchtest results for [PATCH 3/4] recipeutils/get_recipe_upgrade_status: group recipes when they need to be upgraded together

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3091 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/3-4-recipeutils-get_recipe_upgrade_status-group-recipes-when-they-need-to-be-upgraded-together.patch

FAIL: test max line length: Patch line too long (current length 217, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
FAIL: test shortlog length: Edit shortlog so that it is 90 characters or less (currently 91 characters) (test_mbox.TestMbox.test_shortlog_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

Re: [OE-core] [PATCH 2/4] lib/recipeutils: add a function to determine recipes with shared include files

[Richard Purdie]

On Wed, 2024-07-17 at 20:22 +0200, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> This functionality is needed for 'lockstep version upgrades' where several
> recipes need to be upgraded at the same time to produce a buildable
> outcome.
> 
> The function itself obtains BBINCLUDED for each recipe and then massages
> the data until it takes the form of a list of sets:
> 
> [{'cmake','cmake-native'},
>  {'qemu','qemu-native','qemu-system-native'},
> ... ]
> 
> There's also a selftest that checks for the above.
> 
> Unfortunately this won't detect mutually exclusive recipes like mesa and mesa-gl
> as they're chosen with PREFERRED_PROVIDER and can't be enabled in the same build
> at the same time. ('devtool upgrade' will also accept just one of them but not the other)

This is partly why I was suggesting the internal list of files that
bitbake has instead of BBINCLUDED, even if it comes from the same data.

Even if a recipe is skipped (as conflicting providers are), I think the
cache entry in bitbake is still needed to know if/when to reparse the
recipe.

Your patch is good and I'm happy to merge as is, I just wanted to
mention that it might be possible to catch the mesa issue.

The challenge is that even if it were identifiable, the code still
probably can't know how to actually enable it for the upgrade/testing 
:(.

Cheers,

Richard

Re: [OE-core] [PATCH 2/4] lib/recipeutils: add a function to determine recipes with shared include files

[Alexander Kanavin]

On Mon, 22 Jul 2024 at 16:44, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Your patch is good and I'm happy to merge as is, I just wanted to
> mention that it might be possible to catch the mesa issue.

Thanks, I just pushed the corresponding changes to AUH. It's possible
next AUH run won't be perfect, we have to wait and see (I did test
passing/failing lockstep upggrades with glib, but AUH is notorious for
running into corner cases over the complete recipe set). But you
should be getting perfect qemu upgrade patches from now on ;)

> The challenge is that even if it were identifiable, the code still
> probably can't know how to actually enable it for the upgrade/testing
> :(.

Yes. We could insert that data into the recipe to be used by
AUH/devtool, but that's even more work, and all for a single known
recipe in core (mesa). For mesa I'm tempted to go back to the 'simple
file rename' proposal that's already implemented; in reality no one
tests mesa-gl either when they submit mesa version updates. And we
have to keep telling people to include mesa-gl, because it's only
natural to overlook that.

Alex

Re: [OE-core] [PATCH 2/4] lib/recipeutils: add a function to determine recipes with shared include files

[Richard Purdie]

On Tue, 2024-07-23 at 09:15 +0200, Alexander Kanavin wrote:
> On Mon, 22 Jul 2024 at 16:44, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > Your patch is good and I'm happy to merge as is, I just wanted to
> > mention that it might be possible to catch the mesa issue.
> 
> Thanks, I just pushed the corresponding changes to AUH. It's possible
> next AUH run won't be perfect, we have to wait and see (I did test
> passing/failing lockstep upggrades with glib, but AUH is notorious
> for
> running into corner cases over the complete recipe set). But you
> should be getting perfect qemu upgrade patches from now on ;)
> 
> > The challenge is that even if it were identifiable, the code still
> > probably can't know how to actually enable it for the
> > upgrade/testing
> > :(.
> 
> Yes. We could insert that data into the recipe to be used by
> AUH/devtool, but that's even more work, and all for a single known
> recipe in core (mesa). For mesa I'm tempted to go back to the 'simple
> file rename' proposal that's already implemented; in reality no one
> tests mesa-gl either when they submit mesa version updates. And we
> have to keep telling people to include mesa-gl, because it's only
> natural to overlook that.

FWIW, PV doesn't have to come from the filename. We could set PV from
the inc file and rename the files so PV isn't there...

Cheers,

Richard

Re: [OE-core] [PATCH 2/4] lib/recipeutils: add a function to determine recipes with shared include files

[Alexander Kanavin]

On Tue, 23 Jul 2024 at 09:21, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> FWIW, PV doesn't have to come from the filename. We could set PV from
> the inc file and rename the files so PV isn't there...

That's fine with me too. Let's do that then.

Alex