[PATCH v5 0/5] Add bblock helper scripts

[PATCH v5 0/5] Add bblock helper scripts

[Julien Stephan]

Hi all,

This is v5 for bblock script.

Improvement from v4:
* Fix self test error reported by autobuilder
* Allow to specify task with/without the "do_" prefix:
  example: bblock bc -t compile,do_install
* Fix test_lock_single_recipe_all_tasks
* Fix getTaskSignatures: now retrieve all tasks and filter out the
* setscene and nostamps ones

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
  display a Note when recipe contains locked task(s)

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the
  following advantages:
    * findSigInfo needs the task to be already built to get the siginfo
      file, meaning we cannot lock a recipe on a fresh build
    * we can now generate the signatures for all available task of a given
      recipe
* Check if a given task is already locked. If so, don't duplicate
  entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some
corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you
  switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock
  only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying
  to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before
  locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (5):
  bitbake.conf: include bblock.conf
  bitbake: cooker: add a new function to retrieve task signatures
  sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
  scripts/bblock: add a script to lock/unlock recipes
  oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  31 ++++
 bitbake/lib/bb/event.py                |   8 +
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  18 ++-
 meta/lib/oeqa/selftest/cases/bblock.py | 201 +++++++++++++++++++++++++
 scripts/bblock                         | 184 ++++++++++++++++++++++
 7 files changed, 448 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

[PATCH v5 1/5] bitbake.conf: include bblock.conf

[Julien Stephan]

include conf/bblock.conf. This file is generated by the bblock tool. It
locks some package tasks by fixing their signatures. See bblock -h for
more details

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 67577c8fe7e..6a8deee4f72 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -829,6 +829,7 @@ include conf/documentation.conf
 include conf/licenses.conf
 require conf/sanity.conf
 require conf/cve-check-map.conf
+include conf/bblock.conf
 
 ##################################################################
 # Weak variables (usually to retain backwards compatibility)
-- 
2.41.0

[PATCH v5 2/5] bitbake: cooker: add a new function to retrieve task signatures

[Julien Stephan]

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the
signatures using tinfoil

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 31 +++++++++++++++++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 45 insertions(+)

diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index 8663eed9331..f2ee5871616 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -781,3 +781,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 87aa71bb657..599c7ddaa28 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1442,6 +1442,37 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
+    def getTaskSignatures(self, target, tasks):
+        sig = []
+        getAllTaskSignatures = False
+
+        if not tasks:
+            tasks = ["do_build"]
+            getAllTaskSignatures = True
+
+        for task in tasks:
+            taskdata, runlist = self.buildTaskData(target, task, self.configuration.halt)
+            rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+            rq.rqdata.prepare()
+
+            for l in runlist:
+                mc, pn, taskname, fn = l
+
+                taskdep = rq.rqdata.dataCaches[mc].task_deps[fn]
+                for t in taskdep['tasks']:
+                    if t in taskdep['nostamp'] or "setscene" in t:
+                        continue
+                    tid = bb.runqueue.build_tid(mc, fn, t)
+
+                    if t in task or getAllTaskSignatures:
+                        try:
+                            rq.rqdata.prepare_task_hash(tid)
+                            sig.append([pn, t, rq.rqdata.get_task_unihash(tid)])
+                        except KeyError:
+                            sig.append(self.getTaskSignatures(target, [t])[0])
+
+        return sig
+
     def buildTargets(self, targets, task):
         """
         Attempt to build the targets specified
diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 0d0e0a68aac..f8acacd80d1 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
         Event.__init__(self)
         self.result = result
 
+class GetTaskSignatureResult(Event):
+    """
+    Event to return results from GetTaskSignatures command
+    """
+    def __init__(self, sig):
+        Event.__init__(self)
+        self.sig = sig
+
 class ParseError(Event):
     """
     Event to indicate parse failed
-- 
2.41.0

[PATCH v5 3/5] sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK

[Julien Stephan]

as of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and
"error", displaying respectively a warning or a fatal error message
only when a task is locked and the task signature is different from
the locked one.

The "info" level is introduced to add a "note" message to remind the
user that a recipe is locked even if the signature is equivalent to the
locked one.

The "warn" and "error" level display the warn/error message for each
task having a mismatch of the signature. Doing this with the "info"
level would result in very verbose output if there are several tasks
locked, so the info level will only print once the list of recipes that
have locked signature.

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/lib/oe/sstatesig.py | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 633a0fd4502..5bf1697e727 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -104,6 +104,8 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         self.lockedhashfn = {}
         self.machine = data.getVar("MACHINE")
         self.mismatch_msgs = []
+        self.mismatch_number = 0
+        self.lockedsigs_msgs = ""
         self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or
                                 "").split()
         self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
@@ -187,6 +189,7 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                 #bb.warn("Using %s %s %s" % (recipename, task, h))
 
                 if h != h_locked and h_locked != unihash:
+                    self.mismatch_number += 1
                     self.mismatch_msgs.append('The %s:%s sig is computed to be %s, but the sig is locked to %s in %s'
                                           % (recipename, task, h, h_locked, var))
 
@@ -267,6 +270,15 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         warn_msgs = []
         error_msgs = []
         sstate_missing_msgs = []
+        info_msgs = None
+
+        if self.lockedsigs:
+            if len(self.lockedsigs) > 10:
+                self.lockedsigs_msgs = "There are %s recipes with locked tasks (%s task(s) have non matching signature)" % (len(self.lockedsigs), self.mismatch_number)
+            else:
+                self.lockedsigs_msgs = "The following recipes have locked tasks:"
+                for pn in self.lockedsigs:
+                    self.lockedsigs_msgs += " %s" % (pn)
 
         for tid in sq_data['hash']:
             if tid not in found:
@@ -279,7 +291,9 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                                                % (pn, taskname, sq_data['hash'][tid]))
 
         checklevel = d.getVar("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK")
-        if checklevel == 'warn':
+        if checklevel == 'info':
+            info_msgs = self.lockedsigs_msgs
+        if checklevel == 'warn' or checklevel == 'info':
             warn_msgs += self.mismatch_msgs
         elif checklevel == 'error':
             error_msgs += self.mismatch_msgs
@@ -290,6 +304,8 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         elif checklevel == 'error':
             error_msgs += sstate_missing_msgs
 
+        if info_msgs:
+            bb.note(info_msgs)
         if warn_msgs:
             bb.warn("\n".join(warn_msgs))
         if error_msgs:
-- 
2.41.0

[PATCH v5 4/5] scripts/bblock: add a script to lock/unlock recipes

[Julien Stephan]

bblock script allows to lock/unlock recipes to latest task signatures.
The idea is to prevent some recipes to be rebuilt during development.
For example when working on rust recipe, one may not want rust-native to be
rebuilt.

This tool can be used, with proper environment set up, using the following
command:

bblock <recipe_name>

See help for more details

if a <recipe_name>'s task signature change, this task will not be built again and
sstate cache will be used.

[YOCTO #13425]

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/bblock | 184 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 184 insertions(+)
 create mode 100755 scripts/bblock

diff --git a/scripts/bblock b/scripts/bblock
new file mode 100755
index 00000000000..0082059af81
--- /dev/null
+++ b/scripts/bblock
@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def getTaskSignatures(tinfoil, pn, tasks):
+    tinfoil.set_event_mask(
+        [
+            "bb.event.GetTaskSignatureResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("getTaskSignatures", pn, tasks)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.GetTaskSignatureResult):
+                    sig = event.sig
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from getTaskSignatures command")
+        sys.exit(2)
+    return sig
+
+
+def parseRecipe(tinfoil, recipe):
+    try:
+        tinfoil.parse_recipes()
+        d = tinfoil.parse_recipe(recipe)
+    except Exception:
+        logger.error("Failed to get recipe info for: %s" % recipe)
+        sys.exit(1)
+    return d
+
+
+def bblockDump(lockfile):
+    try:
+        with open(lockfile, "r") as lockfile:
+            for line in lockfile:
+                print(line.strip())
+    except IOError:
+        return 1
+    return 0
+
+
+def bblockReset(lockfile, pns, package_archs, tasks):
+    if not pns:
+        logger.info("Unlocking all recipes")
+        try:
+            os.remove(lockfile)
+        except FileNotFoundError:
+            pass
+    else:
+        logger.info("Unlocking {pns}".format(pns=pns))
+        tmp_lockfile = lockfile + ".tmp"
+        with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+            for line in infile:
+                if not (
+                    any(element in line for element in pns)
+                    and any(element in line for element in package_archs.split())
+                ):
+                    outfile.write(line)
+                else:
+                    if tasks and not any(element in line for element in tasks):
+                        outfile.write(line)
+        os.remove(lockfile)
+        os.rename(tmp_lockfile, lockfile)
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+    parser.add_argument(
+        "-t",
+        "--tasks",
+        help="Comma separated list of tasks",
+        type=lambda s: [
+            task if task.startswith("do_") else "do_" + task for task in s.split(",")
+        ],
+    )
+    parser.add_argument(
+        "-r",
+        "--reset",
+        action="store_true",
+        help="Unlock pn recipes, or all recipes if pn is empty",
+    )
+    parser.add_argument(
+        "-d",
+        "--dump",
+        action="store_true",
+        help="Dump generated bblock.conf file",
+    )
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+
+        package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+        if global_args.dump:
+            bblockDump(lockfile)
+            return 0
+
+        if global_args.reset:
+            bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks)
+            return 0
+
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+                s += "\n"
+
+            for pn in global_args.pn:
+                d = parseRecipe(tinfoil, pn)
+                package_arch = d.getVar("PACKAGE_ARCH")
+                siggen_locked_sigs_package_arch = d.getVar(
+                    "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch)
+                )
+                sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks)
+                for sig in sigs:
+                    new_entry = "{pn}:{taskname}:{sig}".format(
+                        pn=sig[0], taskname=sig[1], sig=sig[2]
+                    )
+                    if (
+                        siggen_locked_sigs_package_arch
+                        and not new_entry in siggen_locked_sigs_package_arch
+                    ) or not siggen_locked_sigs_package_arch:
+                        s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format(
+                            package_arch=package_arch, new_entry=new_entry
+                        )
+            lockfile.write(s)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.41.0

[PATCH v5 5/5] oeqa/selftest/bblock: add self test for bblock tool

[Julien Stephan]

it implements various combination of locking single/multiple recipe(s)/task(s)

it also tests that locked sig are architecture dependant

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/lib/oeqa/selftest/cases/bblock.py | 201 +++++++++++++++++++++++++
 1 file changed, 201 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py

diff --git a/meta/lib/oeqa/selftest/cases/bblock.py b/meta/lib/oeqa/selftest/cases/bblock.py
new file mode 100644
index 00000000000..07143c59676
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/bblock.py
@@ -0,0 +1,201 @@
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import re
+import bb.tinfoil
+
+import oeqa.utils.ftools as ftools
+from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake
+
+from oeqa.selftest.case import OESelftestTestCase
+
+
+class BBLock(OESelftestTestCase):
+    @classmethod
+    def setUpClass(cls):
+        super(BBLock, cls).setUpClass()
+        cls.lockfile = cls.builddir + "/conf/bblock.conf"
+
+    def unlock_recipes(self, recipes=None, tasks=None):
+        cmd = "bblock -r "
+        if recipes:
+            cmd += " ".join(recipes)
+        if tasks:
+            cmd += " -t " + ",".join(tasks)
+        result = runCmd(cmd)
+
+        if recipes:
+            # ensure all signatures are removed from lockfile
+            contents = ftools.read_file(self.lockfile)
+            for recipe in recipes:
+                for task in tasks:
+                    find_in_contents = re.search(
+                        'SIGGEN_LOCKEDSIGS_.+\s\+=\s"%s:%s:.*"' % (recipe, task),
+                        contents,
+                    )
+                    self.assertFalse(
+                        find_in_contents,
+                        msg="%s:%s should not be present into bblock.conf anymore"
+                        % (recipe, task),
+                    )
+                self.assertExists(self.lockfile)
+        else:
+            self.assertNotExists(self.lockfile)
+
+    def lock_recipes(self, recipes, tasks=None):
+        cmd = "bblock " + " ".join(recipes)
+        if tasks:
+            cmd += " -t " + ",".join(tasks)
+
+        result = runCmd(cmd)
+
+        self.assertExists(self.lockfile)
+
+        # ensure all signatures are added to lockfile
+        contents = ftools.read_file(self.lockfile)
+        for recipe in recipes:
+            if tasks:
+                for task in tasks:
+                    find_in_contents = re.search(
+                        'SIGGEN_LOCKEDSIGS_.+\s\+=\s"%s:%s:.*"' % (recipe, task),
+                        contents,
+                    )
+                    self.assertTrue(
+                        find_in_contents,
+                        msg="%s:%s was not added into bblock.conf. bblock output: %s"
+                        % (recipe, task, result.output),
+                    )
+
+    def modify_tasks(self, recipes, tasks):
+        task_append = ""
+        for recipe in recipes:
+            bb_vars = get_bb_vars(["PV"], recipe)
+            recipe_pv = bb_vars["PV"]
+            recipe_append_file = recipe + "_" + recipe_pv + ".bbappend"
+
+            os.mkdir(os.path.join(self.testlayer_path, "recipes-test", recipe))
+            recipe_append_path = os.path.join(
+                self.testlayer_path, "recipes-test", recipe, recipe_append_file
+            )
+
+            for task in tasks:
+                task_append += "%s:append() {\n#modify task hash \n}\n" % task
+            ftools.write_file(recipe_append_path, task_append)
+            self.add_command_to_tearDown(
+                "rm -rf %s" % os.path.join(self.testlayer_path, "recipes-test", recipe)
+            )
+
+    def test_lock_single_recipe_single_task(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_single_recipe_multiple_tasks(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile", "do_install"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_single_recipe_all_tasks(self):
+        recipes = ["quilt"]
+        self._run_test(recipes, None)
+
+    def test_lock_multiple_recipe_single_task(self):
+        recipes = ["quilt", "bc"]
+        tasks = ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_architecture_specific(self):
+        # unlock all recipes and ensure no bblock.conf file exist
+        self.unlock_recipes()
+
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+
+        # lock quilt's do_compile task for another machine
+        if self.td["MACHINE"] == "qemux86-64":
+            machine = "qemuarm"
+        else:
+            machine = "qemux86-64"
+
+        self.write_config('MACHINE = "%s"\n' % machine)
+
+        self.lock_recipes(recipes, tasks)
+
+        self.write_config('MACHINE = "%s"\n' % self.td["MACHINE"])
+        # modify quilt's do_compile task
+        self.modify_tasks(recipes, tasks)
+
+        # build quilt using the default machine
+        # No Note/Warning should be emitted since sig is locked for another machine
+        # (quilt package is architecture dependant)
+        info_message = "NOTE: The following recipes have locked tasks: " + recipes[0]
+        warn_message = "The %s:%s sig is computed to be" % (recipes[0], tasks[0])
+        result = bitbake(recipes[0] + " -n")
+        self.assertNotIn(info_message, result.output)
+        self.assertNotIn(warn_message, result.output)
+
+        # unlock all recipes
+        self.unlock_recipes()
+
+    def _run_test(self, recipes, tasks=None):
+        # unlock all recipes and ensure no bblock.conf file exist
+        self.unlock_recipes()
+
+        # lock tasks for recipes
+        result = self.lock_recipes(recipes, tasks)
+
+        if not tasks:
+            tasks = []
+            result = bitbake("-c listtasks " + recipes[0])
+            with bb.tinfoil.Tinfoil() as tinfoil:
+                tinfoil.prepare(config_only=False, quiet=2)
+                d = tinfoil.parse_recipe(recipes[0])
+
+                for line in result.output.splitlines():
+                    if line.startswith("do_"):
+                        task = line.split()[0]
+                        if "setscene" in task:
+                            continue
+                        if d.getVarFlag(task, "nostamp"):
+                            continue
+                        tasks.append(task)
+
+        # build recipes. At this stage we should have a Note about recipes
+        # having locked task's sig, but no warning since sig still match
+        info_message = "NOTE: The following recipes have locked tasks: " + " ".join(
+            recipes
+        )
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
+
+        # modify all tasks that are locked to trigger a sig change then build the recipes
+        # at this stage we should have a Note as before, but also a Warning for all
+        # locked tasks indicating the sig mismatch
+        self.modify_tasks(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertIn(warn_message, result.output)
+
+        # unlock all tasks and rebuild, no more Note/Warning should remain
+        self.unlock_recipes(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertNotIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
+
+        # unlock all recipes
+        self.unlock_recipes()
-- 
2.41.0

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Alexandre Belloni]

Hello,

I had a build were all the tests passed, then I had this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio

On 25/09/2023 10:04:47+0200, Julien Stephan wrote:
> Hi all,
> 
> This is v5 for bblock script.
> 
> Improvement from v4:
> * Fix self test error reported by autobuilder
> * Allow to specify task with/without the "do_" prefix:
>   example: bblock bc -t compile,do_install
> * Fix test_lock_single_recipe_all_tasks
> * Fix getTaskSignatures: now retrieve all tasks and filter out the
> * setscene and nostamps ones
> 
> Limitations:
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
> 
> Improvement from v3:
> * Add self test
> * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
>   display a Note when recipe contains locked task(s)
> 
> Improvement from v2:
> * Add a function in bb.cooker to compute task signatures
> * Replace the findSigInfo function by the new created one. This has the
>   following advantages:
>     * findSigInfo needs the task to be already built to get the siginfo
>       file, meaning we cannot lock a recipe on a fresh build
>     * we can now generate the signatures for all available task of a given
>       recipe
> * Check if a given task is already locked. If so, don't duplicate
>   entry in bblock.conf
> 
> Limitations:
> * Needs to taint tasks that are locked, to display a warning
> * I may be still missing some checks on user input
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
> 
> I did some tests using qemux86-64 and qemuarm but I may be missing some
> corner cases.
> 
> Improvement from V1:
> * Signatures are now package architecture specific meaning that if you
>   switch MACHINE, the lock sig will not be taken into account
> * I added the -r option to unlock recipes
> * I added a -d option to display the current bblock.conf
> * Added an include directive for conf/bblock.conf inside bitbake.conf
> * Added -t option to specify the tasks to lock/unlock
> 
> Limitations:
> * I may be still missing some checks on user input
> * I need to find a way to get the list of tasks ( by default still lock
>   only the do_compile for now, unless -t is specified)
> * Do not check if a particular recipe/task is already locked when trying
>   to add lock. So entries may appear multiple times
> * We still need the signature of the tasks to be already computed before
>   locking. Need to find a way to generate it if missing
> 
> V3: https://lists.openembedded.org/g/openembedded-core/message/184932
> V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> V1: https://lists.openembedded.org/g/openembedded-core/message/184584
> 
> My branch is available here [1]
> 
> Cheers
> Julien
> 
> [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> 
> Julien Stephan (5):
>   bitbake.conf: include bblock.conf
>   bitbake: cooker: add a new function to retrieve task signatures
>   sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
>   scripts/bblock: add a script to lock/unlock recipes
>   oeqa/selftest/bblock: add self test for bblock tool
> 
>  bitbake/lib/bb/command.py              |   6 +
>  bitbake/lib/bb/cooker.py               |  31 ++++
>  bitbake/lib/bb/event.py                |   8 +
>  meta/conf/bitbake.conf                 |   1 +
>  meta/lib/oe/sstatesig.py               |  18 ++-
>  meta/lib/oeqa/selftest/cases/bblock.py | 201 +++++++++++++++++++++++++
>  scripts/bblock                         | 184 ++++++++++++++++++++++
>  7 files changed, 448 insertions(+), 1 deletion(-)
>  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
>  create mode 100755 scripts/bblock
> 
> --
> 2.41.0
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188174): https://lists.openembedded.org/g/openembedded-core/message/188174
> Mute This Topic: https://lists.openembedded.org/mt/101570279/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Julien Stephan]

Le mer. 27 sept. 2023 à 12:16, Alexandre Belloni
<alexandre.belloni@bootlin.com> a écrit :
>
> Hello,
>
> I had a build were all the tests passed, then I had this failure:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
>

Hi Alexandre,
I only see

   bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED

but I can't see the actual error :( Is it possible to have it somewhere?

Cheers
Julien

>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Alexandre Belloni]

On 27/09/2023 14:47:57+0200, Julien Stephan wrote:
> Le mer. 27 sept. 2023 � 12:16, Alexandre Belloni
> <alexandre.belloni@bootlin.com> a �crit :
> >
> > Hello,
> >
> > I had a build were all the tests passed, then I had this failure:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> >
> 
> Hi Alexandre,
> I only see
> 
>    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
> 
> but I can't see the actual error :( Is it possible to have it somewhere?
> 

You can scroll up or click on the looking glass on the top right.

Alternatively, you can download the log here:
https://autobuilder.yoctoproject.org/typhoon/api/v2/logs/8276762/raw
(the button is also on the top right)


The assertion error is fairly large, that's why I didn't copy it.


> Cheers
> Julien
> 
> >
> >
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Alexander Kanavin]

You need to scroll up. The failure is in lines 557 to 609.

Alex


On Wed, 27 Sept 2023 at 14:48, Julien Stephan <jstephan@baylibre.com> wrote:
>
> Le mer. 27 sept. 2023 à 12:16, Alexandre Belloni
> <alexandre.belloni@bootlin.com> a écrit :
> >
> > Hello,
> >
> > I had a build were all the tests passed, then I had this failure:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> >
>
> Hi Alexandre,
> I only see
>
>    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
>
> but I can't see the actual error :( Is it possible to have it somewhere?
>
> Cheers
> Julien
>
> >
> >
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188313): https://lists.openembedded.org/g/openembedded-core/message/188313
> Mute This Topic: https://lists.openembedded.org/mt/101570279/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Julien Stephan]

Le mer. 27 sept. 2023 à 14:59, Alexander Kanavin
<alex.kanavin@gmail.com> a écrit :
>
> You need to scroll up. The failure is in lines 557 to 609.

Hmm stange ctrl-f didn't work. I can see it now. Thank you, I'll look at it

Julien
>
> Alex
>
>
> On Wed, 27 Sept 2023 at 14:48, Julien Stephan <jstephan@baylibre.com> wrote:
> >
> > Le mer. 27 sept. 2023 à 12:16, Alexandre Belloni
> > <alexandre.belloni@bootlin.com> a écrit :
> > >
> > > Hello,
> > >
> > > I had a build were all the tests passed, then I had this failure:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> > >
> >
> > Hi Alexandre,
> > I only see
> >
> >    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
> >
> > but I can't see the actual error :( Is it possible to have it somewhere?
> >
> > Cheers
> > Julien
> >
> > >
> > >
> > > --
> > > Alexandre Belloni, co-owner and COO, Bootlin
> > > Embedded Linux and Kernel engineering
> > > https://bootlin.com
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#188313): https://lists.openembedded.org/g/openembedded-core/message/188313
> > Mute This Topic: https://lists.openembedded.org/mt/101570279/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Alexandre Belloni]

On 27/09/2023 15:02:00+0200, Julien Stephan wrote:
> Le mer. 27 sept. 2023 � 14:59, Alexander Kanavin
> <alex.kanavin@gmail.com> a �crit :
> >
> > You need to scroll up. The failure is in lines 557 to 609.
> 
> Hmm stange ctrl-f didn't work. I can see it now. Thank you, I'll look at it
> 

By default the log is not fully downloaded, you have to click on the
looking glass before ctrl-f.

> Julien
> >
> > Alex
> >
> >
> > On Wed, 27 Sept 2023 at 14:48, Julien Stephan <jstephan@baylibre.com> wrote:
> > >
> > > Le mer. 27 sept. 2023 � 12:16, Alexandre Belloni
> > > <alexandre.belloni@bootlin.com> a �crit :
> > > >
> > > > Hello,
> > > >
> > > > I had a build were all the tests passed, then I had this failure:
> > > >
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> > > >
> > >
> > > Hi Alexandre,
> > > I only see
> > >
> > >    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
> > >
> > > but I can't see the actual error :( Is it possible to have it somewhere?
> > >
> > > Cheers
> > > Julien
> > >
> > > >
> > > >
> > > > --
> > > > Alexandre Belloni, co-owner and COO, Bootlin
> > > > Embedded Linux and Kernel engineering
> > > > https://bootlin.com
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#188313): https://lists.openembedded.org/g/openembedded-core/message/188313
> > > Mute This Topic: https://lists.openembedded.org/mt/101570279/1686489
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Richard Purdie]

On Wed, 2023-09-27 at 14:47 +0200, Julien Stephan wrote:
> Le mer. 27 sept. 2023 à 12:16, Alexandre Belloni
> <alexandre.belloni@bootlin.com> a écrit :
> > 
> > Hello,
> > 
> > I had a build were all the tests passed, then I had this failure:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> > 
> 
> Hi Alexandre,
> I only see
> 
>    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
> 
> but I can't see the actual error :( Is it possible to have it somewhere?

I am a bit worried this will be hard to reproduce/debug so I had a look
at things as I do really want to get things resolved.

The error log says:

WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64

and I couldn't reproduce those stamps in a local build or on the
builder where it failed (once I got access which took a while).

The "while" is important as I think in the meantime things changed
which effected the results.

On the autobuilder, "bitbake quilt -S none" generated stamp files for
quilt. I did this for "build" and "build-st-3046552", the latter being
the one where the test failed. There were two different do_build stamps
generated. The diff:

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a  ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407 
NOTE: Reconnecting to bitbake server...
Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce
    Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
    Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec

i.e. a couple of dependencies were different.

bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec  ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2

showed no output, i.e. they appear to be the same but different.

My best guess at what is going on here is hash equivalence causing
problems. 

So in summary, I think the best solution will be to disable hash
equivalence for these tests.


Cheers,

Richard

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Julien Stephan]

Le mer. 27 sept. 2023 à 23:17, Richard Purdie
<richard.purdie@linuxfoundation.org> a écrit :
>
> On Wed, 2023-09-27 at 14:47 +0200, Julien Stephan wrote:
> > Le mer. 27 sept. 2023 à 12:16, Alexandre Belloni
> > <alexandre.belloni@bootlin.com> a écrit :
> > >
> > > Hello,
> > >
> > > I had a build were all the tests passed, then I had this failure:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> > >
> >
> > Hi Alexandre,
> > I only see
> >
> >    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
> >
> > but I can't see the actual error :( Is it possible to have it somewhere?
>
> I am a bit worried this will be hard to reproduce/debug so I had a look
> at things as I do really want to get things resolved.
>
> The error log says:
>
> WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64
>
> and I couldn't reproduce those stamps in a local build or on the
> builder where it failed (once I got access which took a while).
>
> The "while" is important as I think in the meantime things changed
> which effected the results.
>
> On the autobuilder, "bitbake quilt -S none" generated stamp files for
> quilt. I did this for "build" and "build-st-3046552", the latter being
> the one where the test failed. There were two different do_build stamps
> generated. The diff:
>
> bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a  ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
> NOTE: Reconnecting to bitbake server...
> Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce
>     Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
>     Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
>
> i.e. a couple of dependencies were different.
>
> bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec  ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2
>
> showed no output, i.e. they appear to be the same but different.
>
> My best guess at what is going on here is hash equivalence causing
> problems.
>
> So in summary, I think the best solution will be to disable hash
> equivalence for these tests.
>
>

Hi Richard,

Thank you a lot for digging into this!
How can I disable the hash equivalence for these tests?

Should I use BB_BASEHASH_IGNORE_VARS for that?

I tried the following:
self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")

But this doesn't work, I get a lot of errors like the following:
ERROR: When reparsing
/home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:do_packagedata,
the basehash value changed from
8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to
9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The
metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake quilt -cdo_packagedata -Snone
ERROR: Then:
ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff



Cheers
Julien

> Cheers,
>

> Richard

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Jose Quaresma]

[-- Attachment #1: Type: text/plain, Size: 4878 bytes --]

Hi Julien,

Julien Stephan <jstephan@baylibre.com> escreveu no dia sábado, 30/09/2023
à(s) 15:13:

> Le mer. 27 sept. 2023 à 23:17, Richard Purdie
> <richard.purdie@linuxfoundation.org> a écrit :
> >
> > On Wed, 2023-09-27 at 14:47 +0200, Julien Stephan wrote:
> > > Le mer. 27 sept. 2023 à 12:16, Alexandre Belloni
> > > <alexandre.belloni@bootlin.com> a écrit :
> > > >
> > > > Hello,
> > > >
> > > > I had a build were all the tests passed, then I had this failure:
> > > >
> > > >
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> > > >
> > >
> > > Hi Alexandre,
> > > I only see
> > >
> > >    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
> > >
> > > but I can't see the actual error :( Is it possible to have it
> somewhere?
> >
> > I am a bit worried this will be hard to reproduce/debug so I had a look
> > at things as I do really want to get things resolved.
> >
> > The error log says:
> >
> > WARNING: The quilt:do_build sig is computed to be
> 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the
> sig is locked to
> 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in
> SIGGEN_LOCKEDSIGS_core2-64
> >
> > and I couldn't reproduce those stamps in a local build or on the
> > builder where it failed (once I got access which took a while).
> >
> > The "while" is important as I think in the meantime things changed
> > which effected the results.
> >
> > On the autobuilder, "bitbake quilt -S none" generated stamp files for
> > quilt. I did this for "build" and "build-st-3046552", the latter being
> > the one where the test failed. There were two different do_build stamps
> > generated. The diff:
> >
> > bitbake-diffsigs
> tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a
> ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
> > NOTE: Reconnecting to bitbake server...
> > Hash for task dependency diffstat:do_package_write_rpm changed from
> 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to
> 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce
> >     Hash for task dependency diffstat:do_package changed from
> f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to
> e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
> >     Unable to find matching sigdata for diffstat:do_package with hash
> e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
> >
> > i.e. a couple of dependencies were different.
> >
> > bitbake-diffsigs
> tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec
> ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2
> >
> > showed no output, i.e. they appear to be the same but different.
> >
> > My best guess at what is going on here is hash equivalence causing
> > problems.
> >
> > So in summary, I think the best solution will be to disable hash
> > equivalence for these tests.
> >
> >
>
> Hi Richard,
>
> Thank you a lot for digging into this!
> How can I disable the hash equivalence for these tests?
>

Can you try with BB_SIGNATURE_HANDLER using the bibake default config

self.write_config('BB_SIGNATURE_HANDLER = "OEBasicHash"')

Jose


> Should I use BB_BASEHASH_IGNORE_VARS for that?
>
> I tried the following:
> self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")
>
> But this doesn't work, I get a lot of errors like the following:
> ERROR: When reparsing
>
> /home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:
> do_packagedata,
> the basehash value changed from
> 8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to
> 9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The
> metadata is not deterministic and this needs to be fixed.
> ERROR: The following commands may help:
> ERROR: $ bitbake quilt -cdo_packagedata -Snone
> ERROR: Then:
> ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff
>
>
>
> Cheers
> Julien
>
> > Cheers,
> >
>
> > Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188459):
> https://lists.openembedded.org/g/openembedded-core/message/188459
> Mute This Topic: https://lists.openembedded.org/mt/101570279/5052612
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> quaresma.jose@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
Best regards,

José Quaresma

[-- Attachment #2: Type: text/html, Size: 6750 bytes --]

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Julien Stephan]

Hi Jose,

I just pushed my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
with the fix you just suggested.

I do not reproduce the issue locally, so I am not sure this fix the problem..
Richard, Alexandre what do you think?  Can you restart a build on autobuidler?

Cheers
Julien

Le lun. 2 oct. 2023 à 12:25, Jose Quaresma <quaresma.jose@gmail.com> a écrit :
>
> Hi Julien,
>
> Julien Stephan <jstephan@baylibre.com> escreveu no dia sábado, 30/09/2023 à(s) 15:13:
>>
>> Le mer. 27 sept. 2023 à 23:17, Richard Purdie
>> <richard.purdie@linuxfoundation.org> a écrit :
>> >
>> > On Wed, 2023-09-27 at 14:47 +0200, Julien Stephan wrote:
>> > > Le mer. 27 sept. 2023 à 12:16, Alexandre Belloni
>> > > <alexandre.belloni@bootlin.com> a écrit :
>> > > >
>> > > > Hello,
>> > > >
>> > > > I had a build were all the tests passed, then I had this failure:
>> > > >
>> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
>> > > >
>> > >
>> > > Hi Alexandre,
>> > > I only see
>> > >
>> > >    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
>> > >
>> > > but I can't see the actual error :( Is it possible to have it somewhere?
>> >
>> > I am a bit worried this will be hard to reproduce/debug so I had a look
>> > at things as I do really want to get things resolved.
>> >
>> > The error log says:
>> >
>> > WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64
>> >
>> > and I couldn't reproduce those stamps in a local build or on the
>> > builder where it failed (once I got access which took a while).
>> >
>> > The "while" is important as I think in the meantime things changed
>> > which effected the results.
>> >
>> > On the autobuilder, "bitbake quilt -S none" generated stamp files for
>> > quilt. I did this for "build" and "build-st-3046552", the latter being
>> > the one where the test failed. There were two different do_build stamps
>> > generated. The diff:
>> >
>> > bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a  ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
>> > NOTE: Reconnecting to bitbake server...
>> > Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce
>> >     Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
>> >     Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
>> >
>> > i.e. a couple of dependencies were different.
>> >
>> > bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec  ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2
>> >
>> > showed no output, i.e. they appear to be the same but different.
>> >
>> > My best guess at what is going on here is hash equivalence causing
>> > problems.
>> >
>> > So in summary, I think the best solution will be to disable hash
>> > equivalence for these tests.
>> >
>> >
>>
>> Hi Richard,
>>
>> Thank you a lot for digging into this!
>> How can I disable the hash equivalence for these tests?
>
>
> Can you try with BB_SIGNATURE_HANDLER using the bibake default config
>
> self.write_config('BB_SIGNATURE_HANDLER = "OEBasicHash"')
>
> Jose
>
>>
>> Should I use BB_BASEHASH_IGNORE_VARS for that?
>>
>> I tried the following:
>> self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")
>>
>> But this doesn't work, I get a lot of errors like the following:
>> ERROR: When reparsing
>> /home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:do_packagedata,
>> the basehash value changed from
>> 8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to
>> 9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The
>> metadata is not deterministic and this needs to be fixed.
>> ERROR: The following commands may help:
>> ERROR: $ bitbake quilt -cdo_packagedata -Snone
>> ERROR: Then:
>> ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff
>>
>>
>>
>> Cheers
>> Julien
>>
>> > Cheers,
>> >
>>
>> > Richard
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#188459): https://lists.openembedded.org/g/openembedded-core/message/188459
>> Mute This Topic: https://lists.openembedded.org/mt/101570279/5052612
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [quaresma.jose@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>
>
> --
> Best regards,
>
> José Quaresma

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Alexandre Belloni]

I started an oe-selftest that passed, I'm going to carry that in my
branch to see whether there are other issues.

On 05/10/2023 10:02:22+0200, Julien Stephan wrote:
> Hi Jose,
> 
> I just pushed my branch on poky-contrib
> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
> with the fix you just suggested.
> 
> I do not reproduce the issue locally, so I am not sure this fix the problem..
> Richard, Alexandre what do you think?  Can you restart a build on autobuidler?
> 
> Cheers
> Julien
> 
> Le lun. 2 oct. 2023 � 12:25, Jose Quaresma <quaresma.jose@gmail.com> a �crit :
> >
> > Hi Julien,
> >
> > Julien Stephan <jstephan@baylibre.com> escreveu no dia s�bado, 30/09/2023 �(s) 15:13:
> >>
> >> Le mer. 27 sept. 2023 � 23:17, Richard Purdie
> >> <richard.purdie@linuxfoundation.org> a �crit :
> >> >
> >> > On Wed, 2023-09-27 at 14:47 +0200, Julien Stephan wrote:
> >> > > Le mer. 27 sept. 2023 � 12:16, Alexandre Belloni
> >> > > <alexandre.belloni@bootlin.com> a �crit :
> >> > > >
> >> > > > Hello,
> >> > > >
> >> > > > I had a build were all the tests passed, then I had this failure:
> >> > > >
> >> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5812/steps/14/logs/stdio
> >> > > >
> >> > >
> >> > > Hi Alexandre,
> >> > > I only see
> >> > >
> >> > >    bblock.BBLock.test_lock_single_recipe_all_tasks: FAILED
> >> > >
> >> > > but I can't see the actual error :( Is it possible to have it somewhere?
> >> >
> >> > I am a bit worried this will be hard to reproduce/debug so I had a look
> >> > at things as I do really want to get things resolved.
> >> >
> >> > The error log says:
> >> >
> >> > WARNING: The quilt:do_build sig is computed to be 255536826e64d835fcc6b5a277e9a3dacaae75318e608fa2df289bdfa23df40e, but the sig is locked to 0f48875f011499948d53edc6abbe9e4efd617d5ef9233055e4b61411cf01cb87 in SIGGEN_LOCKEDSIGS_core2-64
> >> >
> >> > and I couldn't reproduce those stamps in a local build or on the
> >> > builder where it failed (once I got access which took a while).
> >> >
> >> > The "while" is important as I think in the meantime things changed
> >> > which effected the results.
> >> >
> >> > On the autobuilder, "bitbake quilt -S none" generated stamp files for
> >> > quilt. I did this for "build" and "build-st-3046552", the latter being
> >> > the one where the test failed. There were two different do_build stamps
> >> > generated. The diff:
> >> >
> >> > bitbake-diffsigs tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.b502a759bb6bceb6c8ccd76b57db2612501bb66bd9c46ece5f99f8bf16d7ac4a  ../build/tmp/stamps/core2-64-poky-linux/quilt/0.67.do_build.sigdata.7cf16a1f87d5058e134f3f6fc02612515e28784625413c87d3545e93ae0ab407
> >> > NOTE: Reconnecting to bitbake server...
> >> > Hash for task dependency diffstat:do_package_write_rpm changed from 41171f7938a2ddbe244372c1cc4300d91b7dca425eb8b214c19262e4949a7f18 to 2211ac20553e9dddc710970cf1a663412ece5b5a2071e47c5471b3ee01efa2ce
> >> >     Hash for task dependency diffstat:do_package changed from f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec to e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
> >> >     Unable to find matching sigdata for diffstat:do_package with hash e5ff0ed8a81d8022b0981162e9b4e286886d9e281adee81175725072c3d413ec
> >> >
> >> > i.e. a couple of dependencies were different.
> >> >
> >> > bitbake-diffsigs tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.f3fe486831eede23d2ebbc1efe2e2510f49f5df6d5d8a8ceb7aa2ddc6e11b1ec  ../build/tmp/stamps/core2-64-poky-linux/diffstat/1.65.do_package.sigdata.af8ebe96c5cf424e1094c1599400a56096302e16d0f691daaeec380b963f37b2
> >> >
> >> > showed no output, i.e. they appear to be the same but different.
> >> >
> >> > My best guess at what is going on here is hash equivalence causing
> >> > problems.
> >> >
> >> > So in summary, I think the best solution will be to disable hash
> >> > equivalence for these tests.
> >> >
> >> >
> >>
> >> Hi Richard,
> >>
> >> Thank you a lot for digging into this!
> >> How can I disable the hash equivalence for these tests?
> >
> >
> > Can you try with BB_SIGNATURE_HANDLER using the bibake default config
> >
> > self.write_config('BB_SIGNATURE_HANDLER = "OEBasicHash"')
> >
> > Jose
> >
> >>
> >> Should I use BB_BASEHASH_IGNORE_VARS for that?
> >>
> >> I tried the following:
> >> self.write_config("BB_BASEHASH_IGNORE_VARS += \"TOPDIR\"")
> >>
> >> But this doesn't work, I get a lot of errors like the following:
> >> ERROR: When reparsing
> >> /home/jstephan/upstream/yocto/poky/meta/recipes-devtools/quilt/quilt_0.67.bb:do_packagedata,
> >> the basehash value changed from
> >> 8b6adb69d1f05eddb3e86143bf9a51060f4fac153742964bb4a3d53bee93d377 to
> >> 9dee5dada20d7f8ccfc89ed3114c428af8c96d665234fead22c25fb118f9f62b. The
> >> metadata is not deterministic and this needs to be fixed.
> >> ERROR: The following commands may help:
> >> ERROR: $ bitbake quilt -cdo_packagedata -Snone
> >> ERROR: Then:
> >> ERROR: $ bitbake quilt -cdo_packagedata -Sprintdiff
> >>
> >>
> >>
> >> Cheers
> >> Julien
> >>
> >> > Cheers,
> >> >
> >>
> >> > Richard
> >>
> >> 
> >>
> >
> >
> > --
> > Best regards,
> >
> > Jos� Quaresma

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#188705): https://lists.openembedded.org/g/openembedded-core/message/188705
> Mute This Topic: https://lists.openembedded.org/mt/101570279/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Richard Purdie]

On Fri, 2023-10-06 at 12:56 +0200, Alexandre Belloni wrote:
> I started an oe-selftest that passed, I'm going to carry that in my
> branch to see whether there are other issues.

I've had a couple of runs in my branch and not seen any issue from that
as yet FWIW.

I am leaning towards merging this at this point.

Cheers,

Richard

Re: [OE-core] [PATCH v5 0/5] Add bblock helper scripts

[Julien Stephan]

Le ven. 6 oct. 2023 à 12:59, Richard Purdie
<richard.purdie@linuxfoundation.org> a écrit :
>
> On Fri, 2023-10-06 at 12:56 +0200, Alexandre Belloni wrote:
> > I started an oe-selftest that passed, I'm going to carry that in my
> > branch to see whether there are other issues.
>
> I've had a couple of runs in my branch and not seen any issue from that
> as yet FWIW.
>
> I am leaning towards merging this at this point.
>
> Cheers,
>
> Richard

Hi Richard and Alexandre,

That is very good news :)

Cheers

Julien