[PATCH v3 0/3] Add bblock helper scripts

[PATCH v3 0/3] Add bblock helper scripts

[Julien Stephan]

Hi all,

This is v3 from bblock script.

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the
  following advantages:
    * findSigInfo needs the task to be already built to get the siginfo
      file, meaning we cannot lock a recipe on a fresh build
    * we can now generate the signatures for all available task of a given
      recipe
* Check if a given task is already locked. If so, don't duplicate
  entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some
corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you
  switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock
  only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying
  to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before
  locking. Need to find a way to generate it if missing

V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (3):
  bitbake.conf: include bblock.conf
  bitbake: cooker: add a new function to retrieve task signatures
  scripts/bblock: add a script to lock/unlock recipes

 bitbake/lib/bb/command.py |   6 ++
 bitbake/lib/bb/cooker.py  |  16 ++++
 bitbake/lib/bb/event.py   |   8 ++
 meta/conf/bitbake.conf    |   1 +
 scripts/bblock            | 182 ++++++++++++++++++++++++++++++++++++++
 5 files changed, 213 insertions(+)
 create mode 100755 scripts/bblock

--
2.41.0

[PATCH v3 1/3] bitbake.conf: include bblock.conf

[Julien Stephan]

include conf/bblock.conf. This file is generated by the bblock tool. It
locks some package tasks by fixing their signatures. See bblock -h for
more details

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9625a6fef4c..a4b2b4b0380 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -831,6 +831,7 @@ include conf/distro/defaultsetup.conf
 include conf/documentation.conf
 include conf/licenses.conf
 require conf/sanity.conf
+include conf/bblock.conf
 
 ##################################################################
 # Weak variables (usually to retain backwards compatibility)
-- 
2.41.0

[PATCH v3 2/3] bitbake: cooker: add a new function to retrieve task signatures

[Julien Stephan]

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the
signatures using tinfoil

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)

diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index a355f56c60c..12202779ac0 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 11c9fa2c40d..687cdde5e6d 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+
     def buildTargets(self, targets, task):
         """
         Attempt to build the targets specified
diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 0d0e0a68aac..f8acacd80d1 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
         Event.__init__(self)
         self.result = result
 
+class GetTaskSignatureResult(Event):
+    """
+    Event to return results from GetTaskSignatures command
+    """
+    def __init__(self, sig):
+        Event.__init__(self)
+        self.sig = sig
+
 class ParseError(Event):
     """
     Event to indicate parse failed
-- 
2.41.0

[PATCH v3 3/3] scripts/bblock: add a script to lock/unlock recipes

[Julien Stephan]

bblock script allows to lock/unlock recipes to latest task signatures.
The idea is to prevent some recipes to be rebuilt during development.
For example when working on rust recipe, one may not want rust-native to be
rebuilt.

This tool can be used, with proper environment set up, using the following
command:

bblock <recipe_name>

if a <recipe_name>'s task signature change, this task will not be built again and
sstate cache will be used.

[YOCTO #13425]

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100755 scripts/bblock

diff --git a/scripts/bblock b/scripts/bblock
new file mode 100755
index 00000000000..a9c5583127e
--- /dev/null
+++ b/scripts/bblock
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def getTaskSignatures(tinfoil, pn, tasks):
+    tinfoil.set_event_mask(
+        [
+            "bb.event.GetTaskSignatureResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("getTaskSignatures", pn, tasks)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.GetTaskSignatureResult):
+                    sig = event.sig
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from getTaskSignatures command")
+        sys.exit(2)
+    return sig
+
+
+def parseRecipe(tinfoil, recipe):
+    try:
+        tinfoil.parse_recipes()
+        d = tinfoil.parse_recipe(recipe)
+    except Exception:
+        logger.error("Failed to get recipe info for: %s" % recipe)
+        sys.exit(1)
+    return d
+
+
+def bblockDump(lockfile):
+    try:
+        with open(lockfile, "r") as lockfile:
+            for line in lockfile:
+                print(line.strip())
+    except IOError:
+        return 1
+    return 0
+
+
+def bblockReset(lockfile, pns, package_archs, tasks):
+    if not pns:
+        logger.info("Unlocking all recipes")
+        try:
+            os.remove(lockfile)
+        except FileNotFoundError:
+            pass
+    else:
+        logger.info("Unlocking {pns}".format(pns=pns))
+        tmp_lockfile = lockfile + ".tmp"
+        with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+            for line in infile:
+                if not (
+                    any(element in line for element in pns)
+                    and any(element in line for element in package_archs.split())
+                ):
+                    outfile.write(line)
+                else:
+                    if tasks and not any(element in line for element in tasks):
+                        outfile.write(line)
+        os.remove(lockfile)
+        os.rename(tmp_lockfile, lockfile)
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+    parser.add_argument(
+        "-t",
+        "--tasks",
+        help="Comma separated list of tasks",
+        type=lambda s: [task for task in s.split(",")],
+    )
+    parser.add_argument(
+        "-r",
+        "--reset",
+        action="store_true",
+        help="Unlock pn recipes, or all recipes if pn is empty",
+    )
+    parser.add_argument(
+        "-d",
+        "--dump",
+        action="store_true",
+        help="Dump generated bblock.conf file",
+    )
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+
+        package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+        if global_args.dump:
+            bblockDump(lockfile)
+            return 0
+
+        if global_args.reset:
+            bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks)
+            return 0
+
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+                s += "\n"
+
+            for pn in global_args.pn:
+                d = parseRecipe(tinfoil, pn)
+                package_arch = d.getVar("PACKAGE_ARCH")
+                siggen_locked_sigs_package_arch = d.getVar(
+                    "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch)
+                )
+                sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks)
+                for sig in sigs:
+                    new_entry = "{pn}:{taskname}:{sig}".format(
+                        pn=sig[0], taskname=sig[1], sig=sig[2]
+                    )
+                    if (
+                        siggen_locked_sigs_package_arch
+                        and not new_entry in siggen_locked_sigs_package_arch
+                    ) or not siggen_locked_sigs_package_arch:
+                        s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format(
+                            package_arch=package_arch, new_entry=new_entry
+                        )
+            lockfile.write(s)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.41.0

Re: [OE-core] [PATCH v3 0/3] Add bblock helper scripts

[Alexandre Belloni]

Hello,

On 27/07/2023 15:56:09+0200, Julien Stephan wrote:
> Hi all,
> 
> This is v3 from bblock script.
> 

This doesn't apply cleanly on master, can you rebase?

Thanks!

> Improvement from v2:
> * Add a function in bb.cooker to compute task signatures
> * Replace the findSigInfo function by the new created one. This has the
>   following advantages:
>     * findSigInfo needs the task to be already built to get the siginfo
>       file, meaning we cannot lock a recipe on a fresh build
>     * we can now generate the signatures for all available task of a given
>       recipe
> * Check if a given task is already locked. If so, don't duplicate
>   entry in bblock.conf
> 
> Limitations:
> * Needs to taint tasks that are locked, to display a warning
> * I may be still missing some checks on user input
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
> 
> I did some tests using qemux86-64 and qemuarm but I may be missing some
> corner cases.
> 
> Improvement from V1:
> * Signatures are now package architecture specific meaning that if you
>   switch MACHINE, the lock sig will not be taken into account
> * I added the -r option to unlock recipes
> * I added a -d option to display the current bblock.conf
> * Added an include directive for conf/bblock.conf inside bitbake.conf
> * Added -t option to specify the tasks to lock/unlock
> 
> Limitations:
> * I may be still missing some checks on user input
> * I need to find a way to get the list of tasks ( by default still lock
>   only the do_compile for now, unless -t is specified)
> * Do not check if a particular recipe/task is already locked when trying
>   to add lock. So entries may appear multiple times
> * We still need the signature of the tasks to be already computed before
>   locking. Need to find a way to generate it if missing
> 
> V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> V1: https://lists.openembedded.org/g/openembedded-core/message/184584
> 
> My branch is available here [1]
> 
> Cheers
> Julien
> 
> [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> 
> Julien Stephan (3):
>   bitbake.conf: include bblock.conf
>   bitbake: cooker: add a new function to retrieve task signatures
>   scripts/bblock: add a script to lock/unlock recipes
> 
>  bitbake/lib/bb/command.py |   6 ++
>  bitbake/lib/bb/cooker.py  |  16 ++++
>  bitbake/lib/bb/event.py   |   8 ++
>  meta/conf/bitbake.conf    |   1 +
>  scripts/bblock            | 182 ++++++++++++++++++++++++++++++++++++++
>  5 files changed, 213 insertions(+)
>  create mode 100755 scripts/bblock
> 
> --
> 2.41.0

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184932): https://lists.openembedded.org/g/openembedded-core/message/184932
> Mute This Topic: https://lists.openembedded.org/mt/100390731/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v3 0/3] Add bblock helper scripts

[Julien Stephan]

Le dim. 30 juil. 2023 à 00:44, Alexandre Belloni
<alexandre.belloni@bootlin.com> a écrit :
>
> Hello,
>
> On 27/07/2023 15:56:09+0200, Julien Stephan wrote:
> > Hi all,
> >
> > This is v3 from bblock script.
> >
>
> This doesn't apply cleanly on master, can you rebase?
>
> Thanks!

Hi Alexandre,

done, just pushed v4 here:
https://lists.openembedded.org/g/openembedded-core/message/185403

Best
Julien
>
> > Improvement from v2:
> > * Add a function in bb.cooker to compute task signatures
> > * Replace the findSigInfo function by the new created one. This has the
> >   following advantages:
> >     * findSigInfo needs the task to be already built to get the siginfo
> >       file, meaning we cannot lock a recipe on a fresh build
> >     * we can now generate the signatures for all available task of a given
> >       recipe
> > * Check if a given task is already locked. If so, don't duplicate
> >   entry in bblock.conf
> >
> > Limitations:
> > * Needs to taint tasks that are locked, to display a warning
> > * I may be still missing some checks on user input
> > * Silently does nothing if given task doesn't exist
> > * Silently does nothing when resetting a recipe that doesn't exist
> >
> > I did some tests using qemux86-64 and qemuarm but I may be missing some
> > corner cases.
> >
> > Improvement from V1:
> > * Signatures are now package architecture specific meaning that if you
> >   switch MACHINE, the lock sig will not be taken into account
> > * I added the -r option to unlock recipes
> > * I added a -d option to display the current bblock.conf
> > * Added an include directive for conf/bblock.conf inside bitbake.conf
> > * Added -t option to specify the tasks to lock/unlock
> >
> > Limitations:
> > * I may be still missing some checks on user input
> > * I need to find a way to get the list of tasks ( by default still lock
> >   only the do_compile for now, unless -t is specified)
> > * Do not check if a particular recipe/task is already locked when trying
> >   to add lock. So entries may appear multiple times
> > * We still need the signature of the tasks to be already computed before
> >   locking. Need to find a way to generate it if missing
> >
> > V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> > V1: https://lists.openembedded.org/g/openembedded-core/message/184584
> >
> > My branch is available here [1]
> >
> > Cheers
> > Julien
> >
> > [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> >
> > Julien Stephan (3):
> >   bitbake.conf: include bblock.conf
> >   bitbake: cooker: add a new function to retrieve task signatures
> >   scripts/bblock: add a script to lock/unlock recipes
> >
> >  bitbake/lib/bb/command.py |   6 ++
> >  bitbake/lib/bb/cooker.py  |  16 ++++
> >  bitbake/lib/bb/event.py   |   8 ++
> >  meta/conf/bitbake.conf    |   1 +
> >  scripts/bblock            | 182 ++++++++++++++++++++++++++++++++++++++
> >  5 files changed, 213 insertions(+)
> >  create mode 100755 scripts/bblock
> >
> > --
> > 2.41.0
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#184932): https://lists.openembedded.org/g/openembedded-core/message/184932
> > Mute This Topic: https://lists.openembedded.org/mt/100390731/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com