[RFC 0/1] Add bblock helper script

[RFC 0/1] Add bblock helper script

[Julien Stephan]

Hi all,

I am currently working on bug #13425 and I would like to post my wip
script as an RFC to get feedback on it, before going further in the
development.

The script `script/bblock` can be used with the following command:

bblock [list of recipes to lock]

Here is a summary of what is currently implemented:
* can lock several recipes at once
* on first execution bblock will append `require bblock.inc` in
  `build/conf/auto.conf` (creates `auto.conf` if it doesn't exist)
* on first execution creates the file `build/conf/bblock.inc` with a
  dedicated header and:
	 - adds SIGGEN_LOCKEDSIGS_TYPES += "t-bblock"
	 - adds SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" to display warning but
	   do not stop the build
* loops over all recipes to lock and adds entry with latest "do_compile"
  signature in `conf/bblock.inc`, such as: SIGGEN_LOCKEDSIGS_t-bblock += "bc:do_compile:e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f"

Limitations:
* only gets do_compile signature for now as a POC
* `bblock reset [list of recipes]` not yet implemented
* no check if a recipe is already locked

Steps to test the script:
bitbake bc --> generate a signature for bc's tasks
bblock bc
# modify do_compile in bc recipe to force signature change
bitbake bc --> throws the following warning: `WARNING: The bc:do_compile sig is computed to be 680bd6c291bf88e379e0c405a773cf5f81851e1a52570398cefd0196000ac1ef, but the sig is locked to e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f in SIGGEN_LOCKEDSIGS_t-bblock`

I also have a point I would like to discuss: I only get signature for
do_compile for the POC but wondering what should I do here:
* have a set of predefined task to lock, in that case how to choose the
  tasks?
* compute the list of all available tasks for a given recipe and get
  signature for all? Is there a way to get such list without using
  `infoil.build_targets(pn, "listtasks", handle_events=True)` as this
  will start bitbake, takes some time and requires some processing of
  the output
* add an option for the user to specify the task he wants to lock? (this
  may be useful to add anyway)

My branch is available here [1]

Am I going into the right direction?

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock

Julien Stephan (1):
  scripts/bblock: add a script to lock/unlock recipes

 scripts/bblock | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100755 scripts/bblock

--
2.41.0

[RFC 1/1] scripts/bblock: add a script to lock/unlock recipes

[Julien Stephan]

bblock script allows to lock recipes to latest signatures. The idea is
to prevent some recipes to be rebuilt during development. For example
when working on rust recipe, one may not want rust-native to be
rebuilt.

This tool can be used, with proper environment set up, using the following
command:

bblock <recipe_name>

if <recipe_name>'s task signatures change it will not be built again and
sstate cache will be used.

[YOCTO #13425]

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/bblock | 110 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 110 insertions(+)
 create mode 100755 scripts/bblock

diff --git a/scripts/bblock b/scripts/bblock
new file mode 100755
index 00000000000..d463ade6212
--- /dev/null
+++ b/scripts/bblock
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2020 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def find_siginfo(tinfoil, pn, taskname, sigs=None):
+    result = None
+    tinfoil.set_event_mask(
+        [
+            "bb.event.FindSigInfoResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("findSigInfo", pn, taskname, sigs)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.FindSigInfoResult):
+                    result = event.result
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from findSigInfo command")
+        sys.exit(2)
+    return result
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        autoconffile = "{builddir}/conf/auto.conf".format(builddir=builddir)
+        lockfile = "{builddir}/conf/bblock.inc".format(builddir=builddir)
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "t-bblock"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"'
+                with open(autoconffile, "a") as autoconffile:
+                    autoconffile.write("require bblock.inc")
+            for pn in global_args.pn:
+                taskname = "do_compile"
+                filedates = find_siginfo(tinfoil, pn, taskname)
+                latestfiles = sorted(
+                    filedates.keys(), key=lambda f: filedates[f], reverse=True
+                )
+                if not latestfiles:
+                    logger.error(
+                        "No sigdata files found matching {pn} {taskname}".format(
+                            pn=pn, taskname=taskname
+                        )
+                    )
+                    sys.exit(1)
+                sig = latestfiles[0].split("sigdata.")[1]
+                s += "\n"
+                s += 'SIGGEN_LOCKEDSIGS_t-bblock += "{pn}:{taskname}:{sig}"\n'.format(
+                    pn=pn, taskname=taskname, sig=sig
+                )
+            lockfile.write(s)
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.41.0

Re: [OE-core] [RFC 0/1] Add bblock helper script

[Richard Purdie]

Hi,

Thanks for looking at this, I think it could become a really useful and
well used extension to bitbake!

On Wed, 2023-07-19 at 16:27 +0200, Julien Stephan wrote:
> I am currently working on bug #13425 and I would like to post my wip
> script as an RFC to get feedback on it, before going further in the
> development.
> 
> The script `script/bblock` can be used with the following command:
> 
> bblock [list of recipes to lock]
> 
> Here is a summary of what is currently implemented:
> * can lock several recipes at once
> * on first execution bblock will append `require bblock.inc` in
>   `build/conf/auto.conf` (creates `auto.conf` if it doesn't exist)
> * on first execution creates the file `build/conf/bblock.inc` with a
>   dedicated header and:
> 	 - adds SIGGEN_LOCKEDSIGS_TYPES += "t-bblock"
> 	 - adds SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" to display warning but
> 	   do not stop the build
> * loops over all recipes to lock and adds entry with latest "do_compile"
>   signature in `conf/bblock.inc`, such as: SIGGEN_LOCKEDSIGS_t-bblock += "bc:do_compile:e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f"
> 
> Limitations:
> * only gets do_compile signature for now as a POC
> * `bblock reset [list of recipes]` not yet implemented
> * no check if a recipe is already locked
> 
> Steps to test the script:
> bitbake bc --> generate a signature for bc's tasks
> bblock bc
> # modify do_compile in bc recipe to force signature change
> bitbake bc --> throws the following warning: `WARNING: The bc:do_compile sig is computed to be 680bd6c291bf88e379e0c405a773cf5f81851e1a52570398cefd0196000ac1ef, but the sig is locked to e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f in SIGGEN_LOCKEDSIGS_t-bblock`

Ideally we should just be able to run bblock bc without any previous
commands and it would lock the file.

We may want to note that some recipes are locked somehow too so the
user realises this if they leave a build and come back to it later,
forgetting what they did (a bit like the -f option leaves a warning on
later builds to remind the user).

To simplify things, I think it would be reasonable to add a bblock.inc
file unconditionally in our core configuration so that it is always
included if present, just to make the tool easier since I think this
will be a common use.

Also, you're going to need to think about package architectures.
Imagine I change MACHINE and then try "bitbake bc", it probably
shouldn't be locked any more, it certainly shouldn't try and match the
previous locked values as in most cases they will be different.

This is why if you do a "bitbake core-image-minimal -S none" and look
at the loacked-sigs.inc file, you'll see sections like:

SIGGEN_LOCKEDSIGS_t-allarch 
SIGGEN_LOCKEDSIGS_t-x86-64 
SIGGEN_LOCKEDSIGS_t-x86-64-v3
SIGGEN_LOCKEDSIGS_t-qemux86-64

and a list of types:

SIGGEN_LOCKEDSIGS_TYPES:qemux86-64 = "t-all t-allarch t-qemux86-64 t-x86-64 t-x86-64-v3"

This means the locks only apply to specific package architectures,
which allows you to change MACHINE.
> 
> I also have a point I would like to discuss: I only get signature for
> do_compile for the POC but wondering what should I do here:
> * have a set of predefined task to lock, in that case how to choose the
>   tasks?

I think by default bblock would lock all tasks for a recipe? It might
take a task option to allow a specific task to be specified?

> * compute the list of all available tasks for a given recipe and get
>   signature for all? Is there a way to get such list without using
>   `infoil.build_targets(pn, "listtasks", handle_events=True)` as this
>   will start bitbake, takes some time and requires some processing of
>   the output

We may need to add some API to bitbake to get this information. The
task hashes do need a full task graph to compute so it isn't a simple
operation with a cold cache.

> * add an option for the user to specify the task he wants to lock? (this
>   may be useful to add anyway)

Yes, I think that would make sense.

> My branch is available here [1]
> 
> Am I going into the right direction?

I think you've made a good start, yes!

Cheers,

Richard