[RFC v2 0/2] Add bblock helper script

[RFC v2 0/2] Add bblock helper script

[Julien Stephan]

Hi all,

This is the v2 of bblock script.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you
  switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock
  only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying
  to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before
  locking. Need to find a way to generate it if missing

I did some tests using qemux86-64 and qemuarm but I may be missing some
corner cases.

I force pushed my branch on poky-contrib [1]

Any feedback are welcome!

Cheers
Julien

V1:
I am currently working on bug #13425 and I would like to post my wip
script as an RFC to get feedback on it, before going further in the
development.

The script `script/bblock` can be used with the following command:

bblock [list of recipes to lock]

Here is a summary of what is currently implemented:
* can lock several recipes at once
* on first execution bblock will append `require bblock.inc` in
  `build/conf/auto.conf` (creates `auto.conf` if it doesn't exist)
* on first execution creates the file `build/conf/bblock.inc` with a
  dedicated header and:
         - adds SIGGEN_LOCKEDSIGS_TYPES += "t-bblock"
         - adds SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" to display warning but
           do not stop the build
* loops over all recipes to lock and adds entry with latest "do_compile"
  signature in `conf/bblock.inc`, such as: SIGGEN_LOCKEDSIGS_t-bblock += "bc:do_compile:e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f"

Limitations:
* only gets do_compile signature for now as a POC
* `bblock reset [list of recipes]` not yet implemented
* no check if a recipe is already locked

Steps to test the script:
bitbake bc --> generate a signature for bc's tasks
bblock bc
# modify do_compile in bc recipe to force signature change
bitbake bc --> throws the following warning: `WARNING: The bc:do_compile sig is computed to be 680bd6c291bf88e379e0c405a773cf5f81851e1a52570398cefd0196000ac1ef, but the sig is locked to e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f in SIGGEN_LOCKEDSIGS_t-bblock`

I also have a point I would like to discuss: I only get signature for
do_compile for the POC but wondering what should I do here:
* have a set of predefined task to lock, in that case how to choose the
  tasks?
* compute the list of all available tasks for a given recipe and get
  signature for all? Is there a way to get such list without using
  `infoil.build_targets(pn, "listtasks", handle_events=True)` as this
  will start bitbake, takes some time and requires some processing of
  the output
* add an option for the user to specify the task he wants to lock? (this
  may be useful to add anyway)

My branch is available here [1]

Am I going into the right direction?

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (2):
  bitbake.conf: include bblock.conf
  scripts/bblock: add a script to lock/unlock recipes

 meta/conf/bitbake.conf |   1 +
 scripts/bblock         | 184 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 185 insertions(+)
 create mode 100755 scripts/bblock

--
2.41.0

[RFC v2 1/2] bitbake.conf: include bblock.conf

[Julien Stephan]

include conf/bblock.conf. This file is generated by the bblock tool. It
locks some package tasks by fixing their signature. See bblock -h for
more details

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9625a6fef4c..a4b2b4b0380 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -831,6 +831,7 @@ include conf/distro/defaultsetup.conf
 include conf/documentation.conf
 include conf/licenses.conf
 require conf/sanity.conf
+include conf/bblock.conf
 
 ##################################################################
 # Weak variables (usually to retain backwards compatibility)
-- 
2.41.0

[RFC v2 2/2] scripts/bblock: add a script to lock/unlock recipes

[Julien Stephan]

bblock script allows to lock/unlock recipes to latest signatures. The idea is
to prevent some recipes to be rebuilt during development. For example
when working on rust recipe, one may not want rust-native to be
rebuilt.

This tool can be used, with proper environment set up, using the following
command:

bblock <recipe_name>

if <recipe_name>'s task signatures change it will not be built again and
sstate cache will be used.

[YOCTO #13425]

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/bblock | 184 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 184 insertions(+)
 create mode 100755 scripts/bblock

diff --git a/scripts/bblock b/scripts/bblock
new file mode 100755
index 00000000000..5a269b33806
--- /dev/null
+++ b/scripts/bblock
@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2020 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def find_siginfo(tinfoil, pn, taskname, sigs=None):
+    result = None
+    tinfoil.set_event_mask(
+        [
+            "bb.event.FindSigInfoResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("findSigInfo", pn, taskname, sigs)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.FindSigInfoResult):
+                    result = event.result
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from findSigInfo command")
+        sys.exit(2)
+    return result
+
+
+def parse_recipe(tinfoil, recipe):
+    try:
+        tinfoil.parse_recipes()
+        d = tinfoil.parse_recipe(recipe)
+    except Exception:
+        logger.error("Failed to get recipe info for: %s" % recipe)
+        sys.exit(1)
+    return d
+
+
+def dump(lockfile):
+    with open(lockfile, "r") as lockfile:
+        for line in lockfile:
+            print(line.strip())
+    return 0
+
+
+def reset(lockfile, pns, package_archs, tasks):
+    if not pns:
+        logger.info("Unlocking all recipes")
+        os.remove(lockfile)
+    else:
+        logger.info("Unlocking {pns}".format(pns=pns))
+        tmp_lockfile = lockfile + ".tmp"
+        with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+            for line in infile:
+                if not (
+                    any(element in line for element in pns)
+                    and any(element in line for element in package_archs.split())
+                ):
+                    outfile.write(line)
+                else:
+                    if tasks and not any(element in line for element in tasks):
+                        outfile.write(line)
+        os.remove(lockfile)
+        os.rename(tmp_lockfile, lockfile)
+    sys.exit(0)
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+    parser.add_argument(
+        "-t",
+        "--tasks",
+        help="Comma separated list of tasks",
+        type=lambda s: [task for task in s.split(",")],
+    )
+    parser.add_argument(
+        "-r",
+        "--reset",
+        action="store_true",
+        help="Unlock pn recipes, or all recipes if pn is empty",
+    )
+
+    parser.add_argument(
+        "-d",
+        "--dump",
+        action="store_true",
+        help="Dump generated bblock.conf file",
+    )
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+
+        package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+        if global_args.dump:
+            dump(lockfile)
+
+        tasks = global_args.tasks
+
+        if global_args.reset:
+            reset(lockfile, global_args.pn, package_archs, tasks)
+
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+                s += "\n"
+
+            for pn in global_args.pn:
+                package_arch = parse_recipe(tinfoil, pn).getVar("PACKAGE_ARCH")
+
+                if not tasks:
+                    # TODO: get a list of all available tasks
+                    tasks = ["do_compile"]
+                for taskname in tasks:
+                    filedates = find_siginfo(tinfoil, pn, taskname)
+                    latestfiles = sorted(
+                        filedates.keys(), key=lambda f: filedates[f], reverse=True
+                    )
+                    if not latestfiles:
+                        logger.error(
+                            "No sigdata files found matching {pn} {taskname}".format(
+                                pn=pn, taskname=taskname
+                            )
+                        )
+                        return 1
+                    sig = latestfiles[0].split("sigdata.")[1]
+
+                    s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{pn}:{taskname}:{sig}"\n'.format(
+                        package_arch=package_arch, pn=pn, taskname=taskname, sig=sig
+                    )
+            lockfile.write(s)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.41.0

Re: [OE-core] [RFC v2 0/2] Add bblock helper script

[Alexander Kanavin]

Thanks for working on this. Should we think about how this could be tested?

Alex

On Fri, 21 Jul 2023 at 16:25, Julien Stephan <jstephan@baylibre.com> wrote:
>
> Hi all,
>
> This is the v2 of bblock script.
>
> Improvement from V1:
> * Signatures are now package architecture specific meaning that if you
>   switch MACHINE, the lock sig will not be taken into account
> * I added the -r option to unlock recipes
> * I added a -d option to display the current bblock.conf
> * Added an include directive for conf/bblock.conf inside bitbake.conf
> * Added -t option to specify the tasks to lock/unlock
>
> Limitations:
> * I may be still missing some checks on user input
> * I need to find a way to get the list of tasks ( by default still lock
>   only the do_compile for now, unless -t is specified)
> * Do not check if a particular recipe/task is already locked when trying
>   to add lock. So entries may appear multiple times
> * We still need the signature of the tasks to be already computed before
>   locking. Need to find a way to generate it if missing
>
> I did some tests using qemux86-64 and qemuarm but I may be missing some
> corner cases.
>
> I force pushed my branch on poky-contrib [1]
>
> Any feedback are welcome!
>
> Cheers
> Julien
>
> V1:
> I am currently working on bug #13425 and I would like to post my wip
> script as an RFC to get feedback on it, before going further in the
> development.
>
> The script `script/bblock` can be used with the following command:
>
> bblock [list of recipes to lock]
>
> Here is a summary of what is currently implemented:
> * can lock several recipes at once
> * on first execution bblock will append `require bblock.inc` in
>   `build/conf/auto.conf` (creates `auto.conf` if it doesn't exist)
> * on first execution creates the file `build/conf/bblock.inc` with a
>   dedicated header and:
>          - adds SIGGEN_LOCKEDSIGS_TYPES += "t-bblock"
>          - adds SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn" to display warning but
>            do not stop the build
> * loops over all recipes to lock and adds entry with latest "do_compile"
>   signature in `conf/bblock.inc`, such as: SIGGEN_LOCKEDSIGS_t-bblock += "bc:do_compile:e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f"
>
> Limitations:
> * only gets do_compile signature for now as a POC
> * `bblock reset [list of recipes]` not yet implemented
> * no check if a recipe is already locked
>
> Steps to test the script:
> bitbake bc --> generate a signature for bc's tasks
> bblock bc
> # modify do_compile in bc recipe to force signature change
> bitbake bc --> throws the following warning: `WARNING: The bc:do_compile sig is computed to be 680bd6c291bf88e379e0c405a773cf5f81851e1a52570398cefd0196000ac1ef, but the sig is locked to e233cd793137a92dd575a417a2877e324ce526c4dc4a7db652abb9512f406f1f in SIGGEN_LOCKEDSIGS_t-bblock`
>
> I also have a point I would like to discuss: I only get signature for
> do_compile for the POC but wondering what should I do here:
> * have a set of predefined task to lock, in that case how to choose the
>   tasks?
> * compute the list of all available tasks for a given recipe and get
>   signature for all? Is there a way to get such list without using
>   `infoil.build_targets(pn, "listtasks", handle_events=True)` as this
>   will start bitbake, takes some time and requires some processing of
>   the output
> * add an option for the user to specify the task he wants to lock? (this
>   may be useful to add anyway)
>
> My branch is available here [1]
>
> Am I going into the right direction?
>
> Cheers
> Julien
>
> [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> Julien Stephan (2):
>   bitbake.conf: include bblock.conf
>   scripts/bblock: add a script to lock/unlock recipes
>
>  meta/conf/bitbake.conf |   1 +
>  scripts/bblock         | 184 +++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 185 insertions(+)
>  create mode 100755 scripts/bblock
>
> --
> 2.41.0
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184697): https://lists.openembedded.org/g/openembedded-core/message/184697
> Mute This Topic: https://lists.openembedded.org/mt/100277504/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [RFC v2 0/2] Add bblock helper script

[Julien Stephan]

On Sun, Jul 23, 2023 at 02:11:14PM +0200, Alexander Kanavin wrote:
> Thanks for working on this. Should we think about how this could be tested?
>
> Alex
>

Hi Alex,

Do you mean we should consider addind self test for bblock? or you want
to discuss about acceptance criteria for bblock?

Cheers
Julien

Re: [OE-core] [RFC v2 0/2] Add bblock helper script

[Alexander Kanavin]

On Mon, 24 Jul 2023 at 10:25, Julien Stephan <jstephan@baylibre.com> wrote:
> > Thanks for working on this. Should we think about how this could be tested?
> Do you mean we should consider addind self test for bblock? or you want
> to discuss about acceptance criteria for bblock?

Hello,
I mean mostly about what should be tested. How can we verify that the
script does what it's supposed to, in an automated way, so it won't
silently break as new changes to bitbake/oe-core are added? What
should such a test contain?
The specific form of tests (whether it's selftest, or maybe a part of
SDK testing, or something else) is secondary.

Alex

Re: [OE-core] [RFC v2 0/2] Add bblock helper script

[Julien Stephan]

On Mon, Jul 24, 2023 at 10:36:28AM +0200, Alexander Kanavin wrote:
> On Mon, 24 Jul 2023 at 10:25, Julien Stephan <jstephan@baylibre.com> wrote:
> > > Thanks for working on this. Should we think about how this could be tested?
> > Do you mean we should consider addind self test for bblock? or you want
> > to discuss about acceptance criteria for bblock?
>
> Hello,
> I mean mostly about what should be tested. How can we verify that the
> script does what it's supposed to, in an automated way, so it won't
> silently break as new changes to bitbake/oe-core are added? What
> should such a test contain?
> The specific form of tests (whether it's selftest, or maybe a part of
> SDK testing, or something else) is secondary.
>
> Alex

Hi Alex,

I just send the v3 with a lot of improvements!

Here is what I do during my dev:
- lock a recipe's task for a given arch (qemux86-64 vs qemuarm)
- modify the given task
- check that for the selected arch, task is locked (but not for other arch)

Maybe we need to be more formal with that?

I would be happy to provide a test script if needed.

Cheers
Julien

Re: [OE-core] [RFC v2 0/2] Add bblock helper script

[Alexander Kanavin]

Sure; if you can write a script and show it here, we can figure out
where to place the test officially.

Alex

On Thu, 27 Jul 2023 at 16:15, Julien Stephan <jstephan@baylibre.com> wrote:
>
> On Mon, Jul 24, 2023 at 10:36:28AM +0200, Alexander Kanavin wrote:
> > On Mon, 24 Jul 2023 at 10:25, Julien Stephan <jstephan@baylibre.com> wrote:
> > > > Thanks for working on this. Should we think about how this could be tested?
> > > Do you mean we should consider addind self test for bblock? or you want
> > > to discuss about acceptance criteria for bblock?
> >
> > Hello,
> > I mean mostly about what should be tested. How can we verify that the
> > script does what it's supposed to, in an automated way, so it won't
> > silently break as new changes to bitbake/oe-core are added? What
> > should such a test contain?
> > The specific form of tests (whether it's selftest, or maybe a part of
> > SDK testing, or something else) is secondary.
> >
> > Alex
>
> Hi Alex,
>
> I just send the v3 with a lot of improvements!
>
> Here is what I do during my dev:
> - lock a recipe's task for a given arch (qemux86-64 vs qemuarm)
> - modify the given task
> - check that for the selected arch, task is locked (but not for other arch)
>
> Maybe we need to be more formal with that?
>
> I would be happy to provide a test script if needed.
>
> Cheers
> Julien

Re: [OE-core] [RFC v2 0/2] Add bblock helper script

[Julien Stephan]

Le jeu. 27 juil. 2023 à 16:28, Alexander Kanavin
<alex.kanavin@gmail.com> a écrit :
>
> Sure; if you can write a script and show it here, we can figure out
> where to place the test officially.
>
> Alex

Hi Alexander,

I sent a v4 with an oeqa self test. What do you think about it?
https://lists.openembedded.org/g/openembedded-core/message/185408

Cheers
Julien

>
> On Thu, 27 Jul 2023 at 16:15, Julien Stephan <jstephan@baylibre.com> wrote:
> >
> > On Mon, Jul 24, 2023 at 10:36:28AM +0200, Alexander Kanavin wrote:
> > > On Mon, 24 Jul 2023 at 10:25, Julien Stephan <jstephan@baylibre.com> wrote:
> > > > > Thanks for working on this. Should we think about how this could be tested?
> > > > Do you mean we should consider addind self test for bblock? or you want
> > > > to discuss about acceptance criteria for bblock?
> > >
> > > Hello,
> > > I mean mostly about what should be tested. How can we verify that the
> > > script does what it's supposed to, in an automated way, so it won't
> > > silently break as new changes to bitbake/oe-core are added? What
> > > should such a test contain?
> > > The specific form of tests (whether it's selftest, or maybe a part of
> > > SDK testing, or something else) is secondary.
> > >
> > > Alex
> >
> > Hi Alex,
> >
> > I just send the v3 with a lot of improvements!
> >
> > Here is what I do during my dev:
> > - lock a recipe's task for a given arch (qemux86-64 vs qemuarm)
> > - modify the given task
> > - check that for the selected arch, task is locked (but not for other arch)
> >
> > Maybe we need to be more formal with that?
> >
> > I would be happy to provide a test script if needed.
> >
> > Cheers
> > Julien

Re: [OE-core] [RFC v2 0/2] Add bblock helper script

[Alexander Kanavin]

On Mon, 7 Aug 2023 at 15:38, Julien Stephan <jstephan@baylibre.com> wrote:
> I sent a v4 with an oeqa self test. What do you think about it?
> https://lists.openembedded.org/g/openembedded-core/message/185408

Right now I don't have any particular comments about v4, but others
do, I think? You may have to make further fixes if something fails in
QA with these changes.

Alex