* [PATCH v3 2/3] bitbake: cooker: add a new function to retrieve task signatures
2023-07-27 13:56 [PATCH v3 0/3] Add bblock helper scripts Julien Stephan
2023-07-27 13:56 ` [PATCH v3 1/3] bitbake.conf: include bblock.conf Julien Stephan
@ 2023-07-27 13:56 ` Julien Stephan
2023-07-27 13:56 ` [PATCH v3 3/3] scripts/bblock: add a script to lock/unlock recipes Julien Stephan
2023-07-29 22:44 ` [OE-core] [PATCH v3 0/3] Add bblock helper scripts Alexandre Belloni
3 siblings, 0 replies; 6+ messages in thread
From: Julien Stephan @ 2023-07-27 13:56 UTC (permalink / raw)
To: openembedded-core; +Cc: Julien Stephan
adding a new command in cooker to compute and get task signatures
this commit also add the associated command and event needed to get the
signatures using tinfoil
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
bitbake/lib/bb/command.py | 6 ++++++
bitbake/lib/bb/cooker.py | 16 ++++++++++++++++
bitbake/lib/bb/event.py | 8 ++++++++
3 files changed, 30 insertions(+)
diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index a355f56c60c..12202779ac0 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -776,3 +776,9 @@ class CommandsAsync:
bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
command.finishAsyncCommand()
findSigInfo.needcache = False
+
+ def getTaskSignatures(self, command, params):
+ res = command.cooker.getTaskSignatures(params[0], params[1])
+ bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+ command.finishAsyncCommand()
+ getTaskSignatures.needcache = True
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 11c9fa2c40d..687cdde5e6d 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1542,6 +1542,22 @@ class BBCooker:
self.idleCallBackRegister(buildFileIdle, rq)
+ def getTaskSignatures(self, target, task):
+ sig = []
+
+ taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+ rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+ rq.rqdata.prepare()
+
+ for key in rq.rqdata.runtaskentries:
+ pn = bb.parse.siggen.tidtopn[key]
+ taskname = bb.runqueue.taskname_from_tid(key)
+ if pn in target:
+ if (task and taskname in task) or (not task):
+ rq.rqdata.prepare_task_hash(key)
+ sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+ return sig
+
def buildTargets(self, targets, task):
"""
Attempt to build the targets specified
diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 0d0e0a68aac..f8acacd80d1 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
Event.__init__(self)
self.result = result
+class GetTaskSignatureResult(Event):
+ """
+ Event to return results from GetTaskSignatures command
+ """
+ def __init__(self, sig):
+ Event.__init__(self)
+ self.sig = sig
+
class ParseError(Event):
"""
Event to indicate parse failed
--
2.41.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v3 3/3] scripts/bblock: add a script to lock/unlock recipes
2023-07-27 13:56 [PATCH v3 0/3] Add bblock helper scripts Julien Stephan
2023-07-27 13:56 ` [PATCH v3 1/3] bitbake.conf: include bblock.conf Julien Stephan
2023-07-27 13:56 ` [PATCH v3 2/3] bitbake: cooker: add a new function to retrieve task signatures Julien Stephan
@ 2023-07-27 13:56 ` Julien Stephan
2023-07-29 22:44 ` [OE-core] [PATCH v3 0/3] Add bblock helper scripts Alexandre Belloni
3 siblings, 0 replies; 6+ messages in thread
From: Julien Stephan @ 2023-07-27 13:56 UTC (permalink / raw)
To: openembedded-core; +Cc: Julien Stephan
bblock script allows to lock/unlock recipes to latest task signatures.
The idea is to prevent some recipes to be rebuilt during development.
For example when working on rust recipe, one may not want rust-native to be
rebuilt.
This tool can be used, with proper environment set up, using the following
command:
bblock <recipe_name>
if a <recipe_name>'s task signature change, this task will not be built again and
sstate cache will be used.
[YOCTO #13425]
Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 182 insertions(+)
create mode 100755 scripts/bblock
diff --git a/scripts/bblock b/scripts/bblock
new file mode 100755
index 00000000000..a9c5583127e
--- /dev/null
+++ b/scripts/bblock
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def getTaskSignatures(tinfoil, pn, tasks):
+ tinfoil.set_event_mask(
+ [
+ "bb.event.GetTaskSignatureResult",
+ "logging.LogRecord",
+ "bb.command.CommandCompleted",
+ "bb.command.CommandFailed",
+ ]
+ )
+ ret = tinfoil.run_command("getTaskSignatures", pn, tasks)
+ if ret:
+ while True:
+ event = tinfoil.wait_event(1)
+ if event:
+ if isinstance(event, bb.command.CommandCompleted):
+ break
+ elif isinstance(event, bb.command.CommandFailed):
+ logger.error(str(event))
+ sys.exit(2)
+ elif isinstance(event, bb.event.GetTaskSignatureResult):
+ sig = event.sig
+ elif isinstance(event, logging.LogRecord):
+ logger.handle(event)
+ else:
+ logger.error("No result returned from getTaskSignatures command")
+ sys.exit(2)
+ return sig
+
+
+def parseRecipe(tinfoil, recipe):
+ try:
+ tinfoil.parse_recipes()
+ d = tinfoil.parse_recipe(recipe)
+ except Exception:
+ logger.error("Failed to get recipe info for: %s" % recipe)
+ sys.exit(1)
+ return d
+
+
+def bblockDump(lockfile):
+ try:
+ with open(lockfile, "r") as lockfile:
+ for line in lockfile:
+ print(line.strip())
+ except IOError:
+ return 1
+ return 0
+
+
+def bblockReset(lockfile, pns, package_archs, tasks):
+ if not pns:
+ logger.info("Unlocking all recipes")
+ try:
+ os.remove(lockfile)
+ except FileNotFoundError:
+ pass
+ else:
+ logger.info("Unlocking {pns}".format(pns=pns))
+ tmp_lockfile = lockfile + ".tmp"
+ with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+ for line in infile:
+ if not (
+ any(element in line for element in pns)
+ and any(element in line for element in package_archs.split())
+ ):
+ outfile.write(line)
+ else:
+ if tasks and not any(element in line for element in tasks):
+ outfile.write(line)
+ os.remove(lockfile)
+ os.rename(tmp_lockfile, lockfile)
+
+
+def main():
+ parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+ parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+ parser.add_argument(
+ "-t",
+ "--tasks",
+ help="Comma separated list of tasks",
+ type=lambda s: [task for task in s.split(",")],
+ )
+ parser.add_argument(
+ "-r",
+ "--reset",
+ action="store_true",
+ help="Unlock pn recipes, or all recipes if pn is empty",
+ )
+ parser.add_argument(
+ "-d",
+ "--dump",
+ action="store_true",
+ help="Dump generated bblock.conf file",
+ )
+
+ global_args, unparsed_args = parser.parse_known_args()
+
+ with bb.tinfoil.Tinfoil() as tinfoil:
+ tinfoil.prepare(config_only=True)
+
+ package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+ builddir = tinfoil.config_data.getVar("TOPDIR")
+ lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+ if global_args.dump:
+ bblockDump(lockfile)
+ return 0
+
+ if global_args.reset:
+ bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks)
+ return 0
+
+ with open(lockfile, "a") as lockfile:
+ s = ""
+ if lockfile.tell() == 0:
+ s = "# Generated by bblock\n"
+ s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n'
+ s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+ s += "\n"
+
+ for pn in global_args.pn:
+ d = parseRecipe(tinfoil, pn)
+ package_arch = d.getVar("PACKAGE_ARCH")
+ siggen_locked_sigs_package_arch = d.getVar(
+ "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch)
+ )
+ sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks)
+ for sig in sigs:
+ new_entry = "{pn}:{taskname}:{sig}".format(
+ pn=sig[0], taskname=sig[1], sig=sig[2]
+ )
+ if (
+ siggen_locked_sigs_package_arch
+ and not new_entry in siggen_locked_sigs_package_arch
+ ) or not siggen_locked_sigs_package_arch:
+ s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format(
+ package_arch=package_arch, new_entry=new_entry
+ )
+ lockfile.write(s)
+ return 0
+
+
+if __name__ == "__main__":
+ try:
+ ret = main()
+ except Exception:
+ ret = 1
+ import traceback
+
+ traceback.print_exc()
+ sys.exit(ret)
--
2.41.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [OE-core] [PATCH v3 0/3] Add bblock helper scripts
2023-07-27 13:56 [PATCH v3 0/3] Add bblock helper scripts Julien Stephan
` (2 preceding siblings ...)
2023-07-27 13:56 ` [PATCH v3 3/3] scripts/bblock: add a script to lock/unlock recipes Julien Stephan
@ 2023-07-29 22:44 ` Alexandre Belloni
2023-08-02 14:26 ` Julien Stephan
3 siblings, 1 reply; 6+ messages in thread
From: Alexandre Belloni @ 2023-07-29 22:44 UTC (permalink / raw)
To: Julien Stephan; +Cc: openembedded-core
Hello,
On 27/07/2023 15:56:09+0200, Julien Stephan wrote:
> Hi all,
>
> This is v3 from bblock script.
>
This doesn't apply cleanly on master, can you rebase?
Thanks!
> Improvement from v2:
> * Add a function in bb.cooker to compute task signatures
> * Replace the findSigInfo function by the new created one. This has the
> following advantages:
> * findSigInfo needs the task to be already built to get the siginfo
> file, meaning we cannot lock a recipe on a fresh build
> * we can now generate the signatures for all available task of a given
> recipe
> * Check if a given task is already locked. If so, don't duplicate
> entry in bblock.conf
>
> Limitations:
> * Needs to taint tasks that are locked, to display a warning
> * I may be still missing some checks on user input
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
>
> I did some tests using qemux86-64 and qemuarm but I may be missing some
> corner cases.
>
> Improvement from V1:
> * Signatures are now package architecture specific meaning that if you
> switch MACHINE, the lock sig will not be taken into account
> * I added the -r option to unlock recipes
> * I added a -d option to display the current bblock.conf
> * Added an include directive for conf/bblock.conf inside bitbake.conf
> * Added -t option to specify the tasks to lock/unlock
>
> Limitations:
> * I may be still missing some checks on user input
> * I need to find a way to get the list of tasks ( by default still lock
> only the do_compile for now, unless -t is specified)
> * Do not check if a particular recipe/task is already locked when trying
> to add lock. So entries may appear multiple times
> * We still need the signature of the tasks to be already computed before
> locking. Need to find a way to generate it if missing
>
> V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> V1: https://lists.openembedded.org/g/openembedded-core/message/184584
>
> My branch is available here [1]
>
> Cheers
> Julien
>
> [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
>
> Julien Stephan (3):
> bitbake.conf: include bblock.conf
> bitbake: cooker: add a new function to retrieve task signatures
> scripts/bblock: add a script to lock/unlock recipes
>
> bitbake/lib/bb/command.py | 6 ++
> bitbake/lib/bb/cooker.py | 16 ++++
> bitbake/lib/bb/event.py | 8 ++
> meta/conf/bitbake.conf | 1 +
> scripts/bblock | 182 ++++++++++++++++++++++++++++++++++++++
> 5 files changed, 213 insertions(+)
> create mode 100755 scripts/bblock
>
> --
> 2.41.0
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#184932): https://lists.openembedded.org/g/openembedded-core/message/184932
> Mute This Topic: https://lists.openembedded.org/mt/100390731/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 6+ messages in thread