Re: [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

Julien Stephan
2023-08-02T14:44:49Z

On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> adding a new command in cooker to compute and get task signatures
>
> this commit also add the associated command and event needed to get the
> signatures using tinfoil
>
> Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> ---
> bitbake/lib/bb/command.py | 6 ++++++
> bitbake/lib/bb/cooker.py | 16 ++++++++++++++++
> bitbake/lib/bb/event.py | 8 ++++++++
> 3 files changed, 30 insertions(+)
>
> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> index a355f56c60c..12202779ac0 100644
> --- a/bitbake/lib/bb/command.py
> +++ b/bitbake/lib/bb/command.py
> @@ -776,3 +776,9 @@ class CommandsAsync:
> bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> command.finishAsyncCommand()
> findSigInfo.needcache = False
> +
> + def getTaskSignatures(self, command, params):
> + res = command.cooker.getTaskSignatures(params[0], params[1])
> + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> + command.finishAsyncCommand()
> + getTaskSignatures.needcache = True
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 11c9fa2c40d..687cdde5e6d 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1542,6 +1542,22 @@ class BBCooker:
>
> self.idleCallBackRegister(buildFileIdle, rq)
>
> + def getTaskSignatures(self, target, task):
> + sig = []
> +
> + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> + rq.rqdata.prepare()
> +
> + for key in rq.rqdata.runtaskentries:
> + pn = bb.parse.siggen.tidtopn[key]
> + taskname = bb.runqueue.taskname_from_tid(key)
> + if pn in target:
> + if (task and taskname in task) or (not task):
> + rq.rqdata.prepare_task_hash(key)
> + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> + return sig
> +

Hi all,

I would like some feedback on this function. The goal of this function
was to be able to get a task signature (for bblock to use it, see bblock
series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks

Using buildTaskData with "do_build" task gives me almost all tasks
signatures, at least, the "normal recipe build tasks" according to the doc [1].
But we don't get the signatures for the "Manually called Tasks"

At first I thought it would be good enough for bblock use case, but now
I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,
but running `bitbake -c listtasks` gives *all* the tasks
* this function as is is not generic enough: we cannot get the task
signature for all the "manually called tasks" (but is that really a
problem?)
* bblock cannot lock those tasks

So what do you think? Should I make this function more generic and
called it several times to get *all* the tasks signature?

Cheers
Julien
[1]: https://docs.yoctoproject.org/ref-manual/tasks.html

> def buildTargets(self, targets, task):
> """
> Attempt to build the targets specified
> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
> index 0d0e0a68aac..f8acacd80d1 100644
> --- a/bitbake/lib/bb/event.py
> +++ b/bitbake/lib/bb/event.py
> @@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
> Event.__init__(self)
> self.result = result
>
> +class GetTaskSignatureResult(Event):
> + """
> + Event to return results from GetTaskSignatures command
> + """
> + def __init__(self, sig):
> + Event.__init__(self)
> + self.sig = sig
> +
> class ParseError(Event):
> """
> Event to indicate parse failed
> --
> 2.41.0
>