Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > adding a new command in cooker to compute and get task signatures
> > 
> > this commit also add the associated command and event needed to get the
> > signatures using tinfoil
> > 
> > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > ---
> >  bitbake/lib/bb/command.py |  6 ++++++
> >  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
> >  bitbake/lib/bb/event.py   |  8 ++++++++
> >  3 files changed, 30 insertions(+)
> > 
> > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > index a355f56c60c..12202779ac0 100644
> > --- a/bitbake/lib/bb/command.py
> > +++ b/bitbake/lib/bb/command.py
> > @@ -776,3 +776,9 @@ class CommandsAsync:
> >          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> >          command.finishAsyncCommand()
> >      findSigInfo.needcache = False
> > +
> > +    def getTaskSignatures(self, command, params):
> > +        res = command.cooker.getTaskSignatures(params[0], params[1])
> > +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > +        command.finishAsyncCommand()
> > +    getTaskSignatures.needcache = True
> > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > index 11c9fa2c40d..687cdde5e6d 100644
> > --- a/bitbake/lib/bb/cooker.py
> > +++ b/bitbake/lib/bb/cooker.py
> > @@ -1542,6 +1542,22 @@ class BBCooker:
> > 
> >          self.idleCallBackRegister(buildFileIdle, rq)
> > 
> > +    def getTaskSignatures(self, target, task):
> > +        sig = []
> > +
> > +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > +        rq.rqdata.prepare()
> > +
> > +        for key in rq.rqdata.runtaskentries:
> > +            pn = bb.parse.siggen.tidtopn[key]
> > +            taskname = bb.runqueue.taskname_from_tid(key)
> > +            if pn in target:
> > +                if (task and taskname in task) or (not task):
> > +                    rq.rqdata.prepare_task_hash(key)
> > +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > +        return sig
> > +

Firstly, thanks for working on this! These changes aren't simple and it
means the review takes longer as we need to get this right.

> I would like some feedback on this function. The goal of this function
> was to be able to get a task signature (for bblock to use it, see bblock
> series). The idea was:
> - get the task signatures of "task"
> - if "task" is empty, get the signature of all tasks
> 
> Using buildTaskData with "do_build" task gives me almost all tasks
> signatures, at least, the "normal recipe build tasks" according to the doc [1].
> But we don't get the signatures for the "Manually called Tasks"
> 
> At first I thought it would be good enough for bblock use case, but now
> I am not so sure anymore:
> * for oeqa self test, I need to have access to the same list of tasks,

I think you need to query the tasks available for a recipe, remove the
"nostamp" ones and then return the signatures for the rest.

>   but running `bitbake -c listtasks` gives *all* the tasks
> * this function as is is not generic enough: we cannot get the task
>   signature for all the "manually called tasks" (but is that really a
>   problem?)
> * bblock cannot lock those tasks
> 
> So what do you think? Should I make this function more generic and
> called it several times to get *all* the tasks signature?

I think bitbake should internally be able to work out the nostamp ones
and return a list of all of them with the nostamp ones excluded.

Let me know if you need some pointers on how to do that in bitbake, it
isn't entirely straightforward and I'd have to look it up. I *think*
(without looking to know for sure) that the dataCache object does have
the task data in it which can be looked up by target and that lists
which ones are nostamp.

Cheers,

Richard