Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

Julien Stephan
2023-08-22T09:28:17Z

Le ven. 11 août 2023 à 17:16, Richard Purdie
<richard.purdie@linuxfoundation.org> a écrit :
>
> On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> > On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > > adding a new command in cooker to compute and get task signatures
> > >
> > > this commit also add the associated command and event needed to get the
> > > signatures using tinfoil
> > >
> > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > ---
> > > bitbake/lib/bb/command.py | 6 ++++++
> > > bitbake/lib/bb/cooker.py | 16 ++++++++++++++++
> > > bitbake/lib/bb/event.py | 8 ++++++++
> > > 3 files changed, 30 insertions(+)
> > >
> > > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > > index a355f56c60c..12202779ac0 100644
> > > --- a/bitbake/lib/bb/command.py
> > > +++ b/bitbake/lib/bb/command.py
> > > @@ -776,3 +776,9 @@ class CommandsAsync:
> > > bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> > > command.finishAsyncCommand()
> > > findSigInfo.needcache = False
> > > +
> > > + def getTaskSignatures(self, command, params):
> > > + res = command.cooker.getTaskSignatures(params[0], params[1])
> > > + bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > > + command.finishAsyncCommand()
> > > + getTaskSignatures.needcache = True
> > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > > index 11c9fa2c40d..687cdde5e6d 100644
> > > --- a/bitbake/lib/bb/cooker.py
> > > +++ b/bitbake/lib/bb/cooker.py
> > > @@ -1542,6 +1542,22 @@ class BBCooker:
> > >
> > > self.idleCallBackRegister(buildFileIdle, rq)
> > >
> > > + def getTaskSignatures(self, target, task):
> > > + sig = []
> > > +
> > > + taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > > + rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > > + rq.rqdata.prepare()
> > > +
> > > + for key in rq.rqdata.runtaskentries:
> > > + pn = bb.parse.siggen.tidtopn[key]
> > > + taskname = bb.runqueue.taskname_from_tid(key)
> > > + if pn in target:
> > > + if (task and taskname in task) or (not task):
> > > + rq.rqdata.prepare_task_hash(key)
> > > + sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > > + return sig
> > > +
>
> Firstly, thanks for working on this! These changes aren't simple and it
> means the review takes longer as we need to get this right.
>
> > I would like some feedback on this function. The goal of this function
> > was to be able to get a task signature (for bblock to use it, see bblock
> > series). The idea was:
> > - get the task signatures of "task"
> > - if "task" is empty, get the signature of all tasks
> >
> > Using buildTaskData with "do_build" task gives me almost all tasks
> > signatures, at least, the "normal recipe build tasks" according to the doc [1].
> > But we don't get the signatures for the "Manually called Tasks"
> >
> > At first I thought it would be good enough for bblock use case, but now
> > I am not so sure anymore:
> > * for oeqa self test, I need to have access to the same list of tasks,
>
> I think you need to query the tasks available for a recipe, remove the
> "nostamp" ones and then return the signatures for the rest.
>
> > but running `bitbake -c listtasks` gives *all* the tasks
> > * this function as is is not generic enough: we cannot get the task
> > signature for all the "manually called tasks" (but is that really a
> > problem?)
> > * bblock cannot lock those tasks
> >
> > So what do you think? Should I make this function more generic and
> > called it several times to get *all* the tasks signature?
>
> I think bitbake should internally be able to work out the nostamp ones
> and return a list of all of them with the nostamp ones excluded.
>
> Let me know if you need some pointers on how to do that in bitbake, it
> isn't entirely straightforward and I'd have to look it up. I *think*
> (without looking to know for sure) that the dataCache object does have
> the task data in it which can be looked up by target and that lists
> which ones are nostamp.

Hi Richard,

Sorry for the late reply I was off last week.
Using rq.rqdata.dataCaches I am able to get the full list of tasks and
filter out the no stamps.
But it is still unclear for me what to do :( Here is what I
understood, please correct me if I am wrong:
* in getTaskSignatures (cooker.py): I should get the tasks list and
filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ),
then compute the signature for all
* in oeqa self test, I get the list of tasks using 'bitbake -c
listtasks', but how to filter out the 'nostamps' ones?

Cheers
Julien
>
> Cheers,
>
> Richard
>
>