[PATCH v4 0/5] Add bblock helper scripts

[PATCH v4 0/5] Add bblock helper scripts

[Julien Stephan]

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
  display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the
  following advantages:
    * findSigInfo needs the task to be already built to get the siginfo
      file, meaning we cannot lock a recipe on a fresh build
    * we can now generate the signatures for all available task of a given
      recipe
* Check if a given task is already locked. If so, don't duplicate
  entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some
corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you
  switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock
  only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying
  to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before
  locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
  bitbake.conf: include bblock.conf
  bitbake: cooker: add a new function to retrieve task signatures
  sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
  scripts/bblock: add a script to lock/unlock recipes
  oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

[PATCH v4 1/5] bitbake.conf: include bblock.conf

[Julien Stephan]

include conf/bblock.conf. This file is generated by the bblock tool. It
locks some package tasks by fixing their signatures. See bblock -h for
more details

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/conf/bitbake.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 475d6523bb8..e4a6bdef0ea 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -832,6 +832,7 @@ include conf/documentation.conf
 include conf/licenses.conf
 require conf/sanity.conf
 require conf/cve-check-map.conf
+include conf/bblock.conf
 
 ##################################################################
 # Weak variables (usually to retain backwards compatibility)
-- 
2.41.0

[PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

[Julien Stephan]

adding a new command in cooker to compute and get task signatures

this commit also add the associated command and event needed to get the
signatures using tinfoil

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 bitbake/lib/bb/command.py |  6 ++++++
 bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
 bitbake/lib/bb/event.py   |  8 ++++++++
 3 files changed, 30 insertions(+)

diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
index a355f56c60c..12202779ac0 100644
--- a/bitbake/lib/bb/command.py
+++ b/bitbake/lib/bb/command.py
@@ -776,3 +776,9 @@ class CommandsAsync:
         bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
         command.finishAsyncCommand()
     findSigInfo.needcache = False
+
+    def getTaskSignatures(self, command, params):
+        res = command.cooker.getTaskSignatures(params[0], params[1])
+        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
+        command.finishAsyncCommand()
+    getTaskSignatures.needcache = True
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 11c9fa2c40d..687cdde5e6d 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1542,6 +1542,22 @@ class BBCooker:
 
         self.idleCallBackRegister(buildFileIdle, rq)
 
+    def getTaskSignatures(self, target, task):
+        sig = []
+
+        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
+        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
+        rq.rqdata.prepare()
+
+        for key in rq.rqdata.runtaskentries:
+            pn = bb.parse.siggen.tidtopn[key]
+            taskname = bb.runqueue.taskname_from_tid(key)
+            if pn in target:
+                if (task and taskname in task) or (not task):
+                    rq.rqdata.prepare_task_hash(key)
+                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
+        return sig
+
     def buildTargets(self, targets, task):
         """
         Attempt to build the targets specified
diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
index 0d0e0a68aac..f8acacd80d1 100644
--- a/bitbake/lib/bb/event.py
+++ b/bitbake/lib/bb/event.py
@@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
         Event.__init__(self)
         self.result = result
 
+class GetTaskSignatureResult(Event):
+    """
+    Event to return results from GetTaskSignatures command
+    """
+    def __init__(self, sig):
+        Event.__init__(self)
+        self.sig = sig
+
 class ParseError(Event):
     """
     Event to indicate parse failed
-- 
2.41.0

Re: [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

[Julien Stephan]

On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> adding a new command in cooker to compute and get task signatures
>
> this commit also add the associated command and event needed to get the
> signatures using tinfoil
>
> Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> ---
>  bitbake/lib/bb/command.py |  6 ++++++
>  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
>  bitbake/lib/bb/event.py   |  8 ++++++++
>  3 files changed, 30 insertions(+)
>
> diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> index a355f56c60c..12202779ac0 100644
> --- a/bitbake/lib/bb/command.py
> +++ b/bitbake/lib/bb/command.py
> @@ -776,3 +776,9 @@ class CommandsAsync:
>          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
>          command.finishAsyncCommand()
>      findSigInfo.needcache = False
> +
> +    def getTaskSignatures(self, command, params):
> +        res = command.cooker.getTaskSignatures(params[0], params[1])
> +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> +        command.finishAsyncCommand()
> +    getTaskSignatures.needcache = True
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 11c9fa2c40d..687cdde5e6d 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1542,6 +1542,22 @@ class BBCooker:
>
>          self.idleCallBackRegister(buildFileIdle, rq)
>
> +    def getTaskSignatures(self, target, task):
> +        sig = []
> +
> +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> +        rq.rqdata.prepare()
> +
> +        for key in rq.rqdata.runtaskentries:
> +            pn = bb.parse.siggen.tidtopn[key]
> +            taskname = bb.runqueue.taskname_from_tid(key)
> +            if pn in target:
> +                if (task and taskname in task) or (not task):
> +                    rq.rqdata.prepare_task_hash(key)
> +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> +        return sig
> +

Hi all,

I would like some feedback on this function. The goal of this function
was to be able to get a task signature (for bblock to use it, see bblock
series). The idea was:
- get the task signatures of "task"
- if "task" is empty, get the signature of all tasks

Using buildTaskData with "do_build" task gives me almost all tasks
signatures, at least, the "normal recipe build tasks" according to the doc [1].
But we don't get the signatures for the "Manually called Tasks"

At first I thought it would be good enough for bblock use case, but now
I am not so sure anymore:
* for oeqa self test, I need to have access to the same list of tasks,
  but running `bitbake -c listtasks` gives *all* the tasks
* this function as is is not generic enough: we cannot get the task
  signature for all the "manually called tasks" (but is that really a
  problem?)
* bblock cannot lock those tasks

So what do you think? Should I make this function more generic and
called it several times to get *all* the tasks signature?

Cheers
Julien
[1]: https://docs.yoctoproject.org/ref-manual/tasks.html

>      def buildTargets(self, targets, task):
>          """
>          Attempt to build the targets specified
> diff --git a/bitbake/lib/bb/event.py b/bitbake/lib/bb/event.py
> index 0d0e0a68aac..f8acacd80d1 100644
> --- a/bitbake/lib/bb/event.py
> +++ b/bitbake/lib/bb/event.py
> @@ -857,6 +857,14 @@ class FindSigInfoResult(Event):
>          Event.__init__(self)
>          self.result = result
>
> +class GetTaskSignatureResult(Event):
> +    """
> +    Event to return results from GetTaskSignatures command
> +    """
> +    def __init__(self, sig):
> +        Event.__init__(self)
> +        self.sig = sig
> +
>  class ParseError(Event):
>      """
>      Event to indicate parse failed
> --
> 2.41.0
>

Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

[Richard Purdie]

On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > adding a new command in cooker to compute and get task signatures
> > 
> > this commit also add the associated command and event needed to get the
> > signatures using tinfoil
> > 
> > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > ---
> >  bitbake/lib/bb/command.py |  6 ++++++
> >  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
> >  bitbake/lib/bb/event.py   |  8 ++++++++
> >  3 files changed, 30 insertions(+)
> > 
> > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > index a355f56c60c..12202779ac0 100644
> > --- a/bitbake/lib/bb/command.py
> > +++ b/bitbake/lib/bb/command.py
> > @@ -776,3 +776,9 @@ class CommandsAsync:
> >          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> >          command.finishAsyncCommand()
> >      findSigInfo.needcache = False
> > +
> > +    def getTaskSignatures(self, command, params):
> > +        res = command.cooker.getTaskSignatures(params[0], params[1])
> > +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > +        command.finishAsyncCommand()
> > +    getTaskSignatures.needcache = True
> > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > index 11c9fa2c40d..687cdde5e6d 100644
> > --- a/bitbake/lib/bb/cooker.py
> > +++ b/bitbake/lib/bb/cooker.py
> > @@ -1542,6 +1542,22 @@ class BBCooker:
> > 
> >          self.idleCallBackRegister(buildFileIdle, rq)
> > 
> > +    def getTaskSignatures(self, target, task):
> > +        sig = []
> > +
> > +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > +        rq.rqdata.prepare()
> > +
> > +        for key in rq.rqdata.runtaskentries:
> > +            pn = bb.parse.siggen.tidtopn[key]
> > +            taskname = bb.runqueue.taskname_from_tid(key)
> > +            if pn in target:
> > +                if (task and taskname in task) or (not task):
> > +                    rq.rqdata.prepare_task_hash(key)
> > +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > +        return sig
> > +

Firstly, thanks for working on this! These changes aren't simple and it
means the review takes longer as we need to get this right.

> I would like some feedback on this function. The goal of this function
> was to be able to get a task signature (for bblock to use it, see bblock
> series). The idea was:
> - get the task signatures of "task"
> - if "task" is empty, get the signature of all tasks
> 
> Using buildTaskData with "do_build" task gives me almost all tasks
> signatures, at least, the "normal recipe build tasks" according to the doc [1].
> But we don't get the signatures for the "Manually called Tasks"
> 
> At first I thought it would be good enough for bblock use case, but now
> I am not so sure anymore:
> * for oeqa self test, I need to have access to the same list of tasks,

I think you need to query the tasks available for a recipe, remove the
"nostamp" ones and then return the signatures for the rest.

>   but running `bitbake -c listtasks` gives *all* the tasks
> * this function as is is not generic enough: we cannot get the task
>   signature for all the "manually called tasks" (but is that really a
>   problem?)
> * bblock cannot lock those tasks
> 
> So what do you think? Should I make this function more generic and
> called it several times to get *all* the tasks signature?

I think bitbake should internally be able to work out the nostamp ones
and return a list of all of them with the nostamp ones excluded.

Let me know if you need some pointers on how to do that in bitbake, it
isn't entirely straightforward and I'd have to look it up. I *think*
(without looking to know for sure) that the dataCache object does have
the task data in it which can be looked up by target and that lists
which ones are nostamp.

Cheers,

Richard

Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

[Julien Stephan]

Le ven. 11 août 2023 à 17:16, Richard Purdie
<richard.purdie@linuxfoundation.org> a écrit :
>
> On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> > On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > > adding a new command in cooker to compute and get task signatures
> > >
> > > this commit also add the associated command and event needed to get the
> > > signatures using tinfoil
> > >
> > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > ---
> > >  bitbake/lib/bb/command.py |  6 ++++++
> > >  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
> > >  bitbake/lib/bb/event.py   |  8 ++++++++
> > >  3 files changed, 30 insertions(+)
> > >
> > > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > > index a355f56c60c..12202779ac0 100644
> > > --- a/bitbake/lib/bb/command.py
> > > +++ b/bitbake/lib/bb/command.py
> > > @@ -776,3 +776,9 @@ class CommandsAsync:
> > >          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> > >          command.finishAsyncCommand()
> > >      findSigInfo.needcache = False
> > > +
> > > +    def getTaskSignatures(self, command, params):
> > > +        res = command.cooker.getTaskSignatures(params[0], params[1])
> > > +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > > +        command.finishAsyncCommand()
> > > +    getTaskSignatures.needcache = True
> > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > > index 11c9fa2c40d..687cdde5e6d 100644
> > > --- a/bitbake/lib/bb/cooker.py
> > > +++ b/bitbake/lib/bb/cooker.py
> > > @@ -1542,6 +1542,22 @@ class BBCooker:
> > >
> > >          self.idleCallBackRegister(buildFileIdle, rq)
> > >
> > > +    def getTaskSignatures(self, target, task):
> > > +        sig = []
> > > +
> > > +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > > +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > > +        rq.rqdata.prepare()
> > > +
> > > +        for key in rq.rqdata.runtaskentries:
> > > +            pn = bb.parse.siggen.tidtopn[key]
> > > +            taskname = bb.runqueue.taskname_from_tid(key)
> > > +            if pn in target:
> > > +                if (task and taskname in task) or (not task):
> > > +                    rq.rqdata.prepare_task_hash(key)
> > > +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > > +        return sig
> > > +
>
> Firstly, thanks for working on this! These changes aren't simple and it
> means the review takes longer as we need to get this right.
>
> > I would like some feedback on this function. The goal of this function
> > was to be able to get a task signature (for bblock to use it, see bblock
> > series). The idea was:
> > - get the task signatures of "task"
> > - if "task" is empty, get the signature of all tasks
> >
> > Using buildTaskData with "do_build" task gives me almost all tasks
> > signatures, at least, the "normal recipe build tasks" according to the doc [1].
> > But we don't get the signatures for the "Manually called Tasks"
> >
> > At first I thought it would be good enough for bblock use case, but now
> > I am not so sure anymore:
> > * for oeqa self test, I need to have access to the same list of tasks,
>
> I think you need to query the tasks available for a recipe, remove the
> "nostamp" ones and then return the signatures for the rest.
>
> >   but running `bitbake -c listtasks` gives *all* the tasks
> > * this function as is is not generic enough: we cannot get the task
> >   signature for all the "manually called tasks" (but is that really a
> >   problem?)
> > * bblock cannot lock those tasks
> >
> > So what do you think? Should I make this function more generic and
> > called it several times to get *all* the tasks signature?
>
> I think bitbake should internally be able to work out the nostamp ones
> and return a list of all of them with the nostamp ones excluded.
>
> Let me know if you need some pointers on how to do that in bitbake, it
> isn't entirely straightforward and I'd have to look it up. I *think*
> (without looking to know for sure) that the dataCache object does have
> the task data in it which can be looked up by target and that lists
> which ones are nostamp.

Hi Richard,

Sorry for the late reply I was off last week.
Using rq.rqdata.dataCaches I am able to get the full list of tasks and
filter out the no stamps.
But it is still unclear for me what to do :( Here is what I
understood, please correct me if I am wrong:
* in getTaskSignatures (cooker.py): I should get the tasks list and
filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ),
then compute the signature for all
* in oeqa self test, I get the list of tasks using 'bitbake -c
listtasks', but how to filter out the 'nostamps' ones?

Cheers
Julien
>
> Cheers,
>
> Richard
>
>

Re: [OE-core] [PATCH v4 2/5] bitbake: cooker: add a new function to retrieve task signatures

[Richard Purdie]

Hi,

Sorry, I meant to reply to this and somehow it got lost with everything
else going on.

On Tue, 2023-08-22 at 11:27 +0200, Julien Stephan wrote:
> Le ven. 11 août 2023 à 17:16, Richard Purdie
> <richard.purdie@linuxfoundation.org> a écrit :
> > 
> > On Wed, 2023-08-02 at 16:44 +0200, Julien Stephan wrote:
> > > On Wed, Aug 02, 2023 at 04:24:29PM +0200, Julien Stephan wrote:
> > > > adding a new command in cooker to compute and get task signatures
> > > > 
> > > > this commit also add the associated command and event needed to get the
> > > > signatures using tinfoil
> > > > 
> > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > > ---
> > > >  bitbake/lib/bb/command.py |  6 ++++++
> > > >  bitbake/lib/bb/cooker.py  | 16 ++++++++++++++++
> > > >  bitbake/lib/bb/event.py   |  8 ++++++++
> > > >  3 files changed, 30 insertions(+)
> > > > 
> > > > diff --git a/bitbake/lib/bb/command.py b/bitbake/lib/bb/command.py
> > > > index a355f56c60c..12202779ac0 100644
> > > > --- a/bitbake/lib/bb/command.py
> > > > +++ b/bitbake/lib/bb/command.py
> > > > @@ -776,3 +776,9 @@ class CommandsAsync:
> > > >          bb.event.fire(bb.event.FindSigInfoResult(res), command.cooker.databuilder.mcdata[mc])
> > > >          command.finishAsyncCommand()
> > > >      findSigInfo.needcache = False
> > > > +
> > > > +    def getTaskSignatures(self, command, params):
> > > > +        res = command.cooker.getTaskSignatures(params[0], params[1])
> > > > +        bb.event.fire(bb.event.GetTaskSignatureResult(res), command.cooker.data)
> > > > +        command.finishAsyncCommand()
> > > > +    getTaskSignatures.needcache = True
> > > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > > > index 11c9fa2c40d..687cdde5e6d 100644
> > > > --- a/bitbake/lib/bb/cooker.py
> > > > +++ b/bitbake/lib/bb/cooker.py
> > > > @@ -1542,6 +1542,22 @@ class BBCooker:
> > > > 
> > > >          self.idleCallBackRegister(buildFileIdle, rq)
> > > > 
> > > > +    def getTaskSignatures(self, target, task):
> > > > +        sig = []
> > > > +
> > > > +        taskdata, runlist = self.buildTaskData(target, "do_build", self.configuration.halt)
> > > > +        rq = bb.runqueue.RunQueue(self, self.data, self.recipecaches, taskdata, runlist)
> > > > +        rq.rqdata.prepare()
> > > > +
> > > > +        for key in rq.rqdata.runtaskentries:
> > > > +            pn = bb.parse.siggen.tidtopn[key]
> > > > +            taskname = bb.runqueue.taskname_from_tid(key)
> > > > +            if pn in target:
> > > > +                if (task and taskname in task) or (not task):
> > > > +                    rq.rqdata.prepare_task_hash(key)
> > > > +                    sig.append([pn, taskname, rq.rqdata.get_task_unihash(key)])
> > > > +        return sig
> > > > +
> > 
> > Firstly, thanks for working on this! These changes aren't simple and it
> > means the review takes longer as we need to get this right.
> > 
> > > I would like some feedback on this function. The goal of this function
> > > was to be able to get a task signature (for bblock to use it, see bblock
> > > series). The idea was:
> > > - get the task signatures of "task"
> > > - if "task" is empty, get the signature of all tasks
> > > 
> > > Using buildTaskData with "do_build" task gives me almost all tasks
> > > signatures, at least, the "normal recipe build tasks" according to the doc [1].
> > > But we don't get the signatures for the "Manually called Tasks"
> > > 
> > > At first I thought it would be good enough for bblock use case, but now
> > > I am not so sure anymore:
> > > * for oeqa self test, I need to have access to the same list of tasks,
> > 
> > I think you need to query the tasks available for a recipe, remove the
> > "nostamp" ones and then return the signatures for the rest.
> > 
> > >   but running `bitbake -c listtasks` gives *all* the tasks
> > > * this function as is is not generic enough: we cannot get the task
> > >   signature for all the "manually called tasks" (but is that really a
> > >   problem?)
> > > * bblock cannot lock those tasks
> > > 
> > > So what do you think? Should I make this function more generic and
> > > called it several times to get *all* the tasks signature?
> > 
> > I think bitbake should internally be able to work out the nostamp ones
> > and return a list of all of them with the nostamp ones excluded.
> > 
> > Let me know if you need some pointers on how to do that in bitbake, it
> > isn't entirely straightforward and I'd have to look it up. I *think*
> > (without looking to know for sure) that the dataCache object does have
> > the task data in it which can be looked up by target and that lists
> > which ones are nostamp.
> 
> Sorry for the late reply I was off last week.
> Using rq.rqdata.dataCaches I am able to get the full list of tasks and
> filter out the no stamps.
> But it is still unclear for me what to do :( Here is what I
> understood, please correct me if I am wrong:
> * in getTaskSignatures (cooker.py): I should get the tasks list and
> filter out the 'nostamps' ones (and keep the 'setscene' ones ??? ),

You can skip the setscene ones as they're not interesting or needed for
what we need here, at least as I understand it as we never lock those.

> then compute the signature for all
> * in oeqa self test, I get the list of tasks using 'bitbake -c
> listtasks', but how to filter out the 'nostamps' ones?

Just thinking out loud, for a given recipe you could run
d.getVarFlag("do_clean", "nostamp") and see that it was set?

"bitbake-getvar -r bash do_clean -f nostamp --value" also shows that.

Cheers,

Richard

[PATCH v4 3/5] sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK

[Julien Stephan]

as of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and
"error", displaying respectively a warning or a fatal error message
only when a task is locked and the task signature is different from
the locked one.

The "info" level is introduced to add a "note" message to remind the
user that a recipe is locked even if the signature is equivalent to the
locked one.

The "warn" and "error" level display the warn/error message for each
task having a mismatch of the signature. Doing this with the "info"
level would result in very verbose output if there are several tasks
locked, so the info level will only print once the list of recipes that
have locked signature.

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/lib/oe/sstatesig.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index f943df181e6..90002f67550 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -104,6 +104,7 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         self.lockedhashfn = {}
         self.machine = data.getVar("MACHINE")
         self.mismatch_msgs = []
+        self.lockedsigs_msgs = ""
         self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or
                                 "").split()
         self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
@@ -264,6 +265,12 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         warn_msgs = []
         error_msgs = []
         sstate_missing_msgs = []
+        info_msgs = None
+
+        if self.lockedsigs:
+            self.lockedsigs_msgs = "The following recipes have locked tasks:"
+            for pn in self.lockedsigs:
+                self.lockedsigs_msgs += " %s" % (pn)
 
         for tid in sq_data['hash']:
             if tid not in found:
@@ -276,7 +283,9 @@ class SignatureGeneratorOEBasicHashMixIn(object):
                                                % (pn, taskname, sq_data['hash'][tid]))
 
         checklevel = d.getVar("SIGGEN_LOCKEDSIGS_TASKSIG_CHECK")
-        if checklevel == 'warn':
+        if checklevel == 'info':
+            info_msgs = self.lockedsigs_msgs
+        if checklevel == 'warn' or checklevel == 'info':
             warn_msgs += self.mismatch_msgs
         elif checklevel == 'error':
             error_msgs += self.mismatch_msgs
@@ -287,6 +296,8 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         elif checklevel == 'error':
             error_msgs += sstate_missing_msgs
 
+        if info_msgs:
+            bb.note(info_msgs)
         if warn_msgs:
             bb.warn("\n".join(warn_msgs))
         if error_msgs:
-- 
2.41.0

Re: [OE-core] [PATCH v4 3/5] sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK

[Richard Purdie]

On Wed, 2023-08-02 at 16:24 +0200, Julien Stephan wrote:
> as of now, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK can take 2 values: "warn" and
> "error", displaying respectively a warning or a fatal error message
> only when a task is locked and the task signature is different from
> the locked one.
> 
> The "info" level is introduced to add a "note" message to remind the
> user that a recipe is locked even if the signature is equivalent to the
> locked one.
> 
> The "warn" and "error" level display the warn/error message for each
> task having a mismatch of the signature. Doing this with the "info"
> level would result in very verbose output if there are several tasks
> locked, so the info level will only print once the list of recipes that
> have locked signature.
> 
> Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> ---
>  meta/lib/oe/sstatesig.py | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index f943df181e6..90002f67550 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -104,6 +104,7 @@ class SignatureGeneratorOEBasicHashMixIn(object):
>          self.lockedhashfn = {}
>          self.machine = data.getVar("MACHINE")
>          self.mismatch_msgs = []
> +        self.lockedsigs_msgs = ""
>          self.unlockedrecipes = (data.getVar("SIGGEN_UNLOCKED_RECIPES") or
>                                  "").split()
>          self.unlockedrecipes = { k: "" for k in self.unlockedrecipes }
> @@ -264,6 +265,12 @@ class SignatureGeneratorOEBasicHashMixIn(object):
>          warn_msgs = []
>          error_msgs = []
>          sstate_missing_msgs = []
> +        info_msgs = None
> +
> +        if self.lockedsigs:
> +            self.lockedsigs_msgs = "The following recipes have locked tasks:"
> +            for pn in self.lockedsigs:
> +                self.lockedsigs_msgs += " %s" % (pn)
>  
>          for tid in sq_data['hash']:
>              if tid not in found:

How about something like:

if len(pn) > 10:
    print "There are XX locked recipes (XX matching and XX not matching
parsing)"

so that in the case where lots of things are locked you get a summary
instead showing the amount of match/mismatch?

Cheers,

Richard

[PATCH v4 4/5] scripts/bblock: add a script to lock/unlock recipes

[Julien Stephan]

bblock script allows to lock/unlock recipes to latest task signatures.
The idea is to prevent some recipes to be rebuilt during development.
For example when working on rust recipe, one may not want rust-native to be
rebuilt.

This tool can be used, with proper environment set up, using the following
command:

bblock <recipe_name>

if a <recipe_name>'s task signature change, this task will not be built again and
sstate cache will be used.

[YOCTO #13425]

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/bblock | 182 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100755 scripts/bblock

diff --git a/scripts/bblock b/scripts/bblock
new file mode 100755
index 00000000000..e5cf78722c7
--- /dev/null
+++ b/scripts/bblock
@@ -0,0 +1,182 @@
+#!/usr/bin/env python3
+# bblock
+# lock/unlock task to latest signature
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import sys
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + "/lib"
+sys.path = sys.path + [lib_path]
+
+import scriptpath
+
+scriptpath.add_bitbake_lib_path()
+
+import bb.tinfoil
+import bb.msg
+
+import argparse_oe
+
+myname = os.path.basename(sys.argv[0])
+logger = bb.msg.logger_create(myname)
+
+
+def getTaskSignatures(tinfoil, pn, tasks):
+    tinfoil.set_event_mask(
+        [
+            "bb.event.GetTaskSignatureResult",
+            "logging.LogRecord",
+            "bb.command.CommandCompleted",
+            "bb.command.CommandFailed",
+        ]
+    )
+    ret = tinfoil.run_command("getTaskSignatures", pn, tasks)
+    if ret:
+        while True:
+            event = tinfoil.wait_event(1)
+            if event:
+                if isinstance(event, bb.command.CommandCompleted):
+                    break
+                elif isinstance(event, bb.command.CommandFailed):
+                    logger.error(str(event))
+                    sys.exit(2)
+                elif isinstance(event, bb.event.GetTaskSignatureResult):
+                    sig = event.sig
+                elif isinstance(event, logging.LogRecord):
+                    logger.handle(event)
+    else:
+        logger.error("No result returned from getTaskSignatures command")
+        sys.exit(2)
+    return sig
+
+
+def parseRecipe(tinfoil, recipe):
+    try:
+        tinfoil.parse_recipes()
+        d = tinfoil.parse_recipe(recipe)
+    except Exception:
+        logger.error("Failed to get recipe info for: %s" % recipe)
+        sys.exit(1)
+    return d
+
+
+def bblockDump(lockfile):
+    try:
+        with open(lockfile, "r") as lockfile:
+            for line in lockfile:
+                print(line.strip())
+    except IOError:
+        return 1
+    return 0
+
+
+def bblockReset(lockfile, pns, package_archs, tasks):
+    if not pns:
+        logger.info("Unlocking all recipes")
+        try:
+            os.remove(lockfile)
+        except FileNotFoundError:
+            pass
+    else:
+        logger.info("Unlocking {pns}".format(pns=pns))
+        tmp_lockfile = lockfile + ".tmp"
+        with open(lockfile, "r") as infile, open(tmp_lockfile, "w") as outfile:
+            for line in infile:
+                if not (
+                    any(element in line for element in pns)
+                    and any(element in line for element in package_archs.split())
+                ):
+                    outfile.write(line)
+                else:
+                    if tasks and not any(element in line for element in tasks):
+                        outfile.write(line)
+        os.remove(lockfile)
+        os.rename(tmp_lockfile, lockfile)
+
+
+def main():
+    parser = argparse_oe.ArgumentParser(description="Lock and unlock a recipe")
+    parser.add_argument("pn", nargs="*", help="Space separated list of recipe to lock")
+    parser.add_argument(
+        "-t",
+        "--tasks",
+        help="Comma separated list of tasks",
+        type=lambda s: [task for task in s.split(",")],
+    )
+    parser.add_argument(
+        "-r",
+        "--reset",
+        action="store_true",
+        help="Unlock pn recipes, or all recipes if pn is empty",
+    )
+    parser.add_argument(
+        "-d",
+        "--dump",
+        action="store_true",
+        help="Dump generated bblock.conf file",
+    )
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    with bb.tinfoil.Tinfoil() as tinfoil:
+        tinfoil.prepare(config_only=True)
+
+        package_archs = tinfoil.config_data.getVar("PACKAGE_ARCHS")
+        builddir = tinfoil.config_data.getVar("TOPDIR")
+        lockfile = "{builddir}/conf/bblock.conf".format(builddir=builddir)
+
+        if global_args.dump:
+            bblockDump(lockfile)
+            return 0
+
+        if global_args.reset:
+            bblockReset(lockfile, global_args.pn, package_archs, global_args.tasks)
+            return 0
+
+        with open(lockfile, "a") as lockfile:
+            s = ""
+            if lockfile.tell() == 0:
+                s = "# Generated by bblock\n"
+                s += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"\n'
+                s += 'SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"\n'
+                s += "\n"
+
+            for pn in global_args.pn:
+                d = parseRecipe(tinfoil, pn)
+                package_arch = d.getVar("PACKAGE_ARCH")
+                siggen_locked_sigs_package_arch = d.getVar(
+                    "SIGGEN_LOCKEDSIGS_{package_arch}".format(package_arch=package_arch)
+                )
+                sigs = getTaskSignatures(tinfoil, [pn], global_args.tasks)
+                for sig in sigs:
+                    new_entry = "{pn}:{taskname}:{sig}".format(
+                        pn=sig[0], taskname=sig[1], sig=sig[2]
+                    )
+                    if (
+                        siggen_locked_sigs_package_arch
+                        and not new_entry in siggen_locked_sigs_package_arch
+                    ) or not siggen_locked_sigs_package_arch:
+                        s += 'SIGGEN_LOCKEDSIGS_{package_arch} += "{new_entry}"\n'.format(
+                            package_arch=package_arch, new_entry=new_entry
+                        )
+            lockfile.write(s)
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.41.0

[PATCH v4 5/5] oeqa/selftest/bblock: add self test for bblock tool

[Julien Stephan]

it implements various combination of locking single/multiple recipe(s)/task(s)

it also tests that locked sig are architecture dependant

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/lib/oeqa/selftest/cases/bblock.py | 146 +++++++++++++++++++++++++
 1 file changed, 146 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py

diff --git a/meta/lib/oeqa/selftest/cases/bblock.py b/meta/lib/oeqa/selftest/cases/bblock.py
new file mode 100644
index 00000000000..803ea3d64c8
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/bblock.py
@@ -0,0 +1,146 @@
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stepahn <jstephan@baylibre.com>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import os
+import re
+
+import oeqa.utils.ftools as ftools
+from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars, bitbake
+
+from oeqa.selftest.case import OESelftestTestCase
+
+class BBLock(OESelftestTestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        super(BBLock, cls).setUpClass()
+        cls.lockfile = cls.builddir+"/conf/bblock.conf"
+
+    def unlock_recipes(self, recipes=None, tasks=None):
+        cmd = 'bblock -r '
+        if recipes:
+            cmd += ' '.join(recipes)
+        if tasks:
+            cmd += ' -t ' + ','.join(tasks)
+        result = runCmd(cmd)
+
+        if recipes:
+            # ensure all signatures are removed from lockfile
+            contents = ftools.read_file(self.lockfile)
+            for recipe in recipes:
+                for task in tasks:
+                    find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents)
+                    self.assertFalse(find_in_contents, msg = "%s:%s should not be present into bblock.conf anymore" % (recipe, task))
+                self.assertExists(self.lockfile)
+        else:
+            self.assertNotExists(self.lockfile)
+
+    def lock_recipes(self, recipes, tasks=None, machine=None):
+        cmd = 'bblock ' + ' '.join(recipes)
+        if tasks:
+            cmd += ' -t ' + ','.join(tasks)
+
+        if machine:
+            cmd = "MACHINE=" + machine + " " + cmd
+        with open("/tmp/tmp.txt", "w") as file:
+            file.write(cmd)
+        result = runCmd(cmd)
+
+        self.assertExists(self.lockfile)
+
+        # ensure all signatures are added to lockfile
+        contents = ftools.read_file(self.lockfile)
+        for recipe in recipes:
+            for task in tasks:
+                find_in_contents = re.search("SIGGEN_LOCKEDSIGS_.+\s\+=\s\"%s:%s:.*\"" % (recipe, task), contents)
+                self.assertTrue(find_in_contents, msg = "%s:%s was not added into bblock.conf. bblock output: %s" % (recipe, task, result.output))
+
+    def modify_tasks(self, recipes, tasks):
+        task_append = ""
+        for recipe in recipes:
+            bb_vars = get_bb_vars(['PV'], recipe)
+            recipe_pv = bb_vars['PV']
+            recipe_append_file = recipe + '_' + recipe_pv + '.bbappend'
+
+            os.mkdir(os.path.join(self.testlayer_path, 'recipes-test', recipe))
+            recipe_append_path = os.path.join(self.testlayer_path, 'recipes-test', recipe, recipe_append_file)
+
+            for task in tasks:
+                task_append += "%s:append() {\necho \"modify task hash\"\n}\n" % task
+            ftools.write_file(recipe_append_path, task_append)
+            self.add_command_to_tearDown('rm -rf %s' % os.path.join(self.testlayer_path, 'recipes-test', recipe))
+
+    def test_lock_single_recipe_single_task(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_single_recipe_multiple_tasks(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile" ,"do_install"]
+        self._run_test(recipes, tasks)
+
+    def test_lock_architecture_specific(self):
+        recipes = ["quilt"]
+        tasks = ["do_compile"]
+
+        # lock quilt's do_compile task for qemuarm machine
+        self.lock_recipes(recipes, tasks, "qemuarm")
+
+        # modify quilt's do_compile task
+        self.modify_tasks(recipes, tasks)
+
+        # build quilt using by default qemux86-64.
+        # No Note/Warning should be emitted since sig is locked for qemuarm
+        # (quilt package is architecture dependant)
+        info_message = "NOTE: The following recipes have locked tasks: " + recipes[0]
+        warn_message = "The %s:%s sig is computed to be" % (recipes[0], tasks[0])
+        result = bitbake(recipes[0] + " -n")
+        self.assertNotIn(info_message, result.output)
+        self.assertNotIn(warn_message, result.output)
+
+    def test_lock_multiple_recipe_single_task(self):
+        recipes = ["quilt", "bc"]
+        tasks= ["do_compile"]
+        self._run_test(recipes, tasks)
+
+    def _run_test(self, recipes, tasks=None):
+        # unlock all recipes and ensure no bblock.conf file exist
+        self.unlock_recipes()
+
+        # lock tasks for recipes
+        result = self.lock_recipes(recipes, tasks)
+
+        # build recipes. At this stage we should have a Note about recipes
+        # having locked task's sig, but no warning since sig still match
+        info_message = "NOTE: The following recipes have locked tasks: " + " ".join(recipes)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
+
+        ## modify all tasks that are locked to trigger a sig change then build the recipes
+        ## at this stage we should have a Note as before, but also a warning for all
+        ## locked stage indicating the sig mismatch
+        self.modify_tasks(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertIn(warn_message, result.output)
+
+        # unlock all tasks and rebuild, no more Note/Warning should remain
+        self.unlock_recipes(recipes, tasks)
+        for recipe in recipes:
+            result = bitbake(recipe + " -n")
+            self.assertNotIn(info_message, result.output)
+            for task in tasks:
+                warn_message = "The %s:%s sig is computed to be" % (recipe, task)
+                self.assertNotIn(warn_message, result.output)
-- 
2.41.0

RE: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Chen, Qi]

When ' include conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default?

Regards,
Qi

-----Original Message-----
From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Julien Stephan
Sent: Wednesday, August 2, 2023 10:24 PM
To: openembedded-core@lists.openembedded.org
Cc: Julien Stephan <jstephan@baylibre.com>
Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

Hi all,

This is v4 for bblock script.

Improvement from v3:
* Add self test
* Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
  display a Note when recipe contains locked task(s)

Limitations:
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

Improvement from v2:
* Add a function in bb.cooker to compute task signatures
* Replace the findSigInfo function by the new created one. This has the
  following advantages:
    * findSigInfo needs the task to be already built to get the siginfo
      file, meaning we cannot lock a recipe on a fresh build
    * we can now generate the signatures for all available task of a given
      recipe
* Check if a given task is already locked. If so, don't duplicate
  entry in bblock.conf

Limitations:
* Needs to taint tasks that are locked, to display a warning
* I may be still missing some checks on user input
* Silently does nothing if given task doesn't exist
* Silently does nothing when resetting a recipe that doesn't exist

I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.

Improvement from V1:
* Signatures are now package architecture specific meaning that if you
  switch MACHINE, the lock sig will not be taken into account
* I added the -r option to unlock recipes
* I added a -d option to display the current bblock.conf
* Added an include directive for conf/bblock.conf inside bitbake.conf
* Added -t option to specify the tasks to lock/unlock

Limitations:
* I may be still missing some checks on user input
* I need to find a way to get the list of tasks ( by default still lock
  only the do_compile for now, unless -t is specified)
* Do not check if a particular recipe/task is already locked when trying
  to add lock. So entries may appear multiple times
* We still need the signature of the tasks to be already computed before
  locking. Need to find a way to generate it if missing

V3: https://lists.openembedded.org/g/openembedded-core/message/184932
V2: https://lists.openembedded.org/g/openembedded-core/message/184697
V1: https://lists.openembedded.org/g/openembedded-core/message/184584

My branch is available here [1]

Cheers
Julien

[1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
Julien Stephan (5):
  bitbake.conf: include bblock.conf
  bitbake: cooker: add a new function to retrieve task signatures
  sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
  scripts/bblock: add a script to lock/unlock recipes
  oeqa/selftest/bblock: add self test for bblock tool

 bitbake/lib/bb/command.py              |   6 +
 bitbake/lib/bb/cooker.py               |  16 +++
 bitbake/lib/bb/event.py                |   8 ++
 meta/conf/bitbake.conf                 |   1 +
 meta/lib/oe/sstatesig.py               |  13 +-
 meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
 scripts/bblock                         | 182 +++++++++++++++++++++++++
 7 files changed, 371 insertions(+), 1 deletion(-)  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
 create mode 100755 scripts/bblock

--
2.41.0

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Julien Stephan]

Le jeu. 3 août 2023 à 04:57, Chen, Qi <Qi.Chen@windriver.com> a écrit :
>
> When ' include conf/bblock.conf' line can be more flexibly added to/removed from local.conf, is there some specific reason why we should put it in bitbake.conf by default?
>
> Regards,
> Qi
>

Hi Qi,

This was a request from Richard on the first submission [1] :

> To simplify things, I think it would be reasonable to add a bblock.inc
> file unconditionally in our core configuration so that it is always
> included if present, just to make the tool easier since I think this
> will be a common use.

Cheers
Julien

[1): https://lists.openembedded.org/g/openembedded-core/message/184586

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Julien Stephan
> Sent: Wednesday, August 2, 2023 10:24 PM
> To: openembedded-core@lists.openembedded.org
> Cc: Julien Stephan <jstephan@baylibre.com>
> Subject: [OE-core] [PATCH v4 0/5] Add bblock helper scripts
>
> Hi all,
>
> This is v4 for bblock script.
>
> Improvement from v3:
> * Add self test
> * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
>   display a Note when recipe contains locked task(s)
>
> Limitations:
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
>
> Improvement from v2:
> * Add a function in bb.cooker to compute task signatures
> * Replace the findSigInfo function by the new created one. This has the
>   following advantages:
>     * findSigInfo needs the task to be already built to get the siginfo
>       file, meaning we cannot lock a recipe on a fresh build
>     * we can now generate the signatures for all available task of a given
>       recipe
> * Check if a given task is already locked. If so, don't duplicate
>   entry in bblock.conf
>
> Limitations:
> * Needs to taint tasks that are locked, to display a warning
> * I may be still missing some checks on user input
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
>
> I did some tests using qemux86-64 and qemuarm but I may be missing some corner cases.
>
> Improvement from V1:
> * Signatures are now package architecture specific meaning that if you
>   switch MACHINE, the lock sig will not be taken into account
> * I added the -r option to unlock recipes
> * I added a -d option to display the current bblock.conf
> * Added an include directive for conf/bblock.conf inside bitbake.conf
> * Added -t option to specify the tasks to lock/unlock
>
> Limitations:
> * I may be still missing some checks on user input
> * I need to find a way to get the list of tasks ( by default still lock
>   only the do_compile for now, unless -t is specified)
> * Do not check if a particular recipe/task is already locked when trying
>   to add lock. So entries may appear multiple times
> * We still need the signature of the tasks to be already computed before
>   locking. Need to find a way to generate it if missing
>
> V3: https://lists.openembedded.org/g/openembedded-core/message/184932
> V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> V1: https://lists.openembedded.org/g/openembedded-core/message/184584
>
> My branch is available here [1]
>
> Cheers
> Julien
>
> [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> Julien Stephan (5):
>   bitbake.conf: include bblock.conf
>   bitbake: cooker: add a new function to retrieve task signatures
>   sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
>   scripts/bblock: add a script to lock/unlock recipes
>   oeqa/selftest/bblock: add self test for bblock tool
>
>  bitbake/lib/bb/command.py              |   6 +
>  bitbake/lib/bb/cooker.py               |  16 +++
>  bitbake/lib/bb/event.py                |   8 ++
>  meta/conf/bitbake.conf                 |   1 +
>  meta/lib/oe/sstatesig.py               |  13 +-
>  meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
>  scripts/bblock                         | 182 +++++++++++++++++++++++++
>  7 files changed, 371 insertions(+), 1 deletion(-)  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
>  create mode 100755 scripts/bblock
>
> --
> 2.41.0

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Alexandre Belloni]

Hello, this causes oe-selftest failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio

2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
Stderr:
2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
    self.assertNotIn(info_message, result.output)
  File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
    raise self.failureException(msg)
AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...



On 02/08/2023 16:24:27+0200, Julien Stephan wrote:
> Hi all,
> 
> This is v4 for bblock script.
> 
> Improvement from v3:
> * Add self test
> * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
>   display a Note when recipe contains locked task(s)
> 
> Limitations:
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
> 
> Improvement from v2:
> * Add a function in bb.cooker to compute task signatures
> * Replace the findSigInfo function by the new created one. This has the
>   following advantages:
>     * findSigInfo needs the task to be already built to get the siginfo
>       file, meaning we cannot lock a recipe on a fresh build
>     * we can now generate the signatures for all available task of a given
>       recipe
> * Check if a given task is already locked. If so, don't duplicate
>   entry in bblock.conf
> 
> Limitations:
> * Needs to taint tasks that are locked, to display a warning
> * I may be still missing some checks on user input
> * Silently does nothing if given task doesn't exist
> * Silently does nothing when resetting a recipe that doesn't exist
> 
> I did some tests using qemux86-64 and qemuarm but I may be missing some
> corner cases.
> 
> Improvement from V1:
> * Signatures are now package architecture specific meaning that if you
>   switch MACHINE, the lock sig will not be taken into account
> * I added the -r option to unlock recipes
> * I added a -d option to display the current bblock.conf
> * Added an include directive for conf/bblock.conf inside bitbake.conf
> * Added -t option to specify the tasks to lock/unlock
> 
> Limitations:
> * I may be still missing some checks on user input
> * I need to find a way to get the list of tasks ( by default still lock
>   only the do_compile for now, unless -t is specified)
> * Do not check if a particular recipe/task is already locked when trying
>   to add lock. So entries may appear multiple times
> * We still need the signature of the tasks to be already computed before
>   locking. Need to find a way to generate it if missing
> 
> V3: https://lists.openembedded.org/g/openembedded-core/message/184932
> V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> V1: https://lists.openembedded.org/g/openembedded-core/message/184584
> 
> My branch is available here [1]
> 
> Cheers
> Julien
> 
> [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> Julien Stephan (5):
>   bitbake.conf: include bblock.conf
>   bitbake: cooker: add a new function to retrieve task signatures
>   sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
>   scripts/bblock: add a script to lock/unlock recipes
>   oeqa/selftest/bblock: add self test for bblock tool
> 
>  bitbake/lib/bb/command.py              |   6 +
>  bitbake/lib/bb/cooker.py               |  16 +++
>  bitbake/lib/bb/event.py                |   8 ++
>  meta/conf/bitbake.conf                 |   1 +
>  meta/lib/oe/sstatesig.py               |  13 +-
>  meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
>  scripts/bblock                         | 182 +++++++++++++++++++++++++
>  7 files changed, 371 insertions(+), 1 deletion(-)
>  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
>  create mode 100755 scripts/bblock
> 
> --
> 2.41.0

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
> Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Alexandre Belloni]

On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote:
> Hello, this causes oe-selftest failures on the autobuilders:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
> 
> 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
> 2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
> Stderr:
> 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
> 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
> 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
> 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
>   File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
>     self.assertNotIn(info_message, result.output)
>   File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
>     self.fail(self._formatMessage(msg, standardMsg))
>   File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
>     raise self.failureException(msg)
> AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
> 

Hum, wait, I got this without the patch series. Let me investigate.

> 
> 
> On 02/08/2023 16:24:27+0200, Julien Stephan wrote:
> > Hi all,
> > 
> > This is v4 for bblock script.
> > 
> > Improvement from v3:
> > * Add self test
> > * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
> >   display a Note when recipe contains locked task(s)
> > 
> > Limitations:
> > * Silently does nothing if given task doesn't exist
> > * Silently does nothing when resetting a recipe that doesn't exist
> > 
> > Improvement from v2:
> > * Add a function in bb.cooker to compute task signatures
> > * Replace the findSigInfo function by the new created one. This has the
> >   following advantages:
> >     * findSigInfo needs the task to be already built to get the siginfo
> >       file, meaning we cannot lock a recipe on a fresh build
> >     * we can now generate the signatures for all available task of a given
> >       recipe
> > * Check if a given task is already locked. If so, don't duplicate
> >   entry in bblock.conf
> > 
> > Limitations:
> > * Needs to taint tasks that are locked, to display a warning
> > * I may be still missing some checks on user input
> > * Silently does nothing if given task doesn't exist
> > * Silently does nothing when resetting a recipe that doesn't exist
> > 
> > I did some tests using qemux86-64 and qemuarm but I may be missing some
> > corner cases.
> > 
> > Improvement from V1:
> > * Signatures are now package architecture specific meaning that if you
> >   switch MACHINE, the lock sig will not be taken into account
> > * I added the -r option to unlock recipes
> > * I added a -d option to display the current bblock.conf
> > * Added an include directive for conf/bblock.conf inside bitbake.conf
> > * Added -t option to specify the tasks to lock/unlock
> > 
> > Limitations:
> > * I may be still missing some checks on user input
> > * I need to find a way to get the list of tasks ( by default still lock
> >   only the do_compile for now, unless -t is specified)
> > * Do not check if a particular recipe/task is already locked when trying
> >   to add lock. So entries may appear multiple times
> > * We still need the signature of the tasks to be already computed before
> >   locking. Need to find a way to generate it if missing
> > 
> > V3: https://lists.openembedded.org/g/openembedded-core/message/184932
> > V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> > V1: https://lists.openembedded.org/g/openembedded-core/message/184584
> > 
> > My branch is available here [1]
> > 
> > Cheers
> > Julien
> > 
> > [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> > Julien Stephan (5):
> >   bitbake.conf: include bblock.conf
> >   bitbake: cooker: add a new function to retrieve task signatures
> >   sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
> >   scripts/bblock: add a script to lock/unlock recipes
> >   oeqa/selftest/bblock: add self test for bblock tool
> > 
> >  bitbake/lib/bb/command.py              |   6 +
> >  bitbake/lib/bb/cooker.py               |  16 +++
> >  bitbake/lib/bb/event.py                |   8 ++
> >  meta/conf/bitbake.conf                 |   1 +
> >  meta/lib/oe/sstatesig.py               |  13 +-
> >  meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
> >  scripts/bblock                         | 182 +++++++++++++++++++++++++
> >  7 files changed, 371 insertions(+), 1 deletion(-)
> >  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
> >  create mode 100755 scripts/bblock
> > 
> > --
> > 2.41.0
> 
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
> > Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> > 
> 
> 
> -- 
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Alexandre Belloni]

On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote:
> On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote:
> > Hello, this causes oe-selftest failures on the autobuilders:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
> > 
> > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
> > 2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
> > Stderr:
> > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
> > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
> > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
> > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
> >   File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
> >     self.assertNotIn(info_message, result.output)
> >   File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
> >     self.fail(self._formatMessage(msg, standardMsg))
> >   File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
> >     raise self.failureException(msg)
> > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
> > 
> 
> Hum, wait, I got this without the patch series. Let me investigate.

Actually, I still had the series :)

> 
> > 
> > 
> > On 02/08/2023 16:24:27+0200, Julien Stephan wrote:
> > > Hi all,
> > > 
> > > This is v4 for bblock script.
> > > 
> > > Improvement from v3:
> > > * Add self test
> > > * Add a new "info" level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK: this allows to
> > >   display a Note when recipe contains locked task(s)
> > > 
> > > Limitations:
> > > * Silently does nothing if given task doesn't exist
> > > * Silently does nothing when resetting a recipe that doesn't exist
> > > 
> > > Improvement from v2:
> > > * Add a function in bb.cooker to compute task signatures
> > > * Replace the findSigInfo function by the new created one. This has the
> > >   following advantages:
> > >     * findSigInfo needs the task to be already built to get the siginfo
> > >       file, meaning we cannot lock a recipe on a fresh build
> > >     * we can now generate the signatures for all available task of a given
> > >       recipe
> > > * Check if a given task is already locked. If so, don't duplicate
> > >   entry in bblock.conf
> > > 
> > > Limitations:
> > > * Needs to taint tasks that are locked, to display a warning
> > > * I may be still missing some checks on user input
> > > * Silently does nothing if given task doesn't exist
> > > * Silently does nothing when resetting a recipe that doesn't exist
> > > 
> > > I did some tests using qemux86-64 and qemuarm but I may be missing some
> > > corner cases.
> > > 
> > > Improvement from V1:
> > > * Signatures are now package architecture specific meaning that if you
> > >   switch MACHINE, the lock sig will not be taken into account
> > > * I added the -r option to unlock recipes
> > > * I added a -d option to display the current bblock.conf
> > > * Added an include directive for conf/bblock.conf inside bitbake.conf
> > > * Added -t option to specify the tasks to lock/unlock
> > > 
> > > Limitations:
> > > * I may be still missing some checks on user input
> > > * I need to find a way to get the list of tasks ( by default still lock
> > >   only the do_compile for now, unless -t is specified)
> > > * Do not check if a particular recipe/task is already locked when trying
> > >   to add lock. So entries may appear multiple times
> > > * We still need the signature of the tasks to be already computed before
> > >   locking. Need to find a way to generate it if missing
> > > 
> > > V3: https://lists.openembedded.org/g/openembedded-core/message/184932
> > > V2: https://lists.openembedded.org/g/openembedded-core/message/184697
> > > V1: https://lists.openembedded.org/g/openembedded-core/message/184584
> > > 
> > > My branch is available here [1]
> > > 
> > > Cheers
> > > Julien
> > > 
> > > [1]: https://git.yoctoproject.org/poky-contrib/commit/?h=jstephan/bblock
> > > Julien Stephan (5):
> > >   bitbake.conf: include bblock.conf
> > >   bitbake: cooker: add a new function to retrieve task signatures
> > >   sstatesig: add a new info level for SIGGEN_LOCKEDSIGS_TASKSIG_CHECK
> > >   scripts/bblock: add a script to lock/unlock recipes
> > >   oeqa/selftest/bblock: add self test for bblock tool
> > > 
> > >  bitbake/lib/bb/command.py              |   6 +
> > >  bitbake/lib/bb/cooker.py               |  16 +++
> > >  bitbake/lib/bb/event.py                |   8 ++
> > >  meta/conf/bitbake.conf                 |   1 +
> > >  meta/lib/oe/sstatesig.py               |  13 +-
> > >  meta/lib/oeqa/selftest/cases/bblock.py | 146 ++++++++++++++++++++
> > >  scripts/bblock                         | 182 +++++++++++++++++++++++++
> > >  7 files changed, 371 insertions(+), 1 deletion(-)
> > >  create mode 100644 meta/lib/oeqa/selftest/cases/bblock.py
> > >  create mode 100755 scripts/bblock
> > > 
> > > --
> > > 2.41.0
> > 
> > > 
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#185403): https://lists.openembedded.org/g/openembedded-core/message/185403
> > > Mute This Topic: https://lists.openembedded.org/mt/100506390/3617179
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > 
> > 
> > 
> > -- 
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com
> 
> -- 
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Julien Stephan]

Hi Alexandre,
sorry for the late reply.

I was not able to reproduce this issue on my local setup, but I did
find another (similar??)  bug, I fixed it, so hopefully it will also
fix the one detected by the autobuilder.
I updated my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
(with this fix and a few other ones), can you try it on autobuilder? I
didn't send a newer version of the series in case the issue is still
there.

Please let me know if I can do anything else
Cheers
Julien

Le ven. 11 août 2023 à 10:45, Alexandre Belloni
<alexandre.belloni@bootlin.com> a écrit :
>
> On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote:
> > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote:
> > > Hello, this causes oe-selftest failures on the autobuilders:
> > >
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
> > >
> > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
> > > 2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
> > > Stderr:
> > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
> > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
> > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
> > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
> > >   File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
> > >     self.assertNotIn(info_message, result.output)
> > >   File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
> > >     self.fail(self._formatMessage(msg, standardMsg))
> > >   File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
> > >     raise self.failureException(msg)
> > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
> > >
> >
> > Hum, wait, I got this without the patch series. Let me investigate.
>
> Actually, I still had the series :)
>

Hi Alexandre,

I am not able to reproduce it locally, so it is difficult to fix the issue.



> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Alexandre Belloni]

Hello,

On 20/09/2023 16:20:42+0200, Julien Stephan wrote:
> Hi Alexandre,
> sorry for the late reply.
> 
> I was not able to reproduce this issue on my local setup, but I did
> find another (similar??)  bug, I fixed it, so hopefully it will also
> fix the one detected by the autobuilder.
> I updated my branch on poky-contrib
> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
> (with this fix and a few other ones), can you try it on autobuilder? I
> didn't send a newer version of the series in case the issue is still
> there.

This was sent to the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio

> 
> Please let me know if I can do anything else
> Cheers
> Julien
> 
> Le ven. 11 ao�t 2023 � 10:45, Alexandre Belloni
> <alexandre.belloni@bootlin.com> a �crit :
> >
> > On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote:
> > > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote:
> > > > Hello, this causes oe-selftest failures on the autobuilders:
> > > >
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
> > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
> > > >
> > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
> > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
> > > > Stderr:
> > > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
> > > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
> > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
> > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
> > > >   File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
> > > >     self.assertNotIn(info_message, result.output)
> > > >   File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
> > > >     self.fail(self._formatMessage(msg, standardMsg))
> > > >   File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
> > > >     raise self.failureException(msg)
> > > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
> > > >
> > >
> > > Hum, wait, I got this without the patch series. Let me investigate.
> >
> > Actually, I still had the series :)
> >
> 
> Hi Alexandre,
> 
> I am not able to reproduce it locally, so it is difficult to fix the issue.
> 
> 
> 
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Julien Stephan]

Le jeu. 21 sept. 2023 à 15:11, Alexandre Belloni
<alexandre.belloni@bootlin.com> a écrit :
>
> Hello,
>
> On 20/09/2023 16:20:42+0200, Julien Stephan wrote:
> > Hi Alexandre,
> > sorry for the late reply.
> >
> > I was not able to reproduce this issue on my local setup, but I did
> > find another (similar??)  bug, I fixed it, so hopefully it will also
> > fix the one detected by the autobuilder.
> > I updated my branch on poky-contrib
> > (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
> > (with this fix and a few other ones), can you try it on autobuilder? I
> > didn't send a newer version of the series in case the issue is still
> > there.
>
> This was sent to the autobuilders:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio
>

Hi Alexandre,

Thank you! Unfortunately I can see that it is still failing on the same test :(
I may need your help here.

Here is what my test is supposed to do:
From the autobuilder log I can see that the machine is set to qemux86-64 so :

$ bblock -d # unlock all recipes and remove build/conf/bblock.conf file
$ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for
qemuarm arch so block.conf file should look like this:

# Generated by bblock
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"

SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>"

# modify the quilt's do_compile task by adding an empty
do_compile:append function
$ bitbake -n quilt # for qemux86-64 machine

Because quilt is supposed to be locked for qemuarm, I am not supposed
to see any warning, but the log on the autobuilder says:
"WARNING: The quilt:do_compile sig is computed to be
d5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but
the sig is locked to
c9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fe in
SIGGEN_LOCKEDSIGS_core2-64\""

meaning bblock.conf contains:

SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig>


So I can see two possible reasons for that:
* MACHINE=qemuarm block quilt -t compile  is not using the machine I
give on the command line, but why??
* bblock.conf gets polluted by another test that also tries to lock
quilt's do_compile function for qemux86-64 because of parallelism  (I
was not able to reproduce it locally)

Do you have an idea why this test is failing?
My next move would be to place all the tests in a single test to
remove the parallelism issue (if that is the issue) but I would prefer
to avoid wasting your time starting builds for me several times..

Also, I didn't manage to print log messages on my test, how can I do that?

Cheers
Julien
> >
> > Please let me know if I can do anything else
> > Cheers
> > Julien
> >
> > Le ven. 11 août 2023 à 10:45, Alexandre Belloni
> > <alexandre.belloni@bootlin.com> a écrit :
> > >
> > > On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote:
> > > > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote:
> > > > > Hello, this causes oe-selftest failures on the autobuilders:
> > > > >
> > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
> > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
> > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
> > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
> > > > >
> > > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
> > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
> > > > > Stderr:
> > > > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
> > > > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
> > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
> > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
> > > > >   File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
> > > > >     self.assertNotIn(info_message, result.output)
> > > > >   File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
> > > > >     self.fail(self._formatMessage(msg, standardMsg))
> > > > >   File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
> > > > >     raise self.failureException(msg)
> > > > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
> > > > >
> > > >
> > > > Hum, wait, I got this without the patch series. Let me investigate.
> > >
> > > Actually, I still had the series :)
> > >
> >
> > Hi Alexandre,
> >
> > I am not able to reproduce it locally, so it is difficult to fix the issue.
> >
> >
> >
> > > Alexandre Belloni, co-owner and COO, Bootlin
> > > Embedded Linux and Kernel engineering
> > > https://bootlin.com
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Alexandre Belloni]

On 21/09/2023 16:24:30+0200, Julien Stephan wrote:
> Le jeu. 21 sept. 2023 � 15:11, Alexandre Belloni
> <alexandre.belloni@bootlin.com> a �crit :
> >
> > Hello,
> >
> > On 20/09/2023 16:20:42+0200, Julien Stephan wrote:
> > > Hi Alexandre,
> > > sorry for the late reply.
> > >
> > > I was not able to reproduce this issue on my local setup, but I did
> > > find another (similar??)  bug, I fixed it, so hopefully it will also
> > > fix the one detected by the autobuilder.
> > > I updated my branch on poky-contrib
> > > (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock)
> > > (with this fix and a few other ones), can you try it on autobuilder? I
> > > didn't send a newer version of the series in case the issue is still
> > > there.
> >
> > This was sent to the autobuilders:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/14/logs/stdio
> >
> 
> Hi Alexandre,
> 
> Thank you! Unfortunately I can see that it is still failing on the same test :(
> I may need your help here.
> 
> Here is what my test is supposed to do:
> From the autobuilder log I can see that the machine is set to qemux86-64 so :
> 
> $ bblock -d # unlock all recipes and remove build/conf/bblock.conf file
> $ MACHINE=qemuarm block quilt -t compile # lock do_compile for bc for
> qemuarm arch so block.conf file should look like this:
> 
> # Generated by bblock
> SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
> SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"
> 
> SIGGEN_LOCKEDSIGS_cortexa15t2hf-neon += "quilt:do_compile:<sig>"
> 
> # modify the quilt's do_compile task by adding an empty
> do_compile:append function
> $ bitbake -n quilt # for qemux86-64 machine
> 
> Because quilt is supposed to be locked for qemuarm, I am not supposed
> to see any warning, but the log on the autobuilder says:
> "WARNING: The quilt:do_compile sig is computed to be
> d5902ba1813663fcd92fc5afb9026e01c01ccdbcc0ff792aad555522ce407c31, but
> the sig is locked to
> c9c5724b6f125763f1f0e2c87e6451acdaf59f7aba7efe387c2908d14486f3fe in
> SIGGEN_LOCKEDSIGS_core2-64\""
> 
> meaning bblock.conf contains:
> 
> SIGGEN_LOCKEDSIGS_core2-64 += "quilt:do_compile:<sig>
> 
> 
> So I can see two possible reasons for that:
> * MACHINE=qemuarm block quilt -t compile  is not using the machine I
> give on the command line, but why??

As I said in the call, look at this log from the build:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio

It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is
then used by the build so overriding it from the command line doesn't
work.

> * bblock.conf gets polluted by another test that also tries to lock
> quilt's do_compile function for qemux86-64 because of parallelism  (I
> was not able to reproduce it locally)
> 
> Do you have an idea why this test is failing?
> My next move would be to place all the tests in a single test to
> remove the parallelism issue (if that is the issue) but I would prefer
> to avoid wasting your time starting builds for me several times..
> 
> Also, I didn't manage to print log messages on my test, how can I do that?
> 
> Cheers
> Julien
> > >
> > > Please let me know if I can do anything else
> > > Cheers
> > > Julien
> > >
> > > Le ven. 11 ao�t 2023 � 10:45, Alexandre Belloni
> > > <alexandre.belloni@bootlin.com> a �crit :
> > > >
> > > > On 11/08/2023 10:00:33+0200, Alexandre Belloni wrote:
> > > > > On 10/08/2023 15:21:14+0200, Alexandre Belloni wrote:
> > > > > > Hello, this causes oe-selftest failures on the autobuilders:
> > > > > >
> > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5589/steps/14/logs/stdio
> > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5577/steps/14/logs/stdio
> > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5536/steps/14/logs/stdio
> > > > > > https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5613/steps/14/logs/stdio
> > > > > >
> > > > > > 2023-08-10 09:44:34,071 - oe-selftest - INFO - bblock.BBLock.test_lock_architecture_specific (subunit.RemotedTestCase)
> > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO -  ... FAIL
> > > > > > Stderr:
> > > > > > 2023-08-10 09:40:23,454 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-2290072/conf/local.conf
> > > > > > 2023-08-10 09:40:23,455 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
> > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - 3: 1/34 29/533 (250.62s) (0 failed) (bblock.BBLock.test_lock_architecture_specific)
> > > > > > 2023-08-10 09:44:34,072 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
> > > > > >   File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bblock.py", line 103, in test_lock_architecture_specific
> > > > > >     self.assertNotIn(info_message, result.output)
> > > > > >   File "/usr/lib64/python3.9/unittest/case.py", line 1111, in assertNotIn
> > > > > >     self.fail(self._formatMessage(msg, standardMsg))
> > > > > >   File "/usr/lib64/python3.9/unittest/case.py", line 676, in fail
> > > > > >     raise self.failureException(msg)
> > > > > > AssertionError: 'NOTE: The following recipes have locked tasks: quilt' unexpectedly found in 'NOTE: Reconnecting to bitbake server...
> > > > > >
> > > > >
> > > > > Hum, wait, I got this without the patch series. Let me investigate.
> > > >
> > > > Actually, I still had the series :)
> > > >
> > >
> > > Hi Alexandre,
> > >
> > > I am not able to reproduce it locally, so it is difficult to fix the issue.
> > >
> > >
> > >
> > > > Alexandre Belloni, co-owner and COO, Bootlin
> > > > Embedded Linux and Kernel engineering
> > > > https://bootlin.com
> >
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Julien Stephan]

> >
> > So I can see two possible reasons for that:
> > * MACHINE=qemuarm block quilt -t compile  is not using the machine I
> > give on the command line, but why??
>
> As I said in the call, look at this log from the build:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio
>
> It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is
> then used by the build so overriding it from the command line doesn't
> work.
>

Hi Alexandre,

Thank you, I managed to reproduce it locally and fixed it properly. I
just force pushed my poky-contrib branch
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can
you start a build? If successful, I'll send the v5 today.

Cheers
Julien

Re: [OE-core] [PATCH v4 0/5] Add bblock helper scripts

[Alexandre Belloni]

This is started:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5792

On 22/09/2023 10:54:56+0200, Julien Stephan wrote:
> > >
> > > So I can see two possible reasons for that:
> > > * MACHINE=qemuarm block quilt -t compile  is not using the machine I
> > > give on the command line, but why??
> >
> > As I said in the call, look at this log from the build:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5785/steps/13/logs/stdio
> >
> > It tells you that MACHINE = "qemux86-64" ends up in auto.conf which is
> > then used by the build so overriding it from the command line doesn't
> > work.
> >
> 
> Hi Alexandre,
> 
> Thank you, I managed to reproduce it locally and fixed it properly. I
> just force pushed my poky-contrib branch
> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock) can
> you start a build? If successful, I'll send the v5 today.
> 
> Cheers
> Julien

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com