[PATCH 1/1] sstate: list missing files for toaster

[PATCH 1/1] sstate: list missing files for toaster

[Alex DAMIAN]

From: Alexandru DAMIAN <alexandru.damian@intel.com>

Toaster needs to record the attempts to restore
setscene tasks that don't have a sstate file.

We build a list of tasks for which we can't find an
sstate file, and if we're running under Toaster data
collection, we send it off with a MetadataEvent.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
 meta/classes/sstate.bbclass | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index f7bd117..f70c62f 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -621,6 +621,7 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     ret = []
+    missed = []
 
     def getpathcomponents(task, d):
         # Magic data from BB_HASHFILENAME
@@ -642,11 +643,13 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
         spec, extrapath, tname = getpathcomponents(task, d)
 
         sstatefile = d.expand("${SSTATE_DIR}/" + extrapath + generate_sstatefn(spec, sq_hash[task], d) + "_" + tname + ".tgz.siginfo")
+
         if os.path.exists(sstatefile):
             bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
             ret.append(task)
             continue
         else:
+            missed.append(task)
             bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile)
 
     mirrors = d.getVar("SSTATE_MIRRORS", True)
@@ -684,9 +687,17 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
                 bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
                 ret.append(task)
             except:
+                missed.append(task)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri)
                 pass     
 
+    inheritlist = d.getVar("INHERIT", True)
+    if "toaster" in inheritlist:
+        evdata = []
+        for task in missed:
+            evdata.append( (sq_fn[task], sq_task[task], sq_hash[task], generate_sstatefn(spec, sq_hash[task],d) ) )
+        bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"
-- 
1.8.3.2

Re: [PATCH 1/1] sstate: list missing files for toaster

[Richard Purdie]

On Mon, 2014-02-17 at 17:33 +0000, Alex DAMIAN wrote:
> From: Alexandru DAMIAN <alexandru.damian@intel.com>
> 
> Toaster needs to record the attempts to restore
> setscene tasks that don't have a sstate file.
> 
> We build a list of tasks for which we can't find an
> sstate file, and if we're running under Toaster data
> collection, we send it off with a MetadataEvent.
> 
> Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
> ---
>  meta/classes/sstate.bbclass | 11 +++++++++++
>  1 file changed, 11 insertions(+)

I'm wondering if we should perhaps do this in bitbake and add a setscene
"missed" event type there based upon the return value of this function?

Ideally I'd like to be able to report the stats in knotty too.

Cheers,

Richard

Re: [PATCH 1/1] sstate: list missing files for toaster

[Damian, Alexandru]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

I would think against it, because

- it's very inefficient to send a Skipped event for each missed task than
to collect all the data and send it in one shot
- it's faster to send the event from the sstate.bbclass than to return the
task ids and then call again a python function to generate the sstate file
names - basically replicating the sstate.bbclass code in bitbake
- the queueTaskSkipped event has a different semantic than what we need
here to return a whole batch of data; creating a new event type from
scratch is a bit pointless since the inheritance would need to be broken
- knotty can receive MetadataEvents just as well as sceneTaskSkipped event,
or any variation of it

I would suggest actually just sending this event at all times, not just
when toaster is enabled.

Please advise,
Alex


On Thu, Feb 20, 2014 at 2:30 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2014-02-17 at 17:33 +0000, Alex DAMIAN wrote:
> > From: Alexandru DAMIAN <alexandru.damian@intel.com>
> >
> > Toaster needs to record the attempts to restore
> > setscene tasks that don't have a sstate file.
> >
> > We build a list of tasks for which we can't find an
> > sstate file, and if we're running under Toaster data
> > collection, we send it off with a MetadataEvent.
> >
> > Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
> > ---
> >  meta/classes/sstate.bbclass | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
>
> I'm wondering if we should perhaps do this in bitbake and add a setscene
> "missed" event type there based upon the return value of this function?
>
> Ideally I'd like to be able to report the stats in knotty too.
>
> Cheers,
>
> Richard
>
>


-- 
Alex Damian
Yocto Project
SSG / OTC

[-- Attachment #2: Type: text/html, Size: 3225 bytes --]

Re: [PATCH 1/1] sstate: list missing files for toaster

[Richard Purdie]

On Thu, 2014-02-20 at 15:41 +0000, Damian, Alexandru wrote:
> I would think against it, because

> - it's very inefficient to send a Skipped event for each missed task
> than to collect all the data and send it in one shot
> - it's faster to send the event from the sstate.bbclass than to return
> the task ids and then call again a python function to generate the
> sstate file names - basically replicating the sstate.bbclass code in
> bitbake
> - the queueTaskSkipped event has a different semantic than what we
> need here to return a whole batch of data; creating a new event type
> from scratch is a bit pointless since the inheritance would need to be
> broken
> - knotty can receive MetadataEvents just as well as sceneTaskSkipped
> event, or any variation of it

I guess I'd like to understand which information from this even you're
particularly interested in?

Do I understand correctly that you want the actual filenames attempted?

Cheers,

Richard

Re: [PATCH 1/1] sstate: list missing files for toaster

[Damian, Alexandru]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

Yep, tasks that were looked up, and actual filenames attempted.

Alex


On Thu, Feb 20, 2014 at 4:06 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2014-02-20 at 15:41 +0000, Damian, Alexandru wrote:
> > I would think against it, because
>
> > - it's very inefficient to send a Skipped event for each missed task
> > than to collect all the data and send it in one shot
> > - it's faster to send the event from the sstate.bbclass than to return
> > the task ids and then call again a python function to generate the
> > sstate file names - basically replicating the sstate.bbclass code in
> > bitbake
> > - the queueTaskSkipped event has a different semantic than what we
> > need here to return a whole batch of data; creating a new event type
> > from scratch is a bit pointless since the inheritance would need to be
> > broken
> > - knotty can receive MetadataEvents just as well as sceneTaskSkipped
> > event, or any variation of it
>
> I guess I'd like to understand which information from this even you're
> particularly interested in?
>
> Do I understand correctly that you want the actual filenames attempted?
>
> Cheers,
>
> Richard
>
>
>


-- 
Alex Damian
Yocto Project
SSG / OTC

[-- Attachment #2: Type: text/html, Size: 1920 bytes --]

Re: [PATCH 1/1] sstate: list missing files for toaster

[Alex Damian]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

Hello,

I am following up with this patch. If there are no other objects, can we
please merge it ?

Thanks,
Alex


On Thu, Feb 20, 2014 at 8:29 PM, Damian, Alexandru <
alexandru.damian@intel.com> wrote:

> Yep, tasks that were looked up, and actual filenames attempted.
>
> Alex
>
>
> On Thu, Feb 20, 2014 at 4:06 PM, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> On Thu, 2014-02-20 at 15:41 +0000, Damian, Alexandru wrote:
>> > I would think against it, because
>>
>> > - it's very inefficient to send a Skipped event for each missed task
>> > than to collect all the data and send it in one shot
>> > - it's faster to send the event from the sstate.bbclass than to return
>> > the task ids and then call again a python function to generate the
>> > sstate file names - basically replicating the sstate.bbclass code in
>> > bitbake
>> > - the queueTaskSkipped event has a different semantic than what we
>> > need here to return a whole batch of data; creating a new event type
>> > from scratch is a bit pointless since the inheritance would need to be
>> > broken
>> > - knotty can receive MetadataEvents just as well as sceneTaskSkipped
>> > event, or any variation of it
>>
>> I guess I'd like to understand which information from this even you're
>> particularly interested in?
>>
>> Do I understand correctly that you want the actual filenames attempted?
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>
>
> --
> Alex Damian
> Yocto Project
> SSG / OTC
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

[-- Attachment #2: Type: text/html, Size: 2909 bytes --]