[PATCH 0/3] toaster-related patchset

[PATCH 0/3] toaster-related patchset

[Alex DAMIAN]

From: Alexandru DAMIAN <alexandru.damian@intel.com>

This is a rebased patchset consisting of patches that have been 
previously submitted to the oe-core mail list separately, but not yet merged.

In order to make merging easier, I'm resubmitting as a new patchset.



The following changes since commit e8476a9d19dba7a1a6c5a558a89f2c9f89d39ae2:

  gettext: upgrade from 0.18.3.1 to 0.18.3.2 (2014-03-17 14:01:43 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib adamian/submission-17032014
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=adamian/submission-17032014

Alexandru DAMIAN (1):
  sstate: list missing files for toaster

Cristiana Voicu (1):
  toaster.bbclass: read the data needed for license manifest path

Marius Avram (1):
  buildstats: use TaskBase time for elapsed time

 meta/classes/buildstats.bbclass | 23 +++++++++++++++--------
 meta/classes/sstate.bbclass     | 11 +++++++++++
 meta/classes/toaster.bbclass    | 12 ++++++++++++
 3 files changed, 38 insertions(+), 8 deletions(-)

-- 
1.8.3.2

[PATCH 1/3] toaster.bbclass: read the data needed for license manifest path

[Alex DAMIAN]

From: Cristiana Voicu <cristiana.voicu@intel.com>

The license.manifest file is located in DEPLOY_DIR_IMAGE/licenses/
IMAGE_NAME dir. The data needed is collected after rootfs task.

[YOCTO #5649]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
---
 meta/classes/toaster.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
index ddfceb5..563b1f7 100644
--- a/meta/classes/toaster.bbclass
+++ b/meta/classes/toaster.bbclass
@@ -296,6 +296,17 @@ python toaster_buildhistory_dump() {
 
 }
 
+# dump information related to license manifest path
+
+python toaster_licensemanifest_dump() {
+    deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE', True);
+    image_name = d.getVar('IMAGE_NAME', True);
+
+    data = { 'deploy_dir_image' : deploy_dir_image, 'image_name' : image_name }
+
+    bb.event.fire(bb.event.MetadataEvent("LicenseManifestPath", data), d)
+}
+
 # set event handlers
 addhandler toaster_layerinfo_dumpdata
 toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -308,3 +319,4 @@ toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 do_package[postfuncs] += "toaster_package_dumpdata "
 
 do_rootfs[postfuncs] += "toaster_image_dumpdata "
+do_rootfs[postfuncs] += "toaster_licensemanifest_dump "
-- 
1.8.3.2

[PATCH 2/3] sstate: list missing files for toaster

[Alex DAMIAN]

From: Alexandru DAMIAN <alexandru.damian@intel.com>

Toaster needs to record the attempts to restore
setscene tasks that don't have a sstate file.

We build a list of tasks for which we can't find an
sstate file, and if we're running under Toaster data
collection, we send it off with a MetadataEvent.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
---
 meta/classes/sstate.bbclass | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 03ee820..25b8d72 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -625,6 +625,7 @@ BB_HASHCHECK_FUNCTION = "sstate_checkhashes"
 def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
 
     ret = []
+    missed = []
 
     def getpathcomponents(task, d):
         # Magic data from BB_HASHFILENAME
@@ -646,11 +647,13 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
         spec, extrapath, tname = getpathcomponents(task, d)
 
         sstatefile = d.expand("${SSTATE_DIR}/" + extrapath + generate_sstatefn(spec, sq_hash[task], d) + "_" + tname + ".tgz.siginfo")
+
         if os.path.exists(sstatefile):
             bb.debug(2, "SState: Found valid sstate file %s" % sstatefile)
             ret.append(task)
             continue
         else:
+            missed.append(task)
             bb.debug(2, "SState: Looked for but didn't find file %s" % sstatefile)
 
     mirrors = d.getVar("SSTATE_MIRRORS", True)
@@ -688,9 +691,17 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
                 bb.debug(2, "SState: Successful fetch test for %s" % srcuri)
                 ret.append(task)
             except:
+                missed.append(task)
                 bb.debug(2, "SState: Unsuccessful fetch test for %s" % srcuri)
                 pass     
 
+    inheritlist = d.getVar("INHERIT", True)
+    if "toaster" in inheritlist:
+        evdata = []
+        for task in missed:
+            evdata.append( (sq_fn[task], sq_task[task], sq_hash[task], generate_sstatefn(spec, sq_hash[task],d) ) )
+        bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"
-- 
1.8.3.2

[PATCH 3/3] buildstats: use TaskBase time for elapsed time

[Alex DAMIAN]

From: Marius Avram <marius.avram@intel.com>

To avoid any further inconsistencies between buildstats and buildinfohelper
from toaster, buildstats will measure task duration using the time field
from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.

Signed-off-by: Marius Avram <marius.avram@intel.com>
---
 meta/classes/buildstats.bbclass | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
index 72fff11..89ae72c 100644
--- a/meta/classes/buildstats.bbclass
+++ b/meta/classes/buildstats.bbclass
@@ -109,14 +109,17 @@ def get_diskdata(var, dev, data):
         diskdata["End"+key] = str(int(newdiskdata[key]))    
     return diskdata
     
-def set_timedata(var, data):
+def set_timedata(var, data, server_time=None):
     import time
-    time = time.time()
+    if server_time:
+        time = server_time
+    else:
+        time = time.time()
     cputime = get_cputime()
     proctime = get_process_cputime(os.getpid())
     data.setVar(var, (time, cputime, proctime))
 
-def get_timedata(var, data):
+def get_timedata(var, data, server_time=None):
     import time
     timedata = data.getVar(var, False)
     if timedata is None:
@@ -124,7 +127,11 @@ def get_timedata(var, data):
     oldtime, oldcpu, oldproc = timedata
     procdiff = get_process_cputime(os.getpid()) - oldproc
     cpudiff = get_cputime() - oldcpu
-    timediff = time.time() - oldtime
+    if server_time:
+        end_time = server_time
+    else:
+        end_time = time.time()
+    timediff = end_time - oldtime
     if cpudiff > 0:
         cpuperc = float(procdiff) * 100 / cpudiff
     else:
@@ -136,7 +143,7 @@ def write_task_data(status, logfile, dev, e):
     bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn)
     taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
     file = open(os.path.join(logfile), "a")
-    timedata = get_timedata("__timedata_task", e.data)
+    timedata = get_timedata("__timedata_task", e.data, e.time)
     if timedata:
         elapsedtime, cpu = timedata
         file.write(bb.data.expand("${PF}: %s: Elapsed time: %0.2f seconds \n" %
@@ -160,7 +167,7 @@ def write_task_data(status, logfile, dev, e):
 	    file.write("Status: PASSED \n")
     else:
         file.write("Status: FAILED \n")
-    file.write("Ended: %0.2f \n" % time.time())
+    file.write("Ended: %0.2f \n" % e.time)
     file.close()
 
 python run_buildstats () {
@@ -234,7 +241,7 @@ python run_buildstats () {
         taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
         if device != "NoLogicalDevice":
             set_diskdata("__diskdata_task", device, e.data)
-        set_timedata("__timedata_task", e.data)
+        set_timedata("__timedata_task", e.data, e.time)
         try:
             bb.utils.mkdirhier(taskdir)
         except:
@@ -242,7 +249,7 @@ python run_buildstats () {
         # write into the task event file the name and start time
         file = open(os.path.join(taskdir, e.task), "a")
         file.write("Event: %s \n" % bb.event.getName(e))
-        file.write("Started: %0.2f \n" % time.time())
+        file.write("Started: %0.2f \n" % e.time)
         file.close()
 
     elif isinstance(e, bb.build.TaskSucceeded):
-- 
1.8.3.2

Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time

[Saul Wold]

On 03/17/2014 08:04 AM, Alex DAMIAN wrote:
> From: Marius Avram <marius.avram@intel.com>
>
> To avoid any further inconsistencies between buildstats and buildinfohelper
> from toaster, buildstats will measure task duration using the time field
> from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.
>

This patch seems to be causing some problems:

> ERROR: Execution of event handler 'run_buildstats' failed
> Traceback (most recent call last):
>   File "run_buildstats(e)", line 72, in run_buildstats(e=<bb.build.TaskStarted object at 0x6e45590>)
> AttributeError: 'TaskStarted' object has no attribute 'time'
>
> ERROR: Execution of event handler 'run_buildstats' failed
> Traceback (most recent call last):
>   File "run_buildstats(e)", line 88, in run_buildstats(e=<bb.build.TaskSucceeded object at 0x6e40190>)
>   File "buildstats.bbclass", line 5, in write_task_data(status='passed', logfile='/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_clean', dev='sda5', e=<bb.build.TaskSucceeded object at 0x6e40190>)
> IOError: [Errno 2] No such file or directory: '/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_clean'
>
> ERROR: Execution of event handler 'run_buildstats' failed
> Traceback (most recent call last):
>   File "run_buildstats(e)", line 72, in run_buildstats(e=<bb.build.TaskStarted object at 0x74f59d0>)
> AttributeError: 'TaskStarted' object has no attribute 'time'
>
> ERROR: Execution of event handler 'run_buildstats' failed
> Traceback (most recent call last):
>   File "run_buildstats(e)", line 88, in run_buildstats(e=<bb.build.TaskSucceeded object at 0x6e48dd0>)
>   File "buildstats.bbclass", line 5, in write_task_data(status='passed', logfile='/home/sgw/yocto/builds/world/tmp/buildstats/linux-yocto-qemux86-64/201403181525/linux-yocto-3.10.32+gitAUTOINC+7b3b87d4d5_78afd3095c-r0/do_cleansstate', dev='sda5', e=<bb.build.TaskSucceeded object at 0x6e48dd0>)



Sau!


> Signed-off-by: Marius Avram <marius.avram@intel.com>
> ---
>   meta/classes/buildstats.bbclass | 23 +++++++++++++++--------
>   1 file changed, 15 insertions(+), 8 deletions(-)
>
> diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
> index 72fff11..89ae72c 100644
> --- a/meta/classes/buildstats.bbclass
> +++ b/meta/classes/buildstats.bbclass
> @@ -109,14 +109,17 @@ def get_diskdata(var, dev, data):
>           diskdata["End"+key] = str(int(newdiskdata[key]))
>       return diskdata
>
> -def set_timedata(var, data):
> +def set_timedata(var, data, server_time=None):
>       import time
> -    time = time.time()
> +    if server_time:
> +        time = server_time
> +    else:
> +        time = time.time()
>       cputime = get_cputime()
>       proctime = get_process_cputime(os.getpid())
>       data.setVar(var, (time, cputime, proctime))
>
> -def get_timedata(var, data):
> +def get_timedata(var, data, server_time=None):
>       import time
>       timedata = data.getVar(var, False)
>       if timedata is None:
> @@ -124,7 +127,11 @@ def get_timedata(var, data):
>       oldtime, oldcpu, oldproc = timedata
>       procdiff = get_process_cputime(os.getpid()) - oldproc
>       cpudiff = get_cputime() - oldcpu
> -    timediff = time.time() - oldtime
> +    if server_time:
> +        end_time = server_time
> +    else:
> +        end_time = time.time()
> +    timediff = end_time - oldtime
>       if cpudiff > 0:
>           cpuperc = float(procdiff) * 100 / cpudiff
>       else:
> @@ -136,7 +143,7 @@ def write_task_data(status, logfile, dev, e):
>       bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn)
>       taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
>       file = open(os.path.join(logfile), "a")
> -    timedata = get_timedata("__timedata_task", e.data)
> +    timedata = get_timedata("__timedata_task", e.data, e.time)
>       if timedata:
>           elapsedtime, cpu = timedata
>           file.write(bb.data.expand("${PF}: %s: Elapsed time: %0.2f seconds \n" %
> @@ -160,7 +167,7 @@ def write_task_data(status, logfile, dev, e):
>   	    file.write("Status: PASSED \n")
>       else:
>           file.write("Status: FAILED \n")
> -    file.write("Ended: %0.2f \n" % time.time())
> +    file.write("Ended: %0.2f \n" % e.time)
>       file.close()
>
>   python run_buildstats () {
> @@ -234,7 +241,7 @@ python run_buildstats () {
>           taskdir = os.path.join(bsdir, e.data.expand("${PF}"))
>           if device != "NoLogicalDevice":
>               set_diskdata("__diskdata_task", device, e.data)
> -        set_timedata("__timedata_task", e.data)
> +        set_timedata("__timedata_task", e.data, e.time)
>           try:
>               bb.utils.mkdirhier(taskdir)
>           except:
> @@ -242,7 +249,7 @@ python run_buildstats () {
>           # write into the task event file the name and start time
>           file = open(os.path.join(taskdir, e.task), "a")
>           file.write("Event: %s \n" % bb.event.getName(e))
> -        file.write("Started: %0.2f \n" % time.time())
> +        file.write("Started: %0.2f \n" % e.time)
>           file.close()
>
>       elif isinstance(e, bb.build.TaskSucceeded):
>

Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time

[Richard Purdie]

On Tue, 2014-03-18 at 15:28 -0700, Saul Wold wrote:
> On 03/17/2014 08:04 AM, Alex DAMIAN wrote:
> > From: Marius Avram <marius.avram@intel.com>
> >
> > To avoid any further inconsistencies between buildstats and buildinfohelper
> > from toaster, buildstats will measure task duration using the time field
> > from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed.
> >
> 
> This patch seems to be causing some problems:

I suspect this is dependent on a bitbake change...

Cheers,

Richard

Re: [PATCH 3/3] buildstats: use TaskBase time for elapsed time

[Damian, Alexandru]

[-- Attachment #1: Type: text/plain, Size: 785 bytes --]

Yep, there is a counterpart change submitted to bitbake.

Alex


On Tue, Mar 18, 2014 at 11:15 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2014-03-18 at 15:28 -0700, Saul Wold wrote:
> > On 03/17/2014 08:04 AM, Alex DAMIAN wrote:
> > > From: Marius Avram <marius.avram@intel.com>
> > >
> > > To avoid any further inconsistencies between buildstats and
> buildinfohelper
> > > from toaster, buildstats will measure task duration using the time
> field
> > > from within the TaskBase events: TaskStarted and
> TaskSucceeded/TaskFailed.
> > >
> >
> > This patch seems to be causing some problems:
>
> I suspect this is dependent on a bitbake change...
>
> Cheers,
>
> Richard
>
>


-- 
Alex Damian
Yocto Project
SSG / OTC

[-- Attachment #2: Type: text/html, Size: 1381 bytes --]