[PATCH] buildstats-summary.bbclass: Import useful bbclass from meta-mentor

[PATCH] buildstats-summary.bbclass: Import useful bbclass from meta-mentor

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/buildstats-summary.bbclass | 43 +++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
 create mode 100644 meta/classes/buildstats-summary.bbclass

diff --git a/meta/classes/buildstats-summary.bbclass b/meta/classes/buildstats-summary.bbclass
new file mode 100644
index 0000000..3c02724
--- /dev/null
+++ b/meta/classes/buildstats-summary.bbclass
@@ -0,0 +1,43 @@
+# Imported from meta-mentor
+# http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor/log/classes/buildstats-summary.bbclass
+# with following modifications
+# 1) when both setscene and normal tasks were executed, count it as no_sstate
+# 2) count do_populate_sysroot and do_packagedata tasks, because often we invalidate only runtime deps
+
+python buildstats_summary () {
+    if not isinstance(e, bb.event.BuildCompleted):
+        return
+
+    import os.path
+
+    bn = get_bn(e)
+    bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn)
+    if not os.path.exists(bsdir):
+        return
+
+    sstate_ps, no_sstate_ps, sstate_pd, no_sstate_pd = set(), set(), set(), set()
+    for pf in os.listdir(bsdir):
+        taskdir = os.path.join(bsdir, pf)
+        if not os.path.isdir(taskdir):
+            continue
+
+        tasks = os.listdir(taskdir)
+        if 'do_populate_sysroot' in tasks:
+            no_sstate_ps.add(pf)
+        elif 'do_populate_sysroot_setscene' in tasks:
+            sstate_ps.add(pf)
+        if 'do_packagedata' in tasks:
+            no_sstate_pd.add(pf)
+        elif 'do_packagedata_setscene' in tasks:
+            sstate_pd.add(pf)
+
+    if not sstate_ps and not no_sstate_ps and not sstate_pd and not no_sstate_pd:
+        return
+
+    bb.note("Build completion summary:")
+    if (len(sstate_ps)+len(no_sstate_ps)) > 0:
+        bb.note("  populate_sysroot reused from sstate in %d%% (%d setscene, %d scratch)" % (100*len(sstate_ps)/(len(sstate_ps)+len(no_sstate_ps)), len(sstate_ps), len(no_sstate_ps)))
+    if (len(sstate_pd)+len(no_sstate_pd)) > 0:
+        bb.note("  packagedata reused from sstate in %d%% (%d setscene, %d scratch)" % (100*len(sstate_pd)/(len(sstate_pd)+len(no_sstate_pd)), len(sstate_ps), len(no_sstate_ps)))
+}
+addhandler buildstats_summary
-- 
2.0.0

Re: [PATCH] buildstats-summary.bbclass: Import useful bbclass from meta-mentor

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

On Fri, Jun 6, 2014 at 12:27 PM, Martin Jansa <martin.jansa@gmail.com>
wrote:

> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/classes/buildstats-summary.bbclass | 43
> +++++++++++++++++++++++++++++++++
>  1 file changed, 43 insertions(+)
>  create mode 100644 meta/classes/buildstats-summary.bbclass
>
> diff --git a/meta/classes/buildstats-summary.bbclass b/meta/classes/
> buildstats-summary.bbclass
> new file mode 100644
> index 0000000..3c02724
> --- /dev/null
> +++ b/meta/classes/buildstats-summary.bbclass
> @@ -0,0 +1,43 @@
> +# Imported from meta-mentor
> +# http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor/log/classes/
> buildstats-summary.bbclass
> +# with following modifications
> +# 1) when both setscene and normal tasks were executed, count it as
> no_sstate
> +# 2) count do_populate_sysroot and do_packagedata tasks, because often we
> invalidate only runtime deps
>

Looks like useful improvements, nicely done. Have you thought about not
hardcoding the task list, if we're going to display it this way? See
https://gist.github.com/kergoth/2519927f59712282e9e8 for a version which
obeys SSTATETASKS.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2150 bytes --]

Re: [PATCH] buildstats-summary.bbclass: Import useful bbclass from meta-mentor

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]

On Wed, Jun 11, 2014 at 01:54:38PM -0700, Christopher Larson wrote:
> On Fri, Jun 6, 2014 at 12:27 PM, Martin Jansa <martin.jansa@gmail.com>
> wrote:
> 
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  meta/classes/buildstats-summary.bbclass | 43
> > +++++++++++++++++++++++++++++++++
> >  1 file changed, 43 insertions(+)
> >  create mode 100644 meta/classes/buildstats-summary.bbclass
> >
> > diff --git a/meta/classes/buildstats-summary.bbclass b/meta/classes/
> > buildstats-summary.bbclass
> > new file mode 100644
> > index 0000000..3c02724
> > --- /dev/null
> > +++ b/meta/classes/buildstats-summary.bbclass
> > @@ -0,0 +1,43 @@
> > +# Imported from meta-mentor
> > +# http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor/log/classes/
> > buildstats-summary.bbclass
> > +# with following modifications
> > +# 1) when both setscene and normal tasks were executed, count it as
> > no_sstate
> > +# 2) count do_populate_sysroot and do_packagedata tasks, because often we
> > invalidate only runtime deps
> >
> 
> Looks like useful improvements, nicely done. Have you thought about not
> hardcoding the task list, if we're going to display it this way? See
> https://gist.github.com/kergoth/2519927f59712282e9e8 for a version which
> obeys SSTATETASKS.

I'm fine with your version showing all SSTATETASKS, would you mind
sending PATCHv2 with your modification or should I?

I've hardcoded packagedata only because it was typical case when I was
wondering why the build took so long even with populate_sysroot reused
from sstate (e.g. after some package.bbclass change), but showing all
doesn't hurt.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

[PATCH v2] buildstats-summary.bbclass: Import useful bbclass from meta-mentor

[Christopher Larson]

From: Martin Jansa <martin.jansa@gmail.com>

This class summarizes sstate reuse at the end of the build, so you know how
much of your build was done from scratch.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Christopher Larson <kergoth@gmail.com>
---
v2 changes: obey SSTATETASKS, add file comment -CL

 meta/classes/buildstats-summary.bbclass |   39 +++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100644 meta/classes/buildstats-summary.bbclass

diff --git a/meta/classes/buildstats-summary.bbclass b/meta/classes/buildstats-summary.bbclass
new file mode 100644
index 0000000..c8fbb2f
--- /dev/null
+++ b/meta/classes/buildstats-summary.bbclass
@@ -0,0 +1,39 @@
+# Summarize sstate usage at the end of the build
+python buildstats_summary () {
+    if not isinstance(e, bb.event.BuildCompleted):
+        return
+
+    import collections
+    import os.path
+
+    bn = get_bn(e)
+    bsdir = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), bn)
+    if not os.path.exists(bsdir):
+        return
+
+    sstatetasks = (e.data.getVar('SSTATETASKS', True) or '').split()
+    built = collections.defaultdict(lambda: [set(), set()])
+    for pf in os.listdir(bsdir):
+        taskdir = os.path.join(bsdir, pf)
+        if not os.path.isdir(taskdir):
+            continue
+
+        tasks = os.listdir(taskdir)
+        for t in sstatetasks:
+            no_sstate, sstate = built[t]
+            if t in tasks:
+                no_sstate.add(pf)
+            elif t + '_setscene' in tasks:
+                sstate.add(pf)
+
+    header_printed = False
+    for t in sstatetasks:
+        no_sstate, sstate = built[t]
+        if no_sstate | sstate:
+            if not header_printed:
+                header_printed = True
+                bb.note("Build completion summary:")
+
+            bb.note("  {0}: {1}% sstate reuse ({2} setscene, {3} scratch)".format(t, 100*len(sstate)/(len(sstate)+len(no_sstate)), len(sstate), len(no_sstate)))
+}
+addhandler buildstats_summary
-- 
1.7.9.5