[PATCH 0/3] rm_work enhancements

[PATCH 0/3] rm_work enhancements

[Patrick Ohly]

This is the OE-core side of the rm_work.bbclass enhancements. Depends
on the corresponding bitbake patch series. See the OE-core "rm_work +
pybootchart enhancements" mail thread for further information.

The following changes since commit af330fc0087692039d4256a6fb1e122742e8ec1a:

  packagegroup-core-lsb: add opengl to REQUIRED_DISTRO_FEATURES (2017-01-05 13:53:45 +0000)

are available in the git repository at:

  git://github.com/pohly/openembedded-core rmwork
  https://github.com/pohly/openembedded-core/tree/rmwork

Patrick Ohly (3):
  gcc-source.inc: cleanly disable do_rm_work
  rm_work_and_downloads.bbclass: more aggressively minimize disk usage
  rm_work.bbclass: clean up sooner

 meta/classes/rm_work.bbclass               | 31 ++++++++++++++++------------
 meta/classes/rm_work_and_downloads.bbclass | 33 ++++++++++++++++++++++++++++++
 meta/recipes-devtools/gcc/gcc-source.inc   |  2 +-
 3 files changed, 52 insertions(+), 14 deletions(-)
 create mode 100644 meta/classes/rm_work_and_downloads.bbclass

-- 
2.1.4

[PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work

[Patrick Ohly]

Using "deltask" assumes that do_rm_work has been added already, which
won't be the case anymore in the upcoming improved rm_work.bbclass,
because then an anonymous python method will add do_rm_work.

Setting RM_WORK_EXCLUDE works with the current and upcoming
rm_work.bbclass and is the API that is meant to be used for excluding
recipes from cleaning, so use that.
---
 meta/recipes-devtools/gcc/gcc-source.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc
index 49bde92..0d0edb5 100644
--- a/meta/recipes-devtools/gcc/gcc-source.inc
+++ b/meta/recipes-devtools/gcc/gcc-source.inc
@@ -3,7 +3,7 @@ deltask do_compile
 deltask do_install 
 deltask do_populate_sysroot
 deltask do_populate_lic 
-deltask do_rm_work
+RM_WORK_EXCLUDE += "${PN}"
 
 inherit nopackages
 
-- 
2.1.4

Re: [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work

[Khem Raj]

On Fri, Jan 6, 2017 at 1:55 AM, Patrick Ohly <patrick.ohly@intel.com> wrote:
> Using "deltask" assumes that do_rm_work has been added already, which
> won't be the case anymore in the upcoming improved rm_work.bbclass,
> because then an anonymous python method will add do_rm_work.
>
> Setting RM_WORK_EXCLUDE works with the current and upcoming
> rm_work.bbclass and is the API that is meant to be used for excluding
> recipes from cleaning, so use that.
> ---
>  meta/recipes-devtools/gcc/gcc-source.inc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc
> index 49bde92..0d0edb5 100644
> --- a/meta/recipes-devtools/gcc/gcc-source.inc
> +++ b/meta/recipes-devtools/gcc/gcc-source.inc
> @@ -3,7 +3,7 @@ deltask do_compile
>  deltask do_install
>  deltask do_populate_sysroot
>  deltask do_populate_lic
> -deltask do_rm_work
> +RM_WORK_EXCLUDE += "${PN}"

seems ok

>
>  inherit nopackages
>
> --
> 2.1.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Patrick Ohly]

rm_work.bbclass never deletes downloaded files, even if they are not
going to be needed again during the
build. rm_work_and_downloads.bbclass is more aggressive in minimizing
the used disk space during a build, but has other disadvantages:
- sources required by different recipes need to be fetched once per
  recipe, not once per build
- incremental builds do not work reliably because sources get
  removed without ensuring that sources gets fetched again

That makes rm_work_and_downloads.bbclass useful for one-time builds in
a constrained environment (like a CI system), but not for general use.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/rm_work_and_downloads.bbclass | 33 ++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 meta/classes/rm_work_and_downloads.bbclass

diff --git a/meta/classes/rm_work_and_downloads.bbclass b/meta/classes/rm_work_and_downloads.bbclass
new file mode 100644
index 0000000..7c00bea
--- /dev/null
+++ b/meta/classes/rm_work_and_downloads.bbclass
@@ -0,0 +1,33 @@
+# Author:       Patrick Ohly <patrick.ohly@intel.com>
+# Copyright:    Copyright (C) 2015 Intel Corporation
+#
+# This file is licensed under the MIT license, see COPYING.MIT in
+# this source distribution for the terms.
+
+# This class is used like rm_work:
+# INHERIT += "rm_work_and_downloads"
+#
+# In addition to removing local build directories of a recipe, it also
+# removes the downloaded source. This is achieved by making the DL_DIR
+# recipe-specific. While reducing disk usage, it increases network usage (for
+# example, compiling the same source for target and host implies downloading
+# the source twice).
+#
+# Because the "do_fetch" task does not get re-run after removing the downloaded
+# sources, this class is also not suitable for incremental builds.
+#
+# Where it works well is in well-connected build environments with limited
+# disk space (like TravisCI).
+
+inherit rm_work
+
+# This would ensure that the existing do_rm_work() removes the downloads,
+# but does not work because some recipes have a circular dependency between
+# WORKDIR and DL_DIR (via ${SRCPV}?).
+# DL_DIR = "${WORKDIR}/downloads"
+
+# Instead go up one level and remove ourself.
+DL_DIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/downloads"
+do_rm_work_append () {
+    rm -rf ${DL_DIR}
+}
-- 
2.1.4

Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Randy Witt]

[-- Attachment #1: Type: text/plain, Size: 3250 bytes --]

Hi Patrick,

On Fri, Jan 6, 2017 at 1:55 AM, Patrick Ohly <patrick.ohly@intel.com> wrote:

> rm_work.bbclass never deletes downloaded files, even if they are not
> going to be needed again during the
> build. rm_work_and_downloads.bbclass is more aggressive in minimizing
> the used disk space during a build, but has other disadvantages:
> - sources required by different recipes need to be fetched once per
>   recipe, not once per build
> - incremental builds do not work reliably because sources get
>   removed without ensuring that sources gets fetched again
>
> That makes rm_work_and_downloads.bbclass useful for one-time builds in
> a constrained environment (like a CI system), but not for general use.
>
>
I'd rather see this be a new class independent from rm_work. People can
always in inherit both rm_work and rm_download.

That being said, since this doesn't operate at the fetch level, i.e. only
remove things that were fetched as part of the current invocation of
bitbake, I don't see why the user couldn't just "rm -rf downloads"
themselves as part of the ci teardown.


> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
> ---
>  meta/classes/rm_work_and_downloads.bbclass | 33
> ++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
>  create mode 100644 meta/classes/rm_work_and_downloads.bbclass
>
> diff --git a/meta/classes/rm_work_and_downloads.bbclass
> b/meta/classes/rm_work_and_downloads.bbclass
> new file mode 100644
> index 0000000..7c00bea
> --- /dev/null
> +++ b/meta/classes/rm_work_and_downloads.bbclass
> @@ -0,0 +1,33 @@
> +# Author:       Patrick Ohly <patrick.ohly@intel.com>
> +# Copyright:    Copyright (C) 2015 Intel Corporation
> +#
> +# This file is licensed under the MIT license, see COPYING.MIT in
> +# this source distribution for the terms.
> +
> +# This class is used like rm_work:
> +# INHERIT += "rm_work_and_downloads"
> +#
> +# In addition to removing local build directories of a recipe, it also
> +# removes the downloaded source. This is achieved by making the DL_DIR
> +# recipe-specific. While reducing disk usage, it increases network usage
> (for
> +# example, compiling the same source for target and host implies
> downloading
> +# the source twice).
> +#
> +# Because the "do_fetch" task does not get re-run after removing the
> downloaded
> +# sources, this class is also not suitable for incremental builds.
> +#
> +# Where it works well is in well-connected build environments with limited
> +# disk space (like TravisCI).
> +
> +inherit rm_work
> +
> +# This would ensure that the existing do_rm_work() removes the downloads,
> +# but does not work because some recipes have a circular dependency
> between
> +# WORKDIR and DL_DIR (via ${SRCPV}?).
> +# DL_DIR = "${WORKDIR}/downloads"
> +
> +# Instead go up one level and remove ourself.
> +DL_DIR = "${BASE_WORKDIR}/${MULTIMACH_TARGET_SYS}/${PN}/downloads"
> +do_rm_work_append () {
> +    rm -rf ${DL_DIR}
> +}
> --
> 2.1.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 4283 bytes --]

Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Patrick Ohly]

On Fri, 2017-01-06 at 10:31 -0800, Randy Witt wrote:

> 
> I'd rather see this be a new class independent from rm_work. People
> can always in inherit both rm_work and rm_download.

I'm not sure whether it's worth supporting that mode: what would be the
motivation for removing downloads, but not the work directory?

I'm a bit worried about unexpected breakages when using just
rm_download.

> That being said, since this doesn't operate at the fetch level, i.e.
> only remove things that were fetched as part of the current invocation
> of bitbake, I don't see why the user couldn't just "rm -rf downloads"
> themselves as part of the ci teardown.

The motivation for rm_work_and_downloads.bbclass is the same as for
rm_work.bbclass: reducing the required disk space *during* the build, to
get builds to succeed which otherwise would run out of disk space.

A "rm -rf downloads" at the end of the build doesn't help achieve that
goal.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Randy Witt]

[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]

On Fri, Jan 6, 2017 at 11:22 AM, Patrick Ohly <patrick.ohly@intel.com>
wrote:

> On Fri, 2017-01-06 at 10:31 -0800, Randy Witt wrote:
>
> >
> > I'd rather see this be a new class independent from rm_work. People
> > can always in inherit both rm_work and rm_download.
>
> I'm not sure whether it's worth supporting that mode: what would be the
> motivation for removing downloads, but not the work directory?
>
>
There are times that the work directories help with the debugging of build
failures. The logs aren't always enough. So a person might want to do
something like remove the downloads, but preserve the actual work,
especially in the case of failures.

I'll admit it is a fabricated scenario, but in general I'm against one knob
requiring the turning of another knob.


> The motivation for rm_work_and_downloads.bbclass is the same as for
> rm_work.bbclass: reducing the required disk space *during* the build, to
> get builds to succeed which otherwise would run out of disk space.
>
>
Disk space isn't the only motivation for rm_work. It also allows for
deletion of large amounts of inodes before they've been flushed to disk. So
rather than taking minutes worth of time to clean up post build, and
thrashing media that other tasks may be using, it is almost instantaneous
because everything is still in the page cache.

Also the build itself can be faster for the same reason, there is no
waiting for data to be flushed out to disk( because it was deleted while
still in the page cache) and thus impacting reads in other parts of the
build.


> A "rm -rf downloads" at the end of the build doesn't help achieve that
> goal.
>
>
That's true, I hadn't yet processed it completely. Ignore that comment. :)

As an overall comment though, I really like the idea of rm_work not being
deferred for so long. It always seemed strange to me to see the thundering
herd of rm_works after certain do_builds would finish.

[-- Attachment #2: Type: text/html, Size: 2844 bytes --]

Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Patrick Ohly]

On Fri, 2017-01-06 at 13:29 -0800, Randy Witt wrote:


> There are times that the work directories help with the debugging of
> build failures. The logs aren't always enough. So a person might want
> to do something like remove the downloads, but preserve the actual
> work, especially in the case of failures.
> 
> 
> I'll admit it is a fabricated scenario, but in general I'm against one
> knob requiring the turning of another knob.

In principle I agree. Okay, let's separate the two.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Patrick Ohly]

On Sat, 2017-01-07 at 09:09 +0100, Patrick Ohly wrote:
> On Fri, 2017-01-06 at 13:29 -0800, Randy Witt wrote:
> 
> 
> > There are times that the work directories help with the debugging of
> > build failures. The logs aren't always enough. So a person might want
> > to do something like remove the downloads, but preserve the actual
> > work, especially in the case of failures.
> > 
> > 
> > I'll admit it is a fabricated scenario, but in general I'm against one
> > knob requiring the turning of another knob.
> 
> In principle I agree. Okay, let's separate the two.

I hit one snag when starting to do that: the implementation of download
removal depends on do_rm_work from rm_work.bbclass for hooking into the
build of a recipe at the right place.

I can't think of an easy way of avoiding that without duplicating the
complex logic from rm_work.bbclass, which itself is currently about to
change, so I'd prefer to keep rm_work_and_downloads.bbclass as proposed.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage

[Randy Witt]

[-- Attachment #1: Type: text/plain, Size: 695 bytes --]

On Mon, Jan 9, 2017 at 2:25 AM, Patrick Ohly <patrick.ohly@intel.com> wrote:

>
> >
> > In principle I agree. Okay, let's separate the two.
>
> I hit one snag when starting to do that: the implementation of download
> removal depends on do_rm_work from rm_work.bbclass for hooking into the
> build of a recipe at the right place.
>
> I can't think of an easy way of avoiding that without duplicating the
> complex logic from rm_work.bbclass, which itself is currently about to
> change, so I'd prefer to keep rm_work_and_downloads.bbclass as proposed.
>

Sounds good to me. If it turns out someone would like them separate in the
future, we can schedule the time to do it then.

[-- Attachment #2: Type: text/html, Size: 1086 bytes --]

[PATCH 3/3] rm_work.bbclass: clean up sooner

[Patrick Ohly]

Having do_rm_work depend on do_build had one major disadvantage:
do_build depends on the do_build of other recipes, to ensure that
runtime dependencies also get built. The effect is that when work on a
recipe is complete and it could get cleaned up, do_rm_work still
doesn't run because it waits for those other recipes, thus leading to
more temporary disk space usage than really needed.

The right solution is to inject do_rm_work before do_build and after
all tasks of the recipe. Achieving that depends on the new bitbake
support for prioritizing anonymous functions to ensure that
rm_work.bbclass gets to see a full set of existing tasks when adding
its own one. This is relevant, for example, for do_analyseimage in
meta-security-isafw's isafw.bbclass.

In addition, the new "rm_work" scheduler is used by default. It
prioritizes finishing recipes over continuing with the more
important recipes (with "importance" determined by the number of
reverse-dependencies).

Benchmarking (see "rm_work + pybootchart enhancements" on the OE-core
mailing list) showed that builds with the modified rm_work.bbclass
were both faster (albeit not by much) and required considerably less
disk space (14230MiB instead of 18740MiB for core-image-sato).
Interestingly enough, builds with rm_work.bbclass were also faster
than those without.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/rm_work.bbclass | 31 ++++++++++++++++++-------------
 1 file changed, 18 insertions(+), 13 deletions(-)

diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
index 3516c7e..1205104 100644
--- a/meta/classes/rm_work.bbclass
+++ b/meta/classes/rm_work.bbclass
@@ -11,16 +11,13 @@
 # RM_WORK_EXCLUDE += "icu-native icu busybox"
 #
 
-# Use the completion scheduler by default when rm_work is active
+# Use the dedicated rm_work scheduler by default when rm_work is active
 # to try and reduce disk usage
-BB_SCHEDULER ?= "completion"
+BB_SCHEDULER ?= "rm_work"
 
 # Run the rm_work task in the idle scheduling class
 BB_TASK_IONICE_LEVEL_task-rm_work = "3.0"
 
-RMWORK_ORIG_TASK := "${BB_DEFAULT_TASK}"
-BB_DEFAULT_TASK = "rm_work_all"
-
 do_rm_work () {
     # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe.
     for p in ${RM_WORK_EXCLUDE}; do
@@ -97,13 +94,6 @@ do_rm_work () {
         rm -f $i
     done
 }
-addtask rm_work after do_${RMWORK_ORIG_TASK}
-
-do_rm_work_all () {
-    :
-}
-do_rm_work_all[recrdeptask] = "do_rm_work"
-addtask rm_work_all after do_rm_work
 
 do_populate_sdk[postfuncs] += "rm_work_populatesdk"
 rm_work_populatesdk () {
@@ -117,7 +107,7 @@ rm_work_rootfs () {
 }
 rm_work_rootfs[cleandirs] = "${WORKDIR}/rootfs"
 
-python () {
+python __anonymous_rm_work() {
     if bb.data.inherits_class('kernel', d):
         d.appendVar("RM_WORK_EXCLUDE", ' ' + d.getVar("PN"))
     # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe.
@@ -126,4 +116,19 @@ python () {
     if pn in excludes:
         d.delVarFlag('rm_work_rootfs', 'cleandirs')
         d.delVarFlag('rm_work_populatesdk', 'cleandirs')
+    else:
+        # Inject do_rm_work into the tasks of the current recipe such that do_build
+        # depends on it and that it runs after all other tasks that block do_build,
+        # i.e. after all work on the current recipe is done. The reason for taking
+        # this approach instead of making do_rm_work depend on do_build is that
+        # do_build inherits additional runtime dependencies on
+        # other recipes and thus will typically run much later than completion of
+        # work in the recipe itself.
+        deps = bb.build.preceedtask('do_build', True, d)
+        if 'do_build' in deps:
+            deps.remove('do_build')
+        bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
 }
+# Higher priority than the normal 100, and thus we run after other
+# classes like package_rpm.bbclass which also add custom tasks.
+__anonymous_rm_work[__anonprio] = "1000"
-- 
2.1.4

Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 9807 bytes --]

Are all changes necessary for this to work already in master?

Yesterday I've noticed that rm_work for some components which are early in
the dependency (like qtbase) are executed relatively late (together with
do_package_qa).

So I've tried very naive way to find out if the rm_work tasks are executed
sooner or not just by comparing Task IDs in build of the same image built
from scratch (without sstate) with Dizzy, Morty and current master.

First I've stripped unnecessary prefix and names of proprietary components
(in case someone wants me to share these lists):
grep "^NOTE: Running task .*, do_rm_work)$" log.build |sed
's#/jenkins/mjansa/build-[^/]*/##;
s#meta-lg-webos/[^:]*:#private-component:#g; s#^NOTE: Running task ##g' >
rm_work.tasks.dizzy

with slightly different regexp for morty and master:
grep "^NOTE: Running task .*:do_rm_work)$" ld.gold/log.m16p |sed
's#/jenkins/mjansa/build-[^/]*/##;
s#meta-lg-webos/[^:]*:#private-component:#g; s#^NOTE: Running task ##g' >
rm_work.tasks.morty

and then I did even more naive thing to compare average task id of rm-work
jobs with following results:
for i in rm_work.tasks.*; do echo $i; export COUNT=0 SUM=0; for
TASK in `cat $i | cut -f 1 -d\  `; do COUNT=`expr $COUNT + 1`; SUM=`expr
$SUM + $TASK`; done; echo "AVG = `expr $SUM / $C
OUNT`; COUNT = $COUNT"; done
rm_work.tasks.dizzy
AVG = 6429; COUNT = 764
rm_work.tasks.master
AVG = 7570; COUNT = 891
rm_work.tasks.master.qemux86
AVG = 5527; COUNT = 665
rm_work.tasks.morty
AVG = 6689; COUNT = 786
rm_work.tasks.morty.gold
AVG = 6764; COUNT = 786

rm_work.tasks.morty.gold is the same build as in rm_work.tasks.morty just
with ld-is-gold added to DISTRO_FEATUREs (as I was testing build time to
compare ld.bfd and ld.gold in our images).
rm_work.tasks.master.qemux86 is the same build as rm_work.tasks.master but
for qemux86, all other builds are for some ARM board we use

Then few interesting steps:

gcc-cross looks good (not available for dizzy build which is using external
toolchain)
$ grep gcc-cross_ rm_work.tasks.*
rm_work.tasks.master:510 of 14470
(oe-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work)
rm_work.tasks.master.qemux86:515 of 10296
(oe-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work)
rm_work.tasks.morty:2592 of 12021
(oe-core/meta/recipes-devtools/gcc/gcc-cross_6.2.bb:do_rm_work)
rm_work.tasks.morty.gold:2734 of 12021
(oe-core/meta/recipes-devtools/gcc/gcc-cross_6.2.bb:do_rm_work)

qtdeclarative-native got rm_work a bit later, whcih might be caused only by
the increased number of tasks thanks to RSS
$ grep native.*qtdeclarative rm_work.tasks.*
rm_work.tasks.dizzy:2101 of 11766 (ID: 11128,
virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb, do_rm_work)
rm_work.tasks.master:2614 of 14470
(virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)
rm_work.tasks.master.qemux86:2521 of 10296
(virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)
rm_work.tasks.morty:1513 of 12021
(virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)
rm_work.tasks.morty.gold:1514 of 12021
(virtual:native:meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)

and here is the target qtdeclarative which trigered this whole naive
analysis:
$ grep qtdeclarative rm_work.tasks.* | grep -v native
rm_work.tasks.dizzy:4952 of 11766 (ID: 6670, meta-qt5/recipes-qt/qt5/
qtdeclarative_git.bb, do_rm_work)
rm_work.tasks.master:4317 of 14470
(meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)
rm_work.tasks.master.qemux86:10142 of 10296
(meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)
rm_work.tasks.morty:6753 of 12021
(meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)
rm_work.tasks.morty.gold:6883 of 12021
(meta-qt5/recipes-qt/qt5/qtdeclarative_git.bb:do_rm_work)

If we dismiss the strange case in rm_work.tasks.master.qemux86 then it
seems to perform at least as good as old completion BB_SCHEDULER.

But I wanted to ask if there is something else we can do or you were
planing to do, because IIRC you shared some longer analysis of what could
be improved here and I'm not sure if you managed to implement it all.

It feels to me that rm_work has high priority, but still it's "blocked" by
e.g. do_package_qa which gets executed late and then immediately followed
by rm_work.

in ideal case I would really like to have a switch which will force rm_work
to take absolute priority over other tasks, it doesn't take very long to
delete the files in tmpfs and would allow me to do tmpfs builds on builders
with smaller RAM.

The "state of bitbake world" builds are performed in 74G tmpfs (for whole
tmpdir-glibc) and yesterday's builds started to fail again (when it happens
to run chromium and chromium-wayland at the same time) - the manual
solution for this I'm using for last couple years is to build in "steps"
which force to run rm_work for all included components, so e.g.

bitbake gcc-cross-arm && bitbake small-image && bitbake chromium && bitbake
chromium-wayland && bitbake big-image && bitbake world

will keep the tmpfs usage peaks much lower than running just bitbake world

On Fri, Jan 6, 2017 at 10:55 AM, Patrick Ohly <patrick.ohly@intel.com>
wrote:

> Having do_rm_work depend on do_build had one major disadvantage:
> do_build depends on the do_build of other recipes, to ensure that
> runtime dependencies also get built. The effect is that when work on a
> recipe is complete and it could get cleaned up, do_rm_work still
> doesn't run because it waits for those other recipes, thus leading to
> more temporary disk space usage than really needed.
>
> The right solution is to inject do_rm_work before do_build and after
> all tasks of the recipe. Achieving that depends on the new bitbake
> support for prioritizing anonymous functions to ensure that
> rm_work.bbclass gets to see a full set of existing tasks when adding
> its own one. This is relevant, for example, for do_analyseimage in
> meta-security-isafw's isafw.bbclass.
>
> In addition, the new "rm_work" scheduler is used by default. It
> prioritizes finishing recipes over continuing with the more
> important recipes (with "importance" determined by the number of
> reverse-dependencies).
>
> Benchmarking (see "rm_work + pybootchart enhancements" on the OE-core
> mailing list) showed that builds with the modified rm_work.bbclass
> were both faster (albeit not by much) and required considerably less
> disk space (14230MiB instead of 18740MiB for core-image-sato).
> Interestingly enough, builds with rm_work.bbclass were also faster
> than those without.
>
> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
> ---
>  meta/classes/rm_work.bbclass | 31 ++++++++++++++++++-------------
>  1 file changed, 18 insertions(+), 13 deletions(-)
>
> diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
> index 3516c7e..1205104 100644
> --- a/meta/classes/rm_work.bbclass
> +++ b/meta/classes/rm_work.bbclass
> @@ -11,16 +11,13 @@
>  # RM_WORK_EXCLUDE += "icu-native icu busybox"
>  #
>
> -# Use the completion scheduler by default when rm_work is active
> +# Use the dedicated rm_work scheduler by default when rm_work is active
>  # to try and reduce disk usage
> -BB_SCHEDULER ?= "completion"
> +BB_SCHEDULER ?= "rm_work"
>
>  # Run the rm_work task in the idle scheduling class
>  BB_TASK_IONICE_LEVEL_task-rm_work = "3.0"
>
> -RMWORK_ORIG_TASK := "${BB_DEFAULT_TASK}"
> -BB_DEFAULT_TASK = "rm_work_all"
> -
>  do_rm_work () {
>      # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe.
>      for p in ${RM_WORK_EXCLUDE}; do
> @@ -97,13 +94,6 @@ do_rm_work () {
>          rm -f $i
>      done
>  }
> -addtask rm_work after do_${RMWORK_ORIG_TASK}
> -
> -do_rm_work_all () {
> -    :
> -}
> -do_rm_work_all[recrdeptask] = "do_rm_work"
> -addtask rm_work_all after do_rm_work
>
>  do_populate_sdk[postfuncs] += "rm_work_populatesdk"
>  rm_work_populatesdk () {
> @@ -117,7 +107,7 @@ rm_work_rootfs () {
>  }
>  rm_work_rootfs[cleandirs] = "${WORKDIR}/rootfs"
>
> -python () {
> +python __anonymous_rm_work() {
>      if bb.data.inherits_class('kernel', d):
>          d.appendVar("RM_WORK_EXCLUDE", ' ' + d.getVar("PN"))
>      # If the recipe name is in the RM_WORK_EXCLUDE, skip the recipe.
> @@ -126,4 +116,19 @@ python () {
>      if pn in excludes:
>          d.delVarFlag('rm_work_rootfs', 'cleandirs')
>          d.delVarFlag('rm_work_populatesdk', 'cleandirs')
> +    else:
> +        # Inject do_rm_work into the tasks of the current recipe such
> that do_build
> +        # depends on it and that it runs after all other tasks that block
> do_build,
> +        # i.e. after all work on the current recipe is done. The reason
> for taking
> +        # this approach instead of making do_rm_work depend on do_build
> is that
> +        # do_build inherits additional runtime dependencies on
> +        # other recipes and thus will typically run much later than
> completion of
> +        # work in the recipe itself.
> +        deps = bb.build.preceedtask('do_build', True, d)
> +        if 'do_build' in deps:
> +            deps.remove('do_build')
> +        bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
>  }
> +# Higher priority than the normal 100, and thus we run after other
> +# classes like package_rpm.bbclass which also add custom tasks.
> +__anonymous_rm_work[__anonprio] = "1000"
> --
> 2.1.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 12462 bytes --]

Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Patrick Ohly]

On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote:
> Are all changes necessary for this to work already in master?

Yes.

> Yesterday I've noticed that rm_work for some components which are
> early in the dependency (like qtbase) are executed relatively late
> (together with do_package_qa).

Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know
that, and therefore schedules do_rm_work after do_package_qa.

If yes, then adding a list of tasks that can be ignored would be
trivial. This can be a variable, so a recipe can even add their own
ones, if necessary.

> So I've tried very naive way to find out if the rm_work tasks are
> executed sooner or not just by comparing Task IDs in build of the same
> image built from scratch (without sstate) with Dizzy, Morty and
> current master.

Interesting, I hadn't thought of testing it like that.

> If we dismiss the strange case in rm_work.tasks.master.qemux86 then it
> seems to perform at least as good as old completion BB_SCHEDULER.
> 
> 
> But I wanted to ask if there is something else we can do or you were
> planing to do, because IIRC you shared some longer analysis of what
> could be improved here and I'm not sure if you managed to implement it
> all.

The other ideas that I mentioned at some point didn't pan out as
intended. In particular allowing do_rm_work tasks to run when the normal
task limit was reached didn't have a big effect and the implementation
was a hack, so I dropped that.

> It feels to me that rm_work has high priority, but still it's
> "blocked" by e.g. do_package_qa which gets executed late and then
> immediately followed by rm_work.

That should be easy to change, perhaps like this (untested):

RM_WORK_TASKS_WHITELIST = "do_build do_package_qa"

        deps = set(bb.build.preceedtask('do_build', True, d))
	whitelist = d.getVar('RM_WORK_TASKS_WHITELIST').split()
        deps.difference_update(whitelist)
        # In practice, addtask() here merely updates the dependencies.
        bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)


-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5264 bytes --]

On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote:
> On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote:
> > Are all changes necessary for this to work already in master?
> 
> Yes.
> 
> > Yesterday I've noticed that rm_work for some components which are
> > early in the dependency (like qtbase) are executed relatively late
> > (together with do_package_qa).
> 
> Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know
> that, and therefore schedules do_rm_work after do_package_qa.
> 
> If yes, then adding a list of tasks that can be ignored would be
> trivial. This can be a variable, so a recipe can even add their own
> ones, if necessary.

That's now what I've meant.

I believe that rm_work needs to be executed after do_package_qa, but I
don't understand the scheduler code enough (at least not yet) to say
that higher priority of rm_work task also makes all the tasks rm_work
depends on e.g. do_package_qa to be executed sooner.

From my observation it looked like do_package_qa is still executed
"late", but immediately followed by rm_work thanks to its high priority
(so it's executed as soon as it can, but it's still late in progress of
the whole build).

Another interesting test from today was to run:
# rm -rf tmp-glibc/*
# bitbake -n zlib | tee log.zlib.rm_work
# cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd ..
# rm -rf tmp-glibc/*
# bitbake -n zlib | tee log.zlib.rm_work.revert

and it shows interesting difference that many rm_work tasks aren't
executed at all:

# grep rm_work log.zlib.rm_work* | grep zlib_
log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all)

# grep rm_work log.zlib.rm_work* | grep gcc
log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work)
log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work)

# grep -c rm_work log.zlib.rm_work*
log.zlib.rm_work:1
log.zlib.rm_work.revert:63

I'll check if it's something in my setup or if this happens everywhere now.

> > So I've tried very naive way to find out if the rm_work tasks are
> > executed sooner or not just by comparing Task IDs in build of the same
> > image built from scratch (without sstate) with Dizzy, Morty and
> > current master.
> 
> Interesting, I hadn't thought of testing it like that.
> 
> > If we dismiss the strange case in rm_work.tasks.master.qemux86 then it
> > seems to perform at least as good as old completion BB_SCHEDULER.
> > 
> > 
> > But I wanted to ask if there is something else we can do or you were
> > planing to do, because IIRC you shared some longer analysis of what
> > could be improved here and I'm not sure if you managed to implement it
> > all.
> 
> The other ideas that I mentioned at some point didn't pan out as
> intended. In particular allowing do_rm_work tasks to run when the normal
> task limit was reached didn't have a big effect and the implementation
> was a hack, so I dropped that.
> 
> > It feels to me that rm_work has high priority, but still it's
> > "blocked" by e.g. do_package_qa which gets executed late and then
> > immediately followed by rm_work.
> 
> That should be easy to change, perhaps like this (untested):
> 
> RM_WORK_TASKS_WHITELIST = "do_build do_package_qa"
> 
>         deps = set(bb.build.preceedtask('do_build', True, d))
> 	whitelist = d.getVar('RM_WORK_TASKS_WHITELIST').split()
>         deps.difference_update(whitelist)
>         # In practice, addtask() here merely updates the dependencies.
>         bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
> 
> 
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Patrick Ohly]

On Wed, 2017-03-01 at 16:52 +0100, Martin Jansa wrote:
> On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote:
> > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote:
> > > Are all changes necessary for this to work already in master?
> > 
> > Yes.
> > 
> > > Yesterday I've noticed that rm_work for some components which are
> > > early in the dependency (like qtbase) are executed relatively late
> > > (together with do_package_qa).
> > 
> > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know
> > that, and therefore schedules do_rm_work after do_package_qa.
> > 
> > If yes, then adding a list of tasks that can be ignored would be
> > trivial. This can be a variable, so a recipe can even add their own
> > ones, if necessary.
> 
> That's now what I've meant.
> 
> I believe that rm_work needs to be executed after do_package_qa, but I
> don't understand the scheduler code enough (at least not yet) to say
> that higher priority of rm_work task also makes all the tasks rm_work
> depends on e.g. do_package_qa to be executed sooner.

I see. do_package_qa should have a high priority, but it's still blocked
by its dependencies. Building those dependencies only gets an indirect
boost from scheduling recipes that many other recipes depend on first (a
feature of the normal scheduler).

I suspect the problem with do_package_qa is similar to the problem with
do_build before: it depends on do_package_qa of everything a recipe
depends on, and thus finishing the build of those other recipes also
blocks finishing the current recipe. That creates very deep dependency
chains and thus increases the number of recipes which are "in progress"
at the same time.

> Another interesting test from today was to run:
> # rm -rf tmp-glibc/*
> # bitbake -n zlib | tee log.zlib.rm_work
> # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd ..
> # rm -rf tmp-glibc/*
> # bitbake -n zlib | tee log.zlib.rm_work.revert
> 
> and it shows interesting difference that many rm_work tasks aren't
> executed at all:
> 
> # grep rm_work log.zlib.rm_work* | grep zlib_
> log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all)
> 
> # grep rm_work log.zlib.rm_work* | grep gcc
> log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work)
> 
> # grep -c rm_work log.zlib.rm_work*
> log.zlib.rm_work:1
> log.zlib.rm_work.revert:63
> 
> I'll check if it's something in my setup or if this happens everywhere now.

I'll try to double-check this myself, too.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 15225 bytes --]

On Wed, Mar 01, 2017 at 05:44:53PM +0100, Patrick Ohly wrote:
> On Wed, 2017-03-01 at 16:52 +0100, Martin Jansa wrote:
> > On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote:
> > > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote:
> > > > Are all changes necessary for this to work already in master?
> > > 
> > > Yes.
> > > 
> > > > Yesterday I've noticed that rm_work for some components which are
> > > > early in the dependency (like qtbase) are executed relatively late
> > > > (together with do_package_qa).
> > > 
> > > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know
> > > that, and therefore schedules do_rm_work after do_package_qa.
> > > 
> > > If yes, then adding a list of tasks that can be ignored would be
> > > trivial. This can be a variable, so a recipe can even add their own
> > > ones, if necessary.
> > 
> > That's now what I've meant.
> > 
> > I believe that rm_work needs to be executed after do_package_qa, but I
> > don't understand the scheduler code enough (at least not yet) to say
> > that higher priority of rm_work task also makes all the tasks rm_work
> > depends on e.g. do_package_qa to be executed sooner.
> 
> I see. do_package_qa should have a high priority, but it's still blocked
> by its dependencies. Building those dependencies only gets an indirect
> boost from scheduling recipes that many other recipes depend on first (a
> feature of the normal scheduler).
> 
> I suspect the problem with do_package_qa is similar to the problem with
> do_build before: it depends on do_package_qa of everything a recipe
> depends on, and thus finishing the build of those other recipes also
> blocks finishing the current recipe. That creates very deep dependency
> chains and thus increases the number of recipes which are "in progress"
> at the same time.

Another (surprising for me) output from bitbake -D -D -D:

OE qemux86@ ~/build/oe-core $ head log.zlib.rm_work.debug3.prio.*
==> log.zlib.rm_work.debug3.prio.completion <==
DEBUG: completion priorities (most important first):
1. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build
2. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
3. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package_write_ipk
4. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_package_write_ipk
5. ID /OE/build/oe-core/openembedded-core/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.9.bb:do_package_write_ipk
6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.6.bb:do_package_write_ipk
7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/make/make_4.2.1.bb:do_package_write_ipk
8. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_package_write_ipk
9. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_package_write_ipk

==> log.zlib.rm_work.debug3.prio.normal <==
DEBUG: original priorities (most important first):
1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch
2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
9. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch

From these 2 I would expect the later one to be completion (to finish whole quilt-native before starting anything else
except when all quilt-native tasks are blocked waiting for dependencies and BB_NUMBER_THREADS allows to run
more task).

But I understand that this isn't the actual order of tasks as explained in bitbake comment:
        # Group tasks of the same kind before tasks of less important
        # kinds at the head of the queue (because earlier = lower
        # priority number = runs earlier), while preserving the
        # ordering by recipe. If recipe foo is more important than
        # bar, then the goal is to work on foo's do_populate_sysroot
        # before bar's do_populate_sysroot and on the more important
        # tasks of foo before any of the less important tasks in any
        # other recipe (if those other recipes are more important than
        # foo).
        #
        # All of this only applies when tasks are runable. Explicit
        # dependencies still override this ordering by priority.
        #
        # Here's an example why this priority re-ordering helps with
        # minimizing disk usage. Consider a recipe foo with a higher
        # priority than bar where foo DEPENDS on bar. Then the
        # implicit rule (from base.bbclass) is that foo's do_configure
        # depends on bar's do_populate_sysroot. This ensures that
        # bar's do_populate_sysroot gets done first. Normally the
        # tasks from foo would continue to run once that is done, and
        # bar only gets completed and cleaned up later. By ordering
        # bar's task that depend on bar's do_populate_sysroot before foo's
        # do_configure, that problem gets avoided.

and the later one still doesn't list rm_worl task for quilt-native which is the real
one I'm interested in.

When I do the same with last rm_work.bbclass change reverted I get this:
$ head -n 12 log.zlib.rm_work.revert.debug3.prio.*
==> log.zlib.rm_work.revert.debug3.prio.completion <==
DEBUG: completion priorities (most important first):
1. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all
2. ID /OE/build/oe-core/openembedded-core/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb:do_rm_work
3. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-support/libpcre/libpcre_8.40.bb:do_rm_work
4. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.0.bb:do_rm_work
5. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work
6. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-support/attr/attr_2.4.47.bb:do_rm_work
7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/python/python-native_2.7.12.bb:do_rm_work
8. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
9. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_rm_work
10. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_rm_work
11. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-extended/pigz/pigz_2.3.4.bb:do_rm_work

==> log.zlib.rm_work.revert.debug3.prio.normal <==
DEBUG: original priorities (most important first):
1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch
2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
9. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch
10. ID virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_fetch
11. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_unpack

Here it doesn't look so good for quilt-native:rm_work, it's included in zlib dependencies (thanks to the revert), but
with "speed" scheduler it's far, while "completion" puts it early:
log.zlib.rm_work.revert.debug3.prio.completion:40. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work
log.zlib.rm_work.revert.debug3.prio.completion:66. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_build
log.zlib.rm_work.revert.debug3.prio.completion:159. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_lic
log.zlib.rm_work.revert.debug3.prio.completion:229. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
log.zlib.rm_work.revert.debug3.prio.completion:299. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
log.zlib.rm_work.revert.debug3.prio.completion:361. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
log.zlib.rm_work.revert.debug3.prio.completion:423. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
log.zlib.rm_work.revert.debug3.prio.completion:484. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
log.zlib.rm_work.revert.debug3.prio.completion:546. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
log.zlib.rm_work.revert.debug3.prio.completion:603. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
log.zlib.rm_work.revert.debug3.prio.completion:660. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch

log.zlib.rm_work.revert.debug3.prio.normal:1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch
log.zlib.rm_work.revert.debug3.prio.normal:2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
log.zlib.rm_work.revert.debug3.prio.normal:3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
log.zlib.rm_work.revert.debug3.prio.normal:4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
log.zlib.rm_work.revert.debug3.prio.normal:5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
log.zlib.rm_work.revert.debug3.prio.normal:6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
log.zlib.rm_work.revert.debug3.prio.normal:7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
log.zlib.rm_work.revert.debug3.prio.normal:8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
log.zlib.rm_work.revert.debug3.prio.normal:561. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_lic
log.zlib.rm_work.revert.debug3.prio.normal:599. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_build
log.zlib.rm_work.revert.debug3.prio.normal:697. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work

From this it looks like setting the weight of do_rm_work above do_populate_sysroot
would work with "speed" scheduler the way I was hoping for.

I'll retry the same on even smaller build and with BB_NUMBER_THREADS = "1" so that
it's easier to grasp what the completion scheduler really tries to do and how it
influences the actual order of runqueue in the end.

> 
> > Another interesting test from today was to run:
> > # rm -rf tmp-glibc/*
> > # bitbake -n zlib | tee log.zlib.rm_work
> > # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd ..
> > # rm -rf tmp-glibc/*
> > # bitbake -n zlib | tee log.zlib.rm_work.revert
> > 
> > and it shows interesting difference that many rm_work tasks aren't
> > executed at all:
> > 
> > # grep rm_work log.zlib.rm_work* | grep zlib_
> > log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all)
> > 
> > # grep rm_work log.zlib.rm_work* | grep gcc
> > log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work)
> > log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work)
> > 
> > # grep -c rm_work log.zlib.rm_work*
> > log.zlib.rm_work:1
> > log.zlib.rm_work.revert:63
> > 
> > I'll check if it's something in my setup or if this happens everywhere now.
> 
> I'll try to double-check this myself, too.
> 
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Patrick Ohly]

On Wed, 2017-03-01 at 19:47 +0100, Martin Jansa wrote:
> Another (surprising for me) output from bitbake -D -D -D:
> 
> OE qemux86@ ~/build/oe-core $ head log.zlib.rm_work.debug3.prio.*
> ==> log.zlib.rm_work.debug3.prio.completion <==
> DEBUG: completion priorities (most important first):
> 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build
> 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
> 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package_write_ipk
> 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_package_write_ipk
> 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.9.bb:do_package_write_ipk
> 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.6.bb:do_package_write_ipk
> 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/make/make_4.2.1.bb:do_package_write_ipk
> 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_package_write_ipk
> 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_package_write_ipk
> 
> ==> log.zlib.rm_work.debug3.prio.normal <==
> DEBUG: original priorities (most important first):
> 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch
> 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
> 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
> 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
> 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
> 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
> 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
> 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
> 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch
> 
> From these 2 I would expect the later one to be completion (to finish whole quilt-native before starting anything else
> except when all quilt-native tasks are blocked waiting for dependencies and BB_NUMBER_THREADS allows to run
> more task).

I've not fully understood what the surprise was. I suppose
log.zlib.rm_work.debug3.prio.normal was generated with
BB_SCHEDULER="speed" and log.zlib.rm_work.debug3.prio.completion with
BB_SCHEDULER="completion".

For speed, quilt-native comes first because it's the recipe that most
recipes depend on. For completion, it's not near the top because it has
none of "important" tasks. It should be the first recipe for
do_populate_sysroot. The patch from my other mail should activate
quilt-native_0.65.bb:do_rm_work and then I'd expect it also at the top
for completion.

> But I understand that this isn't the actual order of tasks as explained in bitbake comment

Exactly.

> and the later one still doesn't list rm_worl task for quilt-native which is the real
> one I'm interested in.

That's the problem with recursive rm_work.

> When I do the same with last rm_work.bbclass change reverted I get this:
> $ head -n 12 log.zlib.rm_work.revert.debug3.prio.*
> ==> log.zlib.rm_work.revert.debug3.prio.completion <==
[...]
> Here it doesn't look so good for quilt-native:rm_work, it's included in zlib dependencies (thanks to the revert), but
> with "speed" scheduler it's far, while "completion" puts it early:
[...]

Here's how it looks with the patch:

# bitbake -n -D -D -D zlib | tee log.zlib.debug3.rm_work.patched
# less log.zlib.debug3.rm_work.patched
...
DEBUG: merged task list: ['do_fetch', 'do_unpack', 'do_patch', 'do_preconfigure', 'do_prepare_recipe_sysroot', 'do_configure', 'do_configure_ptest_base', 'do_compile', 'do_compile_ptest_base', 'do_gcc_stash_builddir', 'do_install', 'do_multilib_install', 'do_extra_symlinks', 'do_install_ptest_base', 'do_stash_locale', 'do_poststash_install_cleanup', 'do_package', 'do_packagedata', 'do_package_write_ipk', 'do_package_qa', 'do_populate_sysroot', 'do_populate_lic', 'do_rm_work', 'do_rm_work_all', 'do_build']
DEBUG: original priorities (most important first):
1. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch
2. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
3. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
4. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
5. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
6. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
7. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
8. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
9. ID /work/iot-ref-kit/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch
...
597. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_populate_lic
598. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/flex/flex_2.6.2.bb:do_rm_work
599. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/ncurses/ncurses_6.0+20161126.bb:do_rm_work
600. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-proto/xproto_7.0.31.bb:do_rm_work
601. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_rm_work
602. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/popt/popt_1.16.bb:do_rm_work
603. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_rm_work
604. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work
605. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work
606. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_rm_work
607. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/ossp-uuid/ossp-uuid_1.6.2.bb:do_rm_work
608. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.1.bb:do_rm_work
609. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/attr/acl_2.2.52.bb:do_rm_work
610. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
611. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work
...
DEBUG: completion priorities (most important first):
1. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build
2. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all
3. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
4. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/flex/flex_2.6.2.bb:do_rm_work
5. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/ncurses/ncurses_6.0+20161126.bb:do_rm_work
6. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-proto/xproto_7.0.31.bb:do_rm_work
7. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_rm_work
8. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/popt/popt_1.16.bb:do_rm_work
9. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_rm_work
10. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work
11. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work
12. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_rm_work
13. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/ossp-uuid/ossp-uuid_1.6.2.bb:do_rm_work
14. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.1.bb:do_rm_work
15. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/attr/acl_2.2.52.bb:do_rm_work
16. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
17. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work
18. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb:do_rm_work
...

That the speed scheduler (aka "original priorities" above) only adds
quilt-native_0.65.bb:do_rm_work at the end is a bit unexpected, I
thought it added all tasks of a recipe in sequence. As a result of that,
the reordered completion priorities do not have
quilt-native_0.65.bb:do_rm_work quite as high as it would be otherwise.

I don't think it matters in practice, though.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Patrick Ohly]

On Wed, 2017-03-01 at 16:52 +0100, Martin Jansa wrote:
> On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote:
> > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote:
> > > Are all changes necessary for this to work already in master?
> > 
> > Yes.
> > 
> > > Yesterday I've noticed that rm_work for some components which are
> > > early in the dependency (like qtbase) are executed relatively late
> > > (together with do_package_qa).
> > 
> > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know
> > that, and therefore schedules do_rm_work after do_package_qa.
> > 
> > If yes, then adding a list of tasks that can be ignored would be
> > trivial. This can be a variable, so a recipe can even add their own
> > ones, if necessary.
> 
> That's now what I've meant.
> 
> I believe that rm_work needs to be executed after do_package_qa, but I
> don't understand the scheduler code enough (at least not yet) to say
> that higher priority of rm_work task also makes all the tasks rm_work
> depends on e.g. do_package_qa to be executed sooner.
> 
> From my observation it looked like do_package_qa is still executed
> "late", but immediately followed by rm_work thanks to its high priority
> (so it's executed as soon as it can, but it's still late in progress of
> the whole build).
> 
> Another interesting test from today was to run:
> # rm -rf tmp-glibc/*
> # bitbake -n zlib | tee log.zlib.rm_work
> # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd ..
> # rm -rf tmp-glibc/*
> # bitbake -n zlib | tee log.zlib.rm_work.revert
> 
> and it shows interesting difference that many rm_work tasks aren't
> executed at all:
> 
> # grep rm_work log.zlib.rm_work* | grep zlib_
> log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all)
> 
> # grep rm_work log.zlib.rm_work* | grep gcc
> log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work)

I can reproduce that.

The root cause for the current behavior is that zlib:do_build does not
depend on zlib-native:do_build and thus does not enable
zlib-native:do_rm_work:

# bitbake -g zlib
# grep -e '->.*zlib-native' task-depends.dot | grep -v '^.zlib-native'
"binutils-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"rpm-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"gcc-cross-initial-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"binutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"gcc-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"file-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"pigz-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"elfutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"python-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"libpcre-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"

Only tasks that are required by the current build target(s) are
executed. From that perspective, the current behavior makes sense
(although it is of course not desirable).

More interesting is why it worked before. The answer is this:

# grep rm_work_all log.zlib.rm_work.revert 
NOTE: Running task 721 of 721 (/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all)
# grep -e '->.*zlib-native' task-depends.dot | grep -v '^.zlib-native'
"elfutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"binutils-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"pigz-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"gcc-cross-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"binutils-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"gcc-cross-initial-x86_64.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"python-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"rpm-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"zlib.do_rm_work_all" -> "zlib-native.do_rm_work"
"libpcre-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"
"file-native.do_prepare_recipe_sysroot" -> "zlib-native.do_populate_sysroot"

In other words, do_rm_work_all (the default task for "bitbake zlib" in
the previous approach) was responsible for triggering do_rm_work in
everything that zlib depended upon.

A side effect of that was that do_rm_work itself also caused more work
to be done than strictly necessary for the build:

# grep Running log.zlib.rm_work | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | wc -l
54
# grep Running log.zlib.rm_work.revert | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | wc -l
156

$ diff <(grep Running log.zlib.rm_work | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | sort) <(grep Running log.zlib.rm_work.revert | sed 's/ [0-9]* of [0-9]*//' | grep -v rm_work | grep -v 'noexec task' | sort)
0a1,11
> NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-connectivity/openssl/openssl_1.0.2k.bb:do_populate_lic_setscene)
> NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-connectivity/openssl/openssl_1.0.2k.bb:do_populate_sysroot_setscene)
> NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_populate_lic_setscene)
> NOTE: Running setscene task (virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_populate_sysroot_setscene)
...
> NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_package_qa_setscene)
20a94
> NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_populate_lic_setscene)
22a97
> NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc-initial_2.25.bb:do_populate_lic_setscene)
...
> NOTE: Running setscene task (/work/iot-ref-kit/openembedded-core/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb:do_populate_sysroot_setscene)

When rm_work.bbclass is not active, the old code did not execute these
tasks for "bitbake zlib".

Ideally, enabling rm_work.bbclass should trigger the recursive
do_rm_work (as before) but without triggering extra work (unlike
before). I just can't think of a way to do that - not with the kind of
dependencies we have in bitbake at the moment. We would have to
distinguish between "strong" dependencies (if A is active and depends on
B, then B must be executed first) and "weak" ordering (if A and B both
must run and A must run before B, then execute A first).

Given that most tasks triggered by do_rm_work on top of
do_populate_sysroot seem to be fairly light, enabling them in addition
to do_rm_work itself is probably better. Here's a tentative patch
achieving that:

# git diff
diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
index 054c93716e..eeb83d386a 100644
--- a/meta/classes/rm_work.bbclass
+++ b/meta/classes/rm_work.bbclass
@@ -102,6 +102,14 @@ do_rm_work () {
         rm -f $i
     done
 }
+do_rm_work_all () {
+    :
+}
+do_rm_work_all[recrdeptask] = "do_rm_work do_rm_work_all"
+do_rm_work_all[noexec] = "1"
+addtask rm_work_all after before do_build
+
+do_rm_work_all[recrdeptask] = "do_rm_work"
 
 do_populate_sdk[postfuncs] += "rm_work_populatesdk"
 rm_work_populatesdk () {
@@ -138,9 +146,8 @@ python inject_rm_work() {
         # do_build inherits additional runtime dependencies on
         # other recipes and thus will typically run much later than completion of
         # work in the recipe itself.
-        deps = bb.build.preceedtask('do_build', True, d)
-        if 'do_build' in deps:
-            deps.remove('do_build')
+        deps = set(bb.build.preceedtask('do_build', True, d))
+        deps.difference_update(('do_build', 'do_rm_work_all'))
         # In practice, addtask() here merely updates the dependencies.
         bb.build.addtask('do_rm_work', 'do_build', ' '.join(deps), d)
 }



# bitbake -n zlib | tee log.zlib.rm_work.patched
$ diff <(grep Running log.zlib.rm_work.patched | sed 's/ [0-9]* of [0-9]*//' | grep -v 'noexec task' | sort) <(grep Running log.zlib.rm_work.revert | sed 's/ [0-9]* of [0-9]*//' | grep -v 'noexec task' | sort)
202a203
> NOTE: Running task (/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all)

The only difference compared to before is because of the slight
do_rm_work_all[noexec] tweak.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

[PATCH 0/3] rm_work enhancements

[Patrick Ohly]

This is the bitbake side of the rm_work.bbclass enhancements. See the
OE-core "rm_work + pybootchart enhancements" mail thread for further
information.

Related-to: YOCTO #10584

The following changes since commit ae6045b84978940c365c95c33d6996359c3e299d:

  bb/cooker: BBCooker stops notifier at shutdown (2017-01-06 00:01:03 +0000)

are available in the git repository at:

  git://github.com/pohly/bitbake rmwork
  https://github.com/pohly/bitbake/tree/rmwork

Patrick Ohly (3):
  recipes: anonymous functions with priorities
  build.py: add preceedtask() API
  runqueue.py: alternative rm_work scheduler

 lib/bb/build.py      |  16 ++++++++
 lib/bb/data_smart.py |   3 ++
 lib/bb/parse/ast.py  |  19 +++++++--
 lib/bb/runqueue.py   | 107 +++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 142 insertions(+), 3 deletions(-)

-- 
2.1.4