Re: [PATCH 3/3] rm_work.bbclass: clean up sooner

[Patrick Ohly <patrick.ohly@intel.com>]

On Wed, 2017-03-01 at 16:52 +0100, Martin Jansa wrote:
> On Thu, Feb 16, 2017 at 11:26:54AM +0100, Patrick Ohly wrote:
> > On Wed, 2017-02-15 at 19:32 +0100, Martin Jansa wrote:
> > > Are all changes necessary for this to work already in master?
> > 
> > Yes.
> > 
> > > Yesterday I've noticed that rm_work for some components which are
> > > early in the dependency (like qtbase) are executed relatively late
> > > (together with do_package_qa).
> > 
> > Could do_rm_work run before do_package_qa? rm_work.bbclass doesn't know
> > that, and therefore schedules do_rm_work after do_package_qa.
> > 
> > If yes, then adding a list of tasks that can be ignored would be
> > trivial. This can be a variable, so a recipe can even add their own
> > ones, if necessary.
> 
> That's now what I've meant.
> 
> I believe that rm_work needs to be executed after do_package_qa, but I
> don't understand the scheduler code enough (at least not yet) to say
> that higher priority of rm_work task also makes all the tasks rm_work
> depends on e.g. do_package_qa to be executed sooner.

I see. do_package_qa should have a high priority, but it's still blocked
by its dependencies. Building those dependencies only gets an indirect
boost from scheduling recipes that many other recipes depend on first (a
feature of the normal scheduler).

I suspect the problem with do_package_qa is similar to the problem with
do_build before: it depends on do_package_qa of everything a recipe
depends on, and thus finishing the build of those other recipes also
blocks finishing the current recipe. That creates very deep dependency
chains and thus increases the number of recipes which are "in progress"
at the same time.

> Another interesting test from today was to run:
> # rm -rf tmp-glibc/*
> # bitbake -n zlib | tee log.zlib.rm_work
> # cd oe-core; git revert -1 936179754c8d0f98e1196ddc6796fdfd72c0c3b4; cd ..
> # rm -rf tmp-glibc/*
> # bitbake -n zlib | tee log.zlib.rm_work.revert
> 
> and it shows interesting difference that many rm_work tasks aren't
> executed at all:
> 
> # grep rm_work log.zlib.rm_work* | grep zlib_
> log.zlib.rm_work:NOTE: Running task 526 of 527 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 128 of 721 (virtual:native:/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 717 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 721 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all)
> 
> # grep rm_work log.zlib.rm_work* | grep gcc
> log.zlib.rm_work.revert:NOTE: Running task 2 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-source_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 240 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 250 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc-initial_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 634 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-cross_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 674 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_rm_work)
> log.zlib.rm_work.revert:NOTE: Running task 678 of 721 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/gcc-runtime_6.3.bb:do_rm_work)
> 
> # grep -c rm_work log.zlib.rm_work*
> log.zlib.rm_work:1
> log.zlib.rm_work.revert:63
> 
> I'll check if it's something in my setup or if this happens everywhere now.

I'll try to double-check this myself, too.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.