From: Patrick Ohly <patrick.ohly@intel.com>
To: Martin Jansa <martin.jansa@gmail.com>
Cc: Patches and discussions about the oe-core layer
<openembedded-core@lists.openembedded.org>
Subject: Re: [PATCH 3/3] rm_work.bbclass: clean up sooner
Date: Thu, 02 Mar 2017 11:13:29 +0100 [thread overview]
Message-ID: <1488449609.7785.226.camel@intel.com> (raw)
In-Reply-To: <20170301184727.GB3290@jama>
On Wed, 2017-03-01 at 19:47 +0100, Martin Jansa wrote:
> Another (surprising for me) output from bitbake -D -D -D:
>
> OE qemux86@ ~/build/oe-core $ head log.zlib.rm_work.debug3.prio.*
> ==> log.zlib.rm_work.debug3.prio.completion <==
> DEBUG: completion priorities (most important first):
> 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build
> 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
> 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_package_write_ipk
> 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/expat/expat_2.2.0.bb:do_package_write_ipk
> 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_4.9.bb:do_package_write_ipk
> 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.6.bb:do_package_write_ipk
> 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/make/make_4.2.1.bb:do_package_write_ipk
> 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_package_write_ipk
> 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/gcc/libgcc_6.3.bb:do_package_write_ipk
>
> ==> log.zlib.rm_work.debug3.prio.normal <==
> DEBUG: original priorities (most important first):
> 1. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch
> 2. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
> 3. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
> 4. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
> 5. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
> 6. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
> 7. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
> 8. ID /OE/build/oe-core/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
> 9. ID /OE/build/oe-core/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch
>
> From these 2 I would expect the later one to be completion (to finish whole quilt-native before starting anything else
> except when all quilt-native tasks are blocked waiting for dependencies and BB_NUMBER_THREADS allows to run
> more task).
I've not fully understood what the surprise was. I suppose
log.zlib.rm_work.debug3.prio.normal was generated with
BB_SCHEDULER="speed" and log.zlib.rm_work.debug3.prio.completion with
BB_SCHEDULER="completion".
For speed, quilt-native comes first because it's the recipe that most
recipes depend on. For completion, it's not near the top because it has
none of "important" tasks. It should be the first recipe for
do_populate_sysroot. The patch from my other mail should activate
quilt-native_0.65.bb:do_rm_work and then I'd expect it also at the top
for completion.
> But I understand that this isn't the actual order of tasks as explained in bitbake comment
Exactly.
> and the later one still doesn't list rm_worl task for quilt-native which is the real
> one I'm interested in.
That's the problem with recursive rm_work.
> When I do the same with last rm_work.bbclass change reverted I get this:
> $ head -n 12 log.zlib.rm_work.revert.debug3.prio.*
> ==> log.zlib.rm_work.revert.debug3.prio.completion <==
[...]
> Here it doesn't look so good for quilt-native:rm_work, it's included in zlib dependencies (thanks to the revert), but
> with "speed" scheduler it's far, while "completion" puts it early:
[...]
Here's how it looks with the patch:
# bitbake -n -D -D -D zlib | tee log.zlib.debug3.rm_work.patched
# less log.zlib.debug3.rm_work.patched
...
DEBUG: merged task list: ['do_fetch', 'do_unpack', 'do_patch', 'do_preconfigure', 'do_prepare_recipe_sysroot', 'do_configure', 'do_configure_ptest_base', 'do_compile', 'do_compile_ptest_base', 'do_gcc_stash_builddir', 'do_install', 'do_multilib_install', 'do_extra_symlinks', 'do_install_ptest_base', 'do_stash_locale', 'do_poststash_install_cleanup', 'do_package', 'do_packagedata', 'do_package_write_ipk', 'do_package_qa', 'do_populate_sysroot', 'do_populate_lic', 'do_rm_work', 'do_rm_work_all', 'do_build']
DEBUG: original priorities (most important first):
1. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_fetch
2. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_unpack
3. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_patch
4. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_prepare_recipe_sysroot
5. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_configure
6. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_compile
7. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_install
8. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_populate_sysroot
9. ID /work/iot-ref-kit/openembedded-core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_fetch
...
597. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_populate_lic
598. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/flex/flex_2.6.2.bb:do_rm_work
599. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/ncurses/ncurses_6.0+20161126.bb:do_rm_work
600. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-proto/xproto_7.0.31.bb:do_rm_work
601. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_rm_work
602. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/popt/popt_1.16.bb:do_rm_work
603. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_rm_work
604. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work
605. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work
606. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_rm_work
607. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/ossp-uuid/ossp-uuid_1.6.2.bb:do_rm_work
608. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.1.bb:do_rm_work
609. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/attr/acl_2.2.52.bb:do_rm_work
610. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
611. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work
...
DEBUG: completion priorities (most important first):
1. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_build
2. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work_all
3. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
4. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/flex/flex_2.6.2.bb:do_rm_work
5. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/ncurses/ncurses_6.0+20161126.bb:do_rm_work
6. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-proto/xproto_7.0.31.bb:do_rm_work
7. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/glibc/glibc_2.25.bb:do_rm_work
8. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/popt/popt_1.16.bb:do_rm_work
9. ID /work/iot-ref-kit/openembedded-core/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_rm_work
10. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-extended/gperf/gperf_3.0.4.bb:do_rm_work
11. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/gcc/gcc-cross-initial_6.3.bb:do_rm_work
12. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/libtool/libtool-native_2.4.6.bb:do_rm_work
13. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/ossp-uuid/ossp-uuid_1.6.2.bb:do_rm_work
14. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-graphics/xorg-util/util-macros_1.19.1.bb:do_rm_work
15. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-support/attr/acl_2.2.52.bb:do_rm_work
16. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-core/zlib/zlib_1.2.8.bb:do_rm_work
17. ID /work/iot-ref-kit/openembedded-core/meta/recipes-devtools/quilt/quilt-native_0.65.bb:do_rm_work
18. ID virtual:native:/work/iot-ref-kit/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb:do_rm_work
...
That the speed scheduler (aka "original priorities" above) only adds
quilt-native_0.65.bb:do_rm_work at the end is a bit unexpected, I
thought it added all tasks of a recipe in sequence. As a result of that,
the reordered completion priorities do not have
quilt-native_0.65.bb:do_rm_work quite as high as it would be otherwise.
I don't think it matters in practice, though.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
next prev parent reply other threads:[~2017-03-02 10:13 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-06 9:55 [PATCH 0/3] rm_work enhancements Patrick Ohly
2017-01-06 9:55 ` [PATCH 1/3] gcc-source.inc: cleanly disable do_rm_work Patrick Ohly
2017-01-09 18:47 ` Khem Raj
2017-01-06 9:55 ` [PATCH 2/3] rm_work_and_downloads.bbclass: more aggressively minimize disk usage Patrick Ohly
2017-01-06 18:31 ` Randy Witt
2017-01-06 19:22 ` Patrick Ohly
2017-01-06 21:29 ` Randy Witt
2017-01-07 8:09 ` Patrick Ohly
2017-01-09 10:25 ` Patrick Ohly
2017-01-09 18:17 ` Randy Witt
2017-01-06 9:55 ` [PATCH 3/3] rm_work.bbclass: clean up sooner Patrick Ohly
2017-02-15 18:32 ` Martin Jansa
2017-02-16 10:26 ` Patrick Ohly
2017-03-01 15:52 ` Martin Jansa
2017-03-01 16:44 ` Patrick Ohly
2017-03-01 18:47 ` Martin Jansa
2017-03-02 10:13 ` Patrick Ohly [this message]
2017-03-02 9:38 ` Patrick Ohly
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1488449609.7785.226.camel@intel.com \
--to=patrick.ohly@intel.com \
--cc=martin.jansa@gmail.com \
--cc=openembedded-core@lists.openembedded.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox