* [PATCH 0/2] Improve performance for native/cross/crosssdk's cleansstate
@ 2018-11-22 11:51 Robert Yang
2018-11-22 11:51 ` [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate Robert Yang
2018-11-22 11:51 ` [PATCH 2/2] sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly Robert Yang
0 siblings, 2 replies; 5+ messages in thread
From: Robert Yang @ 2018-11-22 11:51 UTC (permalink / raw)
To: openembedded-core
Hi RP and Ross,
These 2 patches can improve performance a lot for native/cross/crosssdk's
cleansstate:
This test is based on more than 600,000 sstate files
* Without disk caches
# echo 3 >/proc/sys/vm/drop_caches
$ bitbake -p
$ time bitbake quilt-native -ccleansstate
- Before:
real 4m53.815s
user 0m0.820s
sys 0m0.128s
- After
real 0m58.483s
user 0m0.744s
sys 0m0.108s
Saved 293 - 58 = 235s (80% improvement)
* With disk caches (26.6s -> 5.8s, 78% improvement)
$ bitbake -p
$ time bitbake quilt-native -ccleansstate
- Before:
real 0m26.682s
user 0m0.748s
sys 0m0.088s
- After
real 0m5.846s
user 0m0.716s
sys 0m0.116s
Saved 26.6 - 5.8 = 20.8s (78% improvement)
======
I have another idea is put sstate files into the directory
named by the recipe, e.g.:
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz.siginfo
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:ab4a82a07b10926bb42d2f37242413c7_patch.tgz.siginfo
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:e8a4c952a66942653e36f289eaf68ca5_fetch.tgz.siginfo
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:fd35240c437cefa66e24cfe65d3fb335_unpack.tgz.siginfo
This can make it easier to manage sstate files, and can reduce the load of
glob.glob(), currently, the remove command is something like:
$ rm -f glob.glob(sstate-cache/*/sstate:foo.*bar.*)
If we put them into recipe named directory, then we can run:
$ rm -f glob.glob(sstate-cache/<recipe>/sstate:foo.*bar.*)
Which should be faster, what's your opinion, please ?
// Robert
The following changes since commit fc5418e7bbdecfb27bafe595084e0fd0f991a388:
meta/icecc.bbclass: Update system blacklists (2018-11-21 11:48:18 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/sstate
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/sstate
Robert Yang (2):
nopackages.bbclass: improve performance for cleansstate
sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly
meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------------
meta/classes/sstate.bbclass | 2 +-
2 files changed, 24 insertions(+), 13 deletions(-)
--
2.7.4
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate 2018-11-22 11:51 [PATCH 0/2] Improve performance for native/cross/crosssdk's cleansstate Robert Yang @ 2018-11-22 11:51 ` Robert Yang 2018-11-22 12:15 ` Richard Purdie 2018-11-22 11:51 ` [PATCH 2/2] sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly Robert Yang 1 sibling, 1 reply; 5+ messages in thread From: Robert Yang @ 2018-11-22 11:51 UTC (permalink / raw) To: openembedded-core Leave the task in SSTATETASKS doesn't make sense when it is removed, so also remove it from SSTATETASKS. This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk> -ccleansstate" when there are a lot of sstate files. For example: * Before $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz* Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* There are no package tasks for quilt-native, so the first 4 lines doesn't make any sense, but the glob pattern "sstate-cache/*/*" is very time consuming when there are no disk caches. E.g., I have more than 600,000 sstate files: - Without disk caches # echo 3 >/proc/sys/vm/drop_caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches (e.g., run it in the second time) $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")' real 0m5.128s user 0m2.772s sys 0m2.308s So the 4 removing *package* commands cost more than 20s or 272s in theory. * After $ bitbake quilt-native -ccleansstate - Check log.do_cleansstate: Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz* Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* Now the package tasks are gone when cleanssate for native recipes, which saved 20s or 272s in theory. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/meta/classes/nopackages.bbclass b/meta/classes/nopackages.bbclass index 559f507..66e4936 100644 --- a/meta/classes/nopackages.bbclass +++ b/meta/classes/nopackages.bbclass @@ -1,12 +1,23 @@ -deltask do_package -deltask do_package_write_rpm -deltask do_package_write_ipk -deltask do_package_write_deb -deltask do_package_qa -deltask do_packagedata -deltask do_package_setscene -deltask do_package_write_rpm_setscene -deltask do_package_write_ipk_setscene -deltask do_package_write_deb_setscene -deltask do_package_qa_setscene -deltask do_packagedata_setscene +NO_PACKAGES_TASKS = " \ + do_package \ + do_package_write_rpm \ + do_package_write_ipk \ + do_package_write_deb \ + do_package_qa \ + do_packagedata \ +" + +python() { + sstatetasks = (d.getVar('SSTATETASKS') or '').split() + for task in d.getVar('NO_PACKAGES_TASKS').split(): + setscene_task = '%s_setscene' % task + tasks = (task, setscene_task) + for t in tasks: + bb.build.deltask(t, d) + + if task in sstatetasks: + bb.debug(2, 'Removing sstate task %s from SSTATETASKS' % task) + sstatetasks.remove(task) + d.setVar('SSTATETASKS', ' '.join(sstatetasks)) + +} -- 2.7.4 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate 2018-11-22 11:51 ` [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate Robert Yang @ 2018-11-22 12:15 ` Richard Purdie 2018-11-22 12:47 ` Yang, Liezhi 0 siblings, 1 reply; 5+ messages in thread From: Richard Purdie @ 2018-11-22 12:15 UTC (permalink / raw) To: Robert Yang, openembedded-core On Thu, 2018-11-22 at 19:51 +0800, Robert Yang wrote: > Leave the task in SSTATETASKS doesn't make sense when it is removed, > so also > remove it from SSTATETASKS. > > This can improve the performance a lot for "bitbake <recipe- > native/cross/crosssdk> > -ccleansstate" when there are a lot of sstate files. > > For example: > * Before > $ bitbake quilt-native -ccleansstate > - Check log.do_cleansstate: > Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_package.tgz* > Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_package_qa.tgz* > Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* > Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_packagedata.tgz* > Removing /sstate-cache/*/sstate:quilt- > native::0.65:r0::3:*_populate_lic.tgz* > Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* > > There are no package tasks for quilt-native, so the first 4 lines > doesn't > make any sense, but the glob pattern "sstate-cache/*/*" is very > time > consuming when there are no disk caches. E.g., I have more than > 600,000 > sstate files: > - Without disk caches > # echo 3 >/proc/sys/vm/drop_caches > $ time python3 -c 'import glob; glob.glob("/sstate- > cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_package.tgz*")' > real 4m32.583s > user 0m5.768s > sys 0m12.892s > > - With disk caches (e.g., run it in the second time) > $ time python3 -c 'import glob; glob.glob("/sstate- > cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_package.tgz*")' > real 0m5.128s > user 0m2.772s > sys 0m2.308s > > So the 4 removing *package* commands cost more than 20s or 272s in > theory. > > * After > $ bitbake quilt-native -ccleansstate > - Check log.do_cleansstate: > Removing /sstate-cache/*/sstate:quilt- > native::0.65:r0::3:*_populate_lic.tgz* > Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- > linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* > > Now the package tasks are gone when cleanssate for native recipes, > which > saved 20s or 272s in theory. > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------ > ------ > 1 file changed, 23 insertions(+), 12 deletions(-) > > diff --git a/meta/classes/nopackages.bbclass > b/meta/classes/nopackages.bbclass > index 559f507..66e4936 100644 > --- a/meta/classes/nopackages.bbclass > +++ b/meta/classes/nopackages.bbclass > @@ -1,12 +1,23 @@ > -deltask do_package > -deltask do_package_write_rpm > -deltask do_package_write_ipk > -deltask do_package_write_deb > -deltask do_package_qa > -deltask do_packagedata > -deltask do_package_setscene > -deltask do_package_write_rpm_setscene > -deltask do_package_write_ipk_setscene > -deltask do_package_write_deb_setscene > -deltask do_package_qa_setscene > -deltask do_packagedata_setscene > +NO_PACKAGES_TASKS = " \ > + do_package \ > + do_package_write_rpm \ > + do_package_write_ipk \ > + do_package_write_deb \ > + do_package_qa \ > + do_packagedata \ > +" > + > +python() { > + sstatetasks = (d.getVar('SSTATETASKS') or '').split() > + for task in d.getVar('NO_PACKAGES_TASKS').split(): > + setscene_task = '%s_setscene' % task > + tasks = (task, setscene_task) > + for t in tasks: > + bb.build.deltask(t, d) > + > + if task in sstatetasks: > + bb.debug(2, 'Removing sstate task %s from SSTATETASKS' % > task) > + sstatetasks.remove(task) > + d.setVar('SSTATETASKS', ' '.join(sstatetasks)) > + > +} I really don't want to turn more comparatively readable classes into a mess of anonymous python if we can help it. Can we not just teach cleansstate to check if a task exists before trying to delete the sstate? It should be possible with d.getVarFlag(taskname, "task")? Cheers, Richard ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate 2018-11-22 12:15 ` Richard Purdie @ 2018-11-22 12:47 ` Yang, Liezhi 0 siblings, 0 replies; 5+ messages in thread From: Yang, Liezhi @ 2018-11-22 12:47 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core@lists.openembedded.org Sent from mobile phone > 在 2018年11月22日，20:15，Richard Purdie <richard.purdie@linuxfoundation.org> 写道： > >> On Thu, 2018-11-22 at 19:51 +0800, Robert Yang wrote: >> Leave the task in SSTATETASKS doesn't make sense when it is removed, >> so also >> remove it from SSTATETASKS. >> >> This can improve the performance a lot for "bitbake <recipe- >> native/cross/crosssdk> >> -ccleansstate" when there are a lot of sstate files. >> >> For example: >> * Before >> $ bitbake quilt-native -ccleansstate >> - Check log.do_cleansstate: >> Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_package.tgz* >> Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_package_qa.tgz* >> Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz* >> Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_packagedata.tgz* >> Removing /sstate-cache/*/sstate:quilt- >> native::0.65:r0::3:*_populate_lic.tgz* >> Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* >> >> There are no package tasks for quilt-native, so the first 4 lines >> doesn't >> make any sense, but the glob pattern "sstate-cache/*/*" is very >> time >> consuming when there are no disk caches. E.g., I have more than >> 600,000 >> sstate files: >> - Without disk caches >> # echo 3 >/proc/sys/vm/drop_caches >> $ time python3 -c 'import glob; glob.glob("/sstate- >> cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_package.tgz*")' >> real 4m32.583s >> user 0m5.768s >> sys 0m12.892s >> >> - With disk caches (e.g., run it in the second time) >> $ time python3 -c 'import glob; glob.glob("/sstate- >> cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_package.tgz*")' >> real 0m5.128s >> user 0m2.772s >> sys 0m2.308s >> >> So the 4 removing *package* commands cost more than 20s or 272s in >> theory. >> >> * After >> $ bitbake quilt-native -ccleansstate >> - Check log.do_cleansstate: >> Removing /sstate-cache/*/sstate:quilt- >> native::0.65:r0::3:*_populate_lic.tgz* >> Removing /sstate-cache/*/*/sstate:quilt-native:x86_64- >> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz* >> >> Now the package tasks are gone when cleanssate for native recipes, >> which >> saved 20s or 272s in theory. >> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >> --- >> meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------ >> ------ >> 1 file changed, 23 insertions(+), 12 deletions(-) >> >> diff --git a/meta/classes/nopackages.bbclass >> b/meta/classes/nopackages.bbclass >> index 559f507..66e4936 100644 >> --- a/meta/classes/nopackages.bbclass >> +++ b/meta/classes/nopackages.bbclass >> @@ -1,12 +1,23 @@ >> -deltask do_package >> -deltask do_package_write_rpm >> -deltask do_package_write_ipk >> -deltask do_package_write_deb >> -deltask do_package_qa >> -deltask do_packagedata >> -deltask do_package_setscene >> -deltask do_package_write_rpm_setscene >> -deltask do_package_write_ipk_setscene >> -deltask do_package_write_deb_setscene >> -deltask do_package_qa_setscene >> -deltask do_packagedata_setscene >> +NO_PACKAGES_TASKS = " \ >> + do_package \ >> + do_package_write_rpm \ >> + do_package_write_ipk \ >> + do_package_write_deb \ >> + do_package_qa \ >> + do_packagedata \ >> +" >> + >> +python() { >> + sstatetasks = (d.getVar('SSTATETASKS') or '').split() >> + for task in d.getVar('NO_PACKAGES_TASKS').split(): >> + setscene_task = '%s_setscene' % task >> + tasks = (task, setscene_task) >> + for t in tasks: >> + bb.build.deltask(t, d) >> + >> + if task in sstatetasks: >> + bb.debug(2, 'Removing sstate task %s from SSTATETASKS' % >> task) >> + sstatetasks.remove(task) >> + d.setVar('SSTATETASKS', ' '.join(sstatetasks)) >> + >> +} > > I really don't want to turn more comparatively readable classes into a > mess of anonymous python if we can help it. > > Can we not just teach cleansstate to check if a task exists before > trying to delete the sstate? It should be possible with > d.getVarFlag(taskname, "task")? Thanks, yes, this is my first thought, but I only knew __BBTASKS which is an internal var of bitbake, I will use d.getVarFlag(taskname, "task") to send a v2. // Robert > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly 2018-11-22 11:51 [PATCH 0/2] Improve performance for native/cross/crosssdk's cleansstate Robert Yang 2018-11-22 11:51 ` [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate Robert Yang @ 2018-11-22 11:51 ` Robert Yang 1 sibling, 0 replies; 5+ messages in thread From: Robert Yang @ 2018-11-22 11:51 UTC (permalink / raw) To: openembedded-core The glob.glob("/sstate/*/*/") is very time consuming, set SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time when there are many sstate files. For example, I have more than 600,000 sstate files: * Before - Without disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 4m32.583s user 0m5.768s sys 0m12.892s - With disk caches $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' real 0m4.111s user 0m2.348s sys 0m1.756s * After $ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")' - Without disk caches: real 0m7.928s user 0m0.172s sys 0m0.124s - With disk caches: real 0m0.131s user 0m0.088s sys 0m0.044s We can see that it saves about 3.8s with disk caches, and saves about 264s without disk caches. Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/classes/sstate.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass index efb0096..94fde6d 100644 --- a/meta/classes/sstate.bbclass +++ b/meta/classes/sstate.bbclass @@ -101,7 +101,7 @@ python () { if bb.data.inherits_class('native', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross', d): d.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/") d.setVar('BB_HASHFILENAME', "True ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}") - d.setVar('SSTATE_EXTRAPATHWILDCARD', "*/") + d.setVar('SSTATE_EXTRAPATHWILDCARD', "${NATIVELSBSTRING}/") unique_tasks = sorted(set((d.getVar('SSTATETASKS') or "").split())) d.setVar('SSTATETASKS', " ".join(unique_tasks)) -- 2.7.4 ^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-11-22 12:47 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2018-11-22 11:51 [PATCH 0/2] Improve performance for native/cross/crosssdk's cleansstate Robert Yang 2018-11-22 11:51 ` [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate Robert Yang 2018-11-22 12:15 ` Richard Purdie 2018-11-22 12:47 ` Yang, Liezhi 2018-11-22 11:51 ` [PATCH 2/2] sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly Robert Yang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox