[PATCH 0/2] Improve performance for native/cross/crosssdk's cleansstate

[PATCH 0/2] Improve performance for native/cross/crosssdk's cleansstate

[Robert Yang]

Hi RP and Ross,

These 2 patches can improve performance a lot for native/cross/crosssdk's
cleansstate:

This test is based on more than 600,000 sstate files
* Without disk caches
  # echo 3 >/proc/sys/vm/drop_caches
  $ bitbake -p
  $ time bitbake quilt-native -ccleansstate
  - Before:
    real    4m53.815s
    user    0m0.820s
    sys     0m0.128s

  - After
    real    0m58.483s
    user    0m0.744s
    sys     0m0.108s

  Saved 293 - 58 = 235s (80% improvement)

* With disk caches (26.6s -> 5.8s, 78% improvement)
  $ bitbake -p
  $ time bitbake quilt-native -ccleansstate
  - Before:
    real    0m26.682s
    user    0m0.748s
    sys     0m0.088s

  - After
    real    0m5.846s
    user    0m0.716s
    sys     0m0.116s

  Saved 26.6 - 5.8  = 20.8s (78% improvement)

======
I have another idea is put sstate files into the directory
named by the recipe, e.g.:

sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:773341447192feb6c1519cace0e6be35_populate_lic.tgz.siginfo
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:ab4a82a07b10926bb42d2f37242413c7_patch.tgz.siginfo
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:e8a4c952a66942653e36f289eaf68ca5_fetch.tgz.siginfo
sstate-cache/quilt-native/sstate:quilt-native::0.65:r0::3:fd35240c437cefa66e24cfe65d3fb335_unpack.tgz.siginfo

This can make it easier to manage sstate files, and can reduce the load of
glob.glob(), currently, the remove command is something like:

$ rm -f glob.glob(sstate-cache/*/sstate:foo.*bar.*)

If we put them into recipe named directory, then we can run:

$ rm -f glob.glob(sstate-cache/<recipe>/sstate:foo.*bar.*)

Which should be faster, what's your opinion, please ?

// Robert


The following changes since commit fc5418e7bbdecfb27bafe595084e0fd0f991a388:

  meta/icecc.bbclass: Update system blacklists (2018-11-21 11:48:18 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/sstate
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/sstate

Robert Yang (2):
  nopackages.bbclass: improve performance for cleansstate
  sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly

 meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------------
 meta/classes/sstate.bbclass     |  2 +-
 2 files changed, 24 insertions(+), 13 deletions(-)

-- 
2.7.4

[PATCH 1/2] nopackages.bbclass: improve performance for cleansstate

[Robert Yang]

Leave the task in SSTATETASKS doesn't make sense when it is removed, so also
remove it from SSTATETASKS.

This can improve the performance a lot for "bitbake <recipe-native/cross/crosssdk>
-ccleansstate" when there are a lot of sstate files.

For example:
* Before
  $ bitbake quilt-native -ccleansstate
  - Check log.do_cleansstate:
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz*
  Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*

  There are no package tasks for quilt-native, so the first 4 lines doesn't
  make any sense, but the glob pattern "sstate-cache/*/*" is very time
  consuming when there are no disk caches. E.g., I have more than 600,000
  sstate files:
  - Without disk caches
  # echo 3 >/proc/sys/vm/drop_caches
  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
    real    4m32.583s
    user    0m5.768s
    sys     0m12.892s

  - With disk caches (e.g., run it in the second time)
  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
    real    0m5.128s
    user    0m2.772s
    sys     0m2.308s

  So the 4 removing *package* commands cost more than 20s or 272s in theory.

* After
  $ bitbake quilt-native -ccleansstate
  - Check log.do_cleansstate:
  Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*

  Now the package tasks are gone when cleanssate for native recipes, which
  saved 20s or 272s in theory.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------------
 1 file changed, 23 insertions(+), 12 deletions(-)

diff --git a/meta/classes/nopackages.bbclass b/meta/classes/nopackages.bbclass
index 559f507..66e4936 100644
--- a/meta/classes/nopackages.bbclass
+++ b/meta/classes/nopackages.bbclass
@@ -1,12 +1,23 @@
-deltask do_package
-deltask do_package_write_rpm
-deltask do_package_write_ipk
-deltask do_package_write_deb
-deltask do_package_qa
-deltask do_packagedata
-deltask do_package_setscene
-deltask do_package_write_rpm_setscene
-deltask do_package_write_ipk_setscene
-deltask do_package_write_deb_setscene
-deltask do_package_qa_setscene
-deltask do_packagedata_setscene
+NO_PACKAGES_TASKS = " \
+    do_package \
+    do_package_write_rpm \
+    do_package_write_ipk \
+    do_package_write_deb \
+    do_package_qa \
+    do_packagedata \
+"
+
+python() {
+    sstatetasks = (d.getVar('SSTATETASKS') or '').split()
+    for task in d.getVar('NO_PACKAGES_TASKS').split():
+        setscene_task = '%s_setscene' % task
+        tasks = (task, setscene_task)
+        for t in tasks:
+            bb.build.deltask(t, d)
+
+        if task in sstatetasks:
+            bb.debug(2, 'Removing sstate task %s from SSTATETASKS' % task)
+            sstatetasks.remove(task)
+    d.setVar('SSTATETASKS', ' '.join(sstatetasks))
+
+}
-- 
2.7.4

Re: [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate

[Richard Purdie]

On Thu, 2018-11-22 at 19:51 +0800, Robert Yang wrote:
> Leave the task in SSTATETASKS doesn't make sense when it is removed,
> so also
> remove it from SSTATETASKS.
> 
> This can improve the performance a lot for "bitbake <recipe-
> native/cross/crosssdk>
> -ccleansstate" when there are a lot of sstate files.
> 
> For example:
> * Before
>   $ bitbake quilt-native -ccleansstate
>   - Check log.do_cleansstate:
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package_qa.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_packagedata.tgz*
>   Removing /sstate-cache/*/sstate:quilt-
> native::0.65:r0::3:*_populate_lic.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
> 
>   There are no package tasks for quilt-native, so the first 4 lines
> doesn't
>   make any sense, but the glob pattern "sstate-cache/*/*" is very
> time
>   consuming when there are no disk caches. E.g., I have more than
> 600,000
>   sstate files:
>   - Without disk caches
>   # echo 3 >/proc/sys/vm/drop_caches
>   $ time python3 -c 'import glob; glob.glob("/sstate-
> cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>     real    4m32.583s
>     user    0m5.768s
>     sys     0m12.892s
> 
>   - With disk caches (e.g., run it in the second time)
>   $ time python3 -c 'import glob; glob.glob("/sstate-
> cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>     real    0m5.128s
>     user    0m2.772s
>     sys     0m2.308s
> 
>   So the 4 removing *package* commands cost more than 20s or 272s in
> theory.
> 
> * After
>   $ bitbake quilt-native -ccleansstate
>   - Check log.do_cleansstate:
>   Removing /sstate-cache/*/sstate:quilt-
> native::0.65:r0::3:*_populate_lic.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
> 
>   Now the package tasks are gone when cleanssate for native recipes,
> which
>   saved 20s or 272s in theory.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------
> ------
>  1 file changed, 23 insertions(+), 12 deletions(-)
> 
> diff --git a/meta/classes/nopackages.bbclass
> b/meta/classes/nopackages.bbclass
> index 559f507..66e4936 100644
> --- a/meta/classes/nopackages.bbclass
> +++ b/meta/classes/nopackages.bbclass
> @@ -1,12 +1,23 @@
> -deltask do_package
> -deltask do_package_write_rpm
> -deltask do_package_write_ipk
> -deltask do_package_write_deb
> -deltask do_package_qa
> -deltask do_packagedata
> -deltask do_package_setscene
> -deltask do_package_write_rpm_setscene
> -deltask do_package_write_ipk_setscene
> -deltask do_package_write_deb_setscene
> -deltask do_package_qa_setscene
> -deltask do_packagedata_setscene
> +NO_PACKAGES_TASKS = " \
> +    do_package \
> +    do_package_write_rpm \
> +    do_package_write_ipk \
> +    do_package_write_deb \
> +    do_package_qa \
> +    do_packagedata \
> +"
> +
> +python() {
> +    sstatetasks = (d.getVar('SSTATETASKS') or '').split()
> +    for task in d.getVar('NO_PACKAGES_TASKS').split():
> +        setscene_task = '%s_setscene' % task
> +        tasks = (task, setscene_task)
> +        for t in tasks:
> +            bb.build.deltask(t, d)
> +
> +        if task in sstatetasks:
> +            bb.debug(2, 'Removing sstate task %s from SSTATETASKS' %
> task)
> +            sstatetasks.remove(task)
> +    d.setVar('SSTATETASKS', ' '.join(sstatetasks))
> +
> +}

I really don't want to turn more comparatively readable classes into a
mess of anonymous python if we can help it.

Can we not just teach cleansstate to check if a task exists before
trying to delete the sstate? It should be possible with
d.getVarFlag(taskname, "task")?

Cheers,

Richard

Re: [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate

[Yang, Liezhi]

Sent from mobile phone

> 在 2018年11月22日，20:15，Richard Purdie <richard.purdie@linuxfoundation.org> 写道：
> 
>> On Thu, 2018-11-22 at 19:51 +0800, Robert Yang wrote:
>> Leave the task in SSTATETASKS doesn't make sense when it is removed,
>> so also
>> remove it from SSTATETASKS.
>> 
>> This can improve the performance a lot for "bitbake <recipe-
>> native/cross/crosssdk>
>> -ccleansstate" when there are a lot of sstate files.
>> 
>> For example:
>> * Before
>>  $ bitbake quilt-native -ccleansstate
>>  - Check log.do_cleansstate:
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package_qa.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_packagedata.tgz*
>>  Removing /sstate-cache/*/sstate:quilt-
>> native::0.65:r0::3:*_populate_lic.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
>> 
>>  There are no package tasks for quilt-native, so the first 4 lines
>> doesn't
>>  make any sense, but the glob pattern "sstate-cache/*/*" is very
>> time
>>  consuming when there are no disk caches. E.g., I have more than
>> 600,000
>>  sstate files:
>>  - Without disk caches
>>  # echo 3 >/proc/sys/vm/drop_caches
>>  $ time python3 -c 'import glob; glob.glob("/sstate-
>> cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>>    real    4m32.583s
>>    user    0m5.768s
>>    sys     0m12.892s
>> 
>>  - With disk caches (e.g., run it in the second time)
>>  $ time python3 -c 'import glob; glob.glob("/sstate-
>> cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>>    real    0m5.128s
>>    user    0m2.772s
>>    sys     0m2.308s
>> 
>>  So the 4 removing *package* commands cost more than 20s or 272s in
>> theory.
>> 
>> * After
>>  $ bitbake quilt-native -ccleansstate
>>  - Check log.do_cleansstate:
>>  Removing /sstate-cache/*/sstate:quilt-
>> native::0.65:r0::3:*_populate_lic.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
>> 
>>  Now the package tasks are gone when cleanssate for native recipes,
>> which
>>  saved 20s or 272s in theory.
>> 
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>> meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------
>> ------
>> 1 file changed, 23 insertions(+), 12 deletions(-)
>> 
>> diff --git a/meta/classes/nopackages.bbclass
>> b/meta/classes/nopackages.bbclass
>> index 559f507..66e4936 100644
>> --- a/meta/classes/nopackages.bbclass
>> +++ b/meta/classes/nopackages.bbclass
>> @@ -1,12 +1,23 @@
>> -deltask do_package
>> -deltask do_package_write_rpm
>> -deltask do_package_write_ipk
>> -deltask do_package_write_deb
>> -deltask do_package_qa
>> -deltask do_packagedata
>> -deltask do_package_setscene
>> -deltask do_package_write_rpm_setscene
>> -deltask do_package_write_ipk_setscene
>> -deltask do_package_write_deb_setscene
>> -deltask do_package_qa_setscene
>> -deltask do_packagedata_setscene
>> +NO_PACKAGES_TASKS = " \
>> +    do_package \
>> +    do_package_write_rpm \
>> +    do_package_write_ipk \
>> +    do_package_write_deb \
>> +    do_package_qa \
>> +    do_packagedata \
>> +"
>> +
>> +python() {
>> +    sstatetasks = (d.getVar('SSTATETASKS') or '').split()
>> +    for task in d.getVar('NO_PACKAGES_TASKS').split():
>> +        setscene_task = '%s_setscene' % task
>> +        tasks = (task, setscene_task)
>> +        for t in tasks:
>> +            bb.build.deltask(t, d)
>> +
>> +        if task in sstatetasks:
>> +            bb.debug(2, 'Removing sstate task %s from SSTATETASKS' %
>> task)
>> +            sstatetasks.remove(task)
>> +    d.setVar('SSTATETASKS', ' '.join(sstatetasks))
>> +
>> +}
> 
> I really don't want to turn more comparatively readable classes into a
> mess of anonymous python if we can help it.
> 
> Can we not just teach cleansstate to check if a task exists before
> trying to delete the sstate? It should be possible with
> d.getVarFlag(taskname, "task")?

Thanks, yes, this is my first thought, but I only knew __BBTASKS which is an internal var of bitbake, I will use d.getVarFlag(taskname, "task") to send a v2.

// Robert


> 
> Cheers,
> 
> Richard
> 

[PATCH 2/2] sstate.bbclass: set SSTATE_EXTRAPATHWILDCARD explicitly

[Robert Yang]

The glob.glob("/sstate/*/*/") is very time consuming, set
SSTATE_EXTRAPATHWILDCARD explicity to avoid that. This can save a lot of time
when there are many sstate files.

For example, I have more than 600,000 sstate files:
* Before
  - Without disk caches
  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
    real    4m32.583s
    user    0m5.768s
    sys     0m12.892s

  - With disk caches
  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
  real    0m4.111s
  user    0m2.348s
  sys     0m1.756s

* After
  $ time python3 -c 'import glob; glob.glob("/sstate-cache.bak/universal/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*")'
  - Without disk caches:
  real    0m7.928s
  user    0m0.172s
  sys     0m0.124s

  - With disk caches:
  real    0m0.131s
  user    0m0.088s
  sys     0m0.044s

We can see that it saves about 3.8s with disk caches, and saves about 264s
without disk caches.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/sstate.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index efb0096..94fde6d 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -101,7 +101,7 @@ python () {
     if bb.data.inherits_class('native', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross', d):
         d.setVar('SSTATE_EXTRAPATH', "${NATIVELSBSTRING}/")
         d.setVar('BB_HASHFILENAME', "True ${SSTATE_PKGSPEC} ${SSTATE_SWSPEC}")
-        d.setVar('SSTATE_EXTRAPATHWILDCARD', "*/")
+        d.setVar('SSTATE_EXTRAPATHWILDCARD', "${NATIVELSBSTRING}/")
 
     unique_tasks = sorted(set((d.getVar('SSTATETASKS') or "").split()))
     d.setVar('SSTATETASKS', " ".join(unique_tasks))
-- 
2.7.4