[PATCH 0/4] sstate-cache-management fixes

[PATCH 0/4] sstate-cache-management fixes

[Martin Jansa]

Removing do_package was reported by Jacob Kroon and then I've found
few more issues, the most surprising is the one with populate_lic.

The following changes since commit b16de9af5c7108396a347e7107c73608a2f8d74f:

  packagegroup-toolset-native: Update after ocf-linux -> cryptodev-linux change (2014-03-27 19:53:07 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib jansa/sstate-management
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/sstate-management

Martin Jansa (4):
  sstate-cache-management: rm_by_stamps - include signatures of
    .sigdata. files
  sstate-cache-management: rm_by_stamps - remove .siginfo and .done
    files
  sstate-cache-management: skip populate_lic archives when removing
    duplicates
  sstate-cache-management: rm_by_stamps - discover all suffixes like
    remove_duplicated does

 scripts/sstate-cache-management.sh | 34 ++++++++++++++++++++--------------
 1 file changed, 20 insertions(+), 14 deletions(-)

-- 
1.9.1

[PATCH 1/4] sstate-cache-management: rm_by_stamps - include signatures of .sigdata. files

[Martin Jansa]

* there are .siginfo files in sstate-cache, but STAMPS_DIR calls them
  .sigdata, make sure that such signatures are kept, because some tasks
  like do_package, don't have _setscene or main task entry and are
  removed:

  stamps:
  tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_package.sigdata.cd6f625471ef1b20a9379e90519db6f1
  tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_package_write_ipk.f20fe66285219f23373fc64d5de1d412
  tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_package_write_ipk.sigdata.f20fe66285219f23373fc64d5de1d412
  tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_packagedata.sigdata.c55a3dbe90010c15aa3294753fbd402e
  tmp-eglibc/stamps/i586-oe-linux/busybox/1.22.1-r0.do_packagedata_setscene.c55a3dbe90010c15aa3294753fbd402e.qemux86
  sstate (we want to keep all in this case):
  sstate-cache/c5/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:c55a3dbe90010c15aa3294753fbd402e_packagedata.tgz
  sstate-cache/c5/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:c55a3dbe90010c15aa3294753fbd402e_packagedata.tgz.siginfo
  sstate-cache/cd/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:cd6f625471ef1b20a9379e90519db6f1_package.tgz
  sstate-cache/cd/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:cd6f625471ef1b20a9379e90519db6f1_package.tgz.siginfo
  sstate-cache/f2/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:f20fe66285219f23373fc64d5de1d412_package_write_ipk.tgz
  sstate-cache/f2/sstate:busybox:i586-oe-linux:1.22.1:r0:i586:3:f20fe66285219f23373fc64d5de1d412_package_write_ipk.tgz.siginfo

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index d741ec9..34bde01 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -330,11 +330,11 @@ rm_by_stamps (){
   # Figure out all the md5sums in the stamps dir.
   echo "Figuring out all the md5sums in stamps dir ... "
   for i in $suffixes; do
-      # There is no "\.siginfo" but "_setcene" when it is mirrored
+      # There is no "\.sigdata" but "_setcene" when it is mirrored
       # from the SSTATE_MIRRORS, use them to figure out the sum.
       sums=`find $stamps -maxdepth 3 -name "*.do_$i.*" \
         -o -name "*.do_${i}_setscene.*" | \
-        sed -ne 's#.*_setscene\.##p' -e 's#.*\.siginfo\.##p' | \
+        sed -ne 's#.*_setscene\.##p' -e 's#.*\.sigdata\.##p' | \
         sed -e 's#\..*##' | sort -u`
       all_sums="$all_sums $sums"
   done
-- 
1.9.1

[PATCH 2/4] sstate-cache-management: rm_by_stamps - remove .siginfo and .done files

[Martin Jansa]

* it's possible that corresponding .tgz files were already removed
  (e.g. with -d option and older version of this script) and this
  won't find orphaned .siginfo or .done files to remove
* add sort -u to count files found multiple times only once

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index 34bde01..ffda05e 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -341,9 +341,9 @@ rm_by_stamps (){
   echo "Done"
 
   # Total number of files including sstate-, .siginfo and .done files
-  total_files=`find $cache_dir -name 'sstate*' | wc -l`
+  total_files=`find $cache_dir -type f -name 'sstate*' | wc -l`
   # Save all the state file list to a file
-  find $cache_dir -name 'sstate*.tgz' | sort -u -o $cache_list
+  find $cache_dir -type f -name 'sstate*' | sort -u -o $cache_list
 
   echo "Figuring out the files which will be removed ... "
   for i in $all_sums; do
@@ -355,14 +355,14 @@ rm_by_stamps (){
       sort -u $keep_list -o $keep_list
       to_del=`comm -1 -3 $keep_list $cache_list`
       gen_rmlist $rm_list "$to_del"
-      let total_deleted=`cat $rm_list | wc -w`
+      let total_deleted=`cat $rm_list | sort -u | wc -w`
       if [ $total_deleted -gt 0 ]; then
-          [ $debug -gt 0 ] && cat $rm_list
+          [ $debug -gt 0 ] && cat $rm_list | sort -u
           read_confirm
           if [ "$confirm" = "y" -o "$confirm" = "Y" ]; then
               echo "Removing sstate cache files ... ($total_deleted files)"
               # Remove them one by one to avoid the argument list too long error
-              for i in `cat $rm_list`; do
+              for i in `cat $rm_list | sort -u`; do
                   rm -f $verbose $i
               done
               echo "$total_deleted files have been removed"
-- 
1.9.1

[PATCH 3/4] sstate-cache-management: skip populate_lic archives when removing duplicates

[Martin Jansa]

* there are 2 issues

* all *gcc* recipes produce "duplicate" files and we cannot keep just the newest one:
  OE qemux86@ ~/build/oe-core $ find sstate-cache/ | grep gcc.*populate_lic
  sstate-cache/cd/sstate:gcc::4.8.2:r0::3:cd8d1a33b832662b624f5e4bd9f55154_populate_lic.tgz
  sstate-cache/cd/sstate:gcc::4.8.2:r0::3:cd8d1a33b832662b624f5e4bd9f55154_populate_lic.tgz.siginfo
  sstate-cache/d5/sstate:gcc::4.8.2:r0::3:d591af812395cc386e017fa2c8fa0722_populate_lic.tgz
  sstate-cache/d5/sstate:gcc::4.8.2:r0::3:d591af812395cc386e017fa2c8fa0722_populate_lic.tgz.siginfo
  sstate-cache/f7/sstate:gcc::4.8.2:r0::3:f74cfd4982fd7c349a5418f07297f955_populate_lic.tgz
  sstate-cache/f7/sstate:gcc::4.8.2:r0::3:f74cfd4982fd7c349a5418f07297f955_populate_lic.tgz.siginfo
  sstate-cache/74/sstate:gcc::4.8.2:r0::3:7489e4502cb446506b9829cda13c7630_populate_lic.tgz
  sstate-cache/74/sstate:gcc::4.8.2:r0::3:7489e4502cb446506b9829cda13c7630_populate_lic.tgz.siginfo

  OE qemux86@ ~/build/oe-core $ find tmp-eglibc/stamps/ | grep gcc.*populate_lic
  tmp-eglibc/stamps/i586-oe-linux/gcc-cross-initial/4.8.2-r0.do_populate_lic_setscene.cd8d1a33b832662b624f5e4bd9f55154
  tmp-eglibc/stamps/i586-oe-linux/libgcc/4.8.2-r0.do_populate_lic_setscene.f74cfd4982fd7c349a5418f07297f955
  tmp-eglibc/stamps/i586-oe-linux/gcc-cross/4.8.2-r0.do_populate_lic_setscene.d591af812395cc386e017fa2c8fa0722
  tmp-eglibc/stamps/i586-oe-linux/gcc-runtime/4.8.2-r0.do_populate_lic_setscene.7489e4502cb446506b9829cda13c7630

* the same for target and native versions of the same recipe:
  OE qemux86@ ~/build/oe-core $ find tmp-eglibc/stamps/ | grep opkg-utils.*populate_lic
  tmp-eglibc/stamps/x86_64-linux/opkg-utils-native/0.1.8+gitAUTOINC+c33b217016-r0.do_populate_lic_setscene.dd21a3c5444482ce90be4c9a33d806f1
  tmp-eglibc/stamps/i586-oe-linux/opkg-utils/0.1.8+gitAUTOINC+c33b217016-r0.do_populate_lic_setscene.8571422f9e311dc41bb6b21e71a09bc0

  OE qemux86@ ~/build/oe-core $ find sstate-cache/ | grep opkg-utils.*populate_lic
  sstate-cache/dd/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:dd21a3c5444482ce90be4c9a33d806f1_populate_lic.tgz.siginfo
  sstate-cache/dd/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:dd21a3c5444482ce90be4c9a33d806f1_populate_lic.tgz
  sstate-cache/85/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:8571422f9e311dc41bb6b21e71a09bc0_populate_lic.tgz.siginfo
  sstate-cache/85/sstate:opkg-utils::0.1.8+gitAUTOINC+c33b217016:r0::3:8571422f9e311dc41bb6b21e71a09bc0_populate_lic.tgz

* similar problem for .siginfo file of do_patch:
  OE qemux86@ ~/build/oe-core $ find sstate-cache/ | grep eglibc.*patch
  sstate-cache/99/sstate:eglibc::2.19:r0::3:99b7e1d688e4c6c659b458c9c57611df_patch.tgz.siginfo
  sstate-cache/11/sstate:eglibc::2.19:r0::3:11401ddd208a753fee17696b14dc1e52_patch.tgz.siginfo

  OE qemux86@ ~/build/oe-core $ find tmp-eglibc/stamps/  | grep /eglibc.*patch
  tmp-eglibc/stamps/i586-oe-linux/eglibc-initial/2.19-r0.do_patch.sigdata.99b7e1d688e4c6c659b458c9c57611df
  tmp-eglibc/stamps/i586-oe-linux/eglibc/2.19-r0.do_patch.sigdata.11401ddd208a753fee17696b14dc1e52

* maybe these should have ideally identical signatures, but before it's fixed lets
  just skip removing them

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index ffda05e..77afd68 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -225,6 +225,10 @@ remove_duplicated () {
   # Save the file list which needs to be removed
   local remove_listdir=`mktemp -d` || exit 1
   for suffix in $sstate_suffixes; do
+      if [ "$suffix" = "populate_lic" ] ; then
+          echo "Skipping populate_lic, because removing duplicates doesn't work correctly for them (use --stamps-dir instead)"
+          continue
+      fi
       # Total number of files including .siginfo and .done files
       total_files_suffix=`grep ".*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:_]*_$suffix\.tgz.*" $sstate_files_list | wc -l 2>/dev/null`
       total_tgz_suffix=`grep ".*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:_]*_$suffix\.tgz$" $sstate_files_list | wc -l 2>/dev/null`
-- 
1.9.1

[PATCH 4/4] sstate-cache-management: rm_by_stamps - discover all suffixes like remove_duplicated does

[Martin Jansa]

* instead of hard coding list of sstate enabled task use
  the same function as remove_duplicated to find them in
  sstate-cache directory

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index 77afd68..e5c57d3 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -324,16 +324,23 @@ rm_by_stamps (){
   local cache_list=`mktemp` || exit 1
   local keep_list=`mktemp` || exit 1
   local rm_list=`mktemp` || exit 1
-  local suffixes
   local sums
   local all_sums
 
-  suffixes="populate_sysroot populate_lic package_write_ipk \
-            package_write_rpm package_write_deb package packagedata deploy"
+  # Total number of files including sstate-, .siginfo and .done files
+  total_files=`find $cache_dir -type f -name 'sstate*' | wc -l`
+  # Save all the state file list to a file
+  find $cache_dir -type f -name 'sstate*' | sort -u -o $cache_list
+
+  echo "Figuring out the suffixes in the sstate cache dir ... "
+  local sstate_suffixes="`sed 's%.*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^_]*_\([^:]*\)\.tgz.*%\1%g' $cache_list | sort -u`"
+  echo "Done"
+  echo "The following suffixes have been found in the cache dir:"
+  echo $sstate_suffixes
 
   # Figure out all the md5sums in the stamps dir.
   echo "Figuring out all the md5sums in stamps dir ... "
-  for i in $suffixes; do
+  for i in $sstate_suffixes; do
       # There is no "\.sigdata" but "_setcene" when it is mirrored
       # from the SSTATE_MIRRORS, use them to figure out the sum.
       sums=`find $stamps -maxdepth 3 -name "*.do_$i.*" \
@@ -344,11 +351,6 @@ rm_by_stamps (){
   done
   echo "Done"
 
-  # Total number of files including sstate-, .siginfo and .done files
-  total_files=`find $cache_dir -type f -name 'sstate*' | wc -l`
-  # Save all the state file list to a file
-  find $cache_dir -type f -name 'sstate*' | sort -u -o $cache_list
-
   echo "Figuring out the files which will be removed ... "
   for i in $all_sums; do
       grep ".*/sstate:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:${i}_.*" $cache_list >>$keep_list
-- 
1.9.1

Re: [PATCH 0/4] sstate-cache-management fixes

[Jacob Kroon]

On Sat, 29 Mar 2014, Martin Jansa wrote:

> Removing do_package was reported by Jacob Kroon and then I've found
> few more issues, the most surprising is the one with populate_lic.
>
> The following changes since commit b16de9af5c7108396a347e7107c73608a2f8d74f:
>
>  packagegroup-toolset-native: Update after ocf-linux -> cryptodev-linux change (2014-03-27 19:53:07 +0000)
>
> are available in the git repository at:
>
>  git://git.openembedded.org/openembedded-core-contrib jansa/sstate-management
>  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/sstate-management
>
> Martin Jansa (4):
>  sstate-cache-management: rm_by_stamps - include signatures of
>    .sigdata. files
>  sstate-cache-management: rm_by_stamps - remove .siginfo and .done
>    files
>  sstate-cache-management: skip populate_lic archives when removing
>    duplicates
>  sstate-cache-management: rm_by_stamps - discover all suffixes like
>    remove_duplicated does
>
> scripts/sstate-cache-management.sh | 34 ++++++++++++++++++++--------------
> 1 file changed, 20 insertions(+), 14 deletions(-)
>

For the whole patch series:

Tested-by: Jacob Kroon <jacob.kroon@mikrodidakt.se>