[PATCH 0/4] sstate-cache-management.sh fixes [YOCTO #2897]

[PATCH 0/4] sstate-cache-management.sh fixes [YOCTO #2897]

[Martin Jansa]

Fixes some issues reported in YOCTO #2897

The following changes since commit c2109b765b24a7ffe4781257ad3fe4641a3b2a49:

  sstate: Ensure master.list exists if it doesn't already (2012-08-30 22:44:52 -0700)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib jansa/pull
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/pull

Martin Jansa (4):
  sstate-cache-management.sh: fix regexp in checksum grep
  sstate-cache-management.sh: fix regexp in AVAILTUNES grep
  sstate-cache-management.sh: look in meta* above oe-core dir when
    looking for available tunes
  sstate-cache-management.sh: don't hardcode available machines only to
    qemu*

 scripts/sstate-cache-management.sh | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

-- 
1.7.12

[PATCH 1/4] sstate-cache-management.sh: fix regexp in checksum grep

[Martin Jansa]

* using checksum without recipe/task name doesn't seem like good idea IMHO

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index a1f7c89..2b3100c 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -292,7 +292,7 @@ rm_by_stamps (){
 
   echo -n "Figuring out the files which will be removed ... "
   for i in $all_sums; do
-      grep ".*-${i}_*" $cache_list >>$keep_list
+      grep ".*-${i}_.*" $cache_list >>$keep_list
   done
   echo "Done"
 
-- 
1.7.12

[PATCH 2/4] sstate-cache-management.sh: fix regexp in AVAILTUNES grep

[Martin Jansa]

* it was also picking line
  openembedded-core/meta/conf/machine/include/README:AVAILTUNES - This is a list of all of the tuning definitions currently
  filling all_archs with wrong entries

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index 2b3100c..0f774a3 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -167,7 +167,7 @@ remove_duplicated () {
   topdir=$(dirname $(dirname $(readlink -e $0)))
   tunedirs="`find $topdir/meta* $layers -path '*/meta*/conf/machine/include'`"
   [ -n "$tunedirs" ] || echo_error "Can't find the tune directory"
-  all_archs=`grep -r -h "^AVAILTUNES " $tunedirs | sed -e 's/.*=//' -e 's/\"//g'`
+  all_archs=`grep -r -h "^AVAILTUNES .*=" $tunedirs | sed -e 's/.*=//' -e 's/\"//g'`
   # Add the qemu and native archs
   # Use the "_" to substitute "-", e.g., x86-64 to x86_64
   # Sort to remove the duplicated ones
-- 
1.7.12

[PATCH 3/4] sstate-cache-management.sh: look in meta* above oe-core dir when looking for available tunes

[Martin Jansa]

* many layouts have other layers on the same level as oe-core checkout

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index 0f774a3..e569487 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -153,6 +153,7 @@ gen_rmlist (){
 remove_duplicated () {
 
   local topdir
+  local oe_core_dir
   local tunedirs
   local all_archs
   local ava_archs
@@ -164,8 +165,9 @@ remove_duplicated () {
 
   # Find out the archs in all the layers
   echo -n "Figuring out the archs in the layers ... "
-  topdir=$(dirname $(dirname $(readlink -e $0)))
-  tunedirs="`find $topdir/meta* $layers -path '*/meta*/conf/machine/include'`"
+  oe_core_dir=$(dirname $(dirname $(readlink -e $0)))
+  topdir=$(dirname $oe_core_dir)
+  tunedirs="`find $topdir/meta* ${oe_core_dir}/meta* $layers -path '*/meta*/conf/machine/include'`"
   [ -n "$tunedirs" ] || echo_error "Can't find the tune directory"
   all_archs=`grep -r -h "^AVAILTUNES .*=" $tunedirs | sed -e 's/.*=//' -e 's/\"//g'`
   # Add the qemu and native archs
-- 
1.7.12

Re: [PATCH 3/4] sstate-cache-management.sh: look in meta* above oe-core dir when looking for available tunes

[Paul Eggleton]

On Sunday 02 September 2012 13:13:20 Martin Jansa wrote:
> * many layouts have other layers on the same level as oe-core checkout

This is beginning to indicate that this script really needs to actually know 
something about your configuration rather than trying to guess it.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 3/4] sstate-cache-management.sh: look in meta* above oe-core dir when looking for available tunes

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1130 bytes --]

On Sun, Sep 02, 2012 at 01:07:53PM +0100, Paul Eggleton wrote:
> On Sunday 02 September 2012 13:13:20 Martin Jansa wrote:
> > * many layouts have other layers on the same level as oe-core checkout
> 
> This is beginning to indicate that this script really needs to actually know 
> something about your configuration rather than trying to guess it.

You can pass more layers to script with:

  --extra-layer=<layer1>,<layer2>...<layern>
        Specify the layer which will be used for searching the archs,
        it will search the meta and meta-* layers in the top dir by
        default, and will search meta, meta-*, <layer1>, <layer2>,
        ...<layern> when specified. Use "," as the separator.

But nobody says what is "top dir", in my bblayers.conf it's the
directory with oe-core and meta-* checkouts and I think this is common
layout if you're not using some combo tool to merge all layers to
oe-core checkout.

So this change only makes default search path a bit more usefull and
then it filteres available arch anyway.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH 3/4] sstate-cache-management.sh: look in meta* above oe-core dir when looking for available tunes

[Paul Eggleton]

On Monday 03 September 2012 09:23:17 Martin Jansa wrote:
> On Sun, Sep 02, 2012 at 01:07:53PM +0100, Paul Eggleton wrote:
> > On Sunday 02 September 2012 13:13:20 Martin Jansa wrote:
> > > * many layouts have other layers on the same level as oe-core checkout
> > 
> > This is beginning to indicate that this script really needs to actually
> > know something about your configuration rather than trying to guess it.
> You can pass more layers to script with:
> 
>   --extra-layer=<layer1>,<layer2>...<layern>
>         Specify the layer which will be used for searching the archs,
>         it will search the meta and meta-* layers in the top dir by
>         default, and will search meta, meta-*, <layer1>, <layer2>,
>         ...<layern> when specified. Use "," as the separator.
> 
> But nobody says what is "top dir", in my bblayers.conf it's the
> directory with oe-core and meta-* checkouts and I think this is common
> layout if you're not using some combo tool to merge all layers to
> oe-core checkout.
> 
> So this change only makes default search path a bit more usefull and
> then it filteres available arch anyway.

I don't dispute that your change is an improvement, I just think it would be a 
good future improvement to pick up the configuration from bitbake rather than 
working in the way it currently does.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH 4/4] sstate-cache-management.sh: don't hardcode available machines only to qemu*

[Martin Jansa]

* find them dynamically in layers like AVAILTUNES

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index e569487..491df5f 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -156,6 +156,7 @@ remove_duplicated () {
   local oe_core_dir
   local tunedirs
   local all_archs
+  local all_machines
   local ava_archs
   local arch
   local file_names
@@ -169,11 +170,12 @@ remove_duplicated () {
   topdir=$(dirname $oe_core_dir)
   tunedirs="`find $topdir/meta* ${oe_core_dir}/meta* $layers -path '*/meta*/conf/machine/include'`"
   [ -n "$tunedirs" ] || echo_error "Can't find the tune directory"
+  all_machines="`find $topdir/meta* ${oe_core_dir}/meta* $layers -path '*/meta*/conf/machine/*' -name '*.conf' | sed -e 's/.*\///' -e 's/.conf$//'`"
   all_archs=`grep -r -h "^AVAILTUNES .*=" $tunedirs | sed -e 's/.*=//' -e 's/\"//g'`
   # Add the qemu and native archs
   # Use the "_" to substitute "-", e.g., x86-64 to x86_64
   # Sort to remove the duplicated ones
-  all_archs=$(echo $all_archs qemuarm qemux86 qemumips qemuppc qemux86_64 $(uname -m) \
+  all_archs=$(echo $all_archs $all_machines $(uname -m) \
           | sed -e 's/-/_/g' -e 's/ /\n/g' | sort -u)
   echo "Done"
 
-- 
1.7.12

[PATCH 5/9] sstate-cache-management.sh: add option to enable debug output

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index 491df5f..0cfff77 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -22,6 +22,7 @@ confirm=
 fsym=
 total_deleted=0
 verbose=
+debug=0
 
 usage () {
   cat << EOF
@@ -69,6 +70,9 @@ Options:
   -v, --verbose
         explain what is being done
 
+  -d, --debug
+        show debug info, repeat for more debug info
+
 EOF
 }
 
@@ -215,13 +219,16 @@ remove_duplicated () {
           for arch in $ava_archs; do
               grep -h "/$fn-$arch-" $list_suffix >>$fn_tmp
           done
+          [ $debug -gt 1 ] && echo "Available files for $fn with suffix $suffix:" && cat $fn_tmp
           # Use the modification time
           to_del=$(ls -t $(cat $fn_tmp) | sed -n '1!p')
+          [ $debug -gt 2 ] && echo "Considering to delete: $to_del"
           # The sstate file which is downloaded from the SSTATE_MIRROR is
           # put in SSTATE_DIR, and there is a symlink in SSTATE_DIR/??/ to
           # it, so filter it out from the remove list if it should not be
           # removed.
           to_keep=$(ls -t $(cat $fn_tmp) | sed -n '1p')
+          [ $debug -gt 2 ] && echo "Considering to keep: $to_keep"
           for k in $to_keep; do
               if [ -L "$k" ]; then
                   # The symlink's destination
@@ -235,9 +242,11 @@ remove_duplicated () {
               fi
           done
           rm -f $fn_tmp
+          [ $debug -gt 2 ] && echo "Decided to delete: $to_del"
           gen_rmlist $rm_list "$to_del"
       done
       [ ! -s "$rm_list" ] || deleted=`cat $rm_list | wc -l`
+      [ -s "$rm_list" -a $debug -gt 0 ] && cat $rm_list
       echo "($deleted files will be removed)"
       let total_deleted=$total_deleted+$deleted
   done
@@ -306,6 +315,7 @@ rm_by_stamps (){
       gen_rmlist $rm_list "$to_del"
       let total_deleted=(`cat $rm_list | wc -w`)
       if [ $total_deleted -gt 0 ]; then
+          [ $debug -gt 0 ] && cat $rm_list
           read_confirm
           if [ "$confirm" = "y" -o "$confirm" = "Y" ]; then
               echo "Removing sstate cache files ... ($total_deleted files)"
@@ -374,6 +384,11 @@ while [ -n "$1" ]; do
       verbose="-v"
       shift
         ;;
+    --debug)
+      debug=`expr $debug + 1`
+      echo "Debug level $debug"
+      shift
+        ;;
     --help|-h)
       usage
       exit 0
-- 
1.7.12

Re: [PATCH 5/9] sstate-cache-management.sh: add option to enable debug output

[Saul Wold]

On 09/03/2012 08:05 AM, Martin Jansa wrote:
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>   scripts/sstate-cache-management.sh | 15 +++++++++++++++
>   1 file changed, 15 insertions(+)
>
> diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
> index 491df5f..0cfff77 100755
> --- a/scripts/sstate-cache-management.sh
> +++ b/scripts/sstate-cache-management.sh
> @@ -22,6 +22,7 @@ confirm=
>   fsym=
>   total_deleted=0
>   verbose=
> +debug=0
>
>   usage () {
>     cat << EOF
> @@ -69,6 +70,9 @@ Options:
>     -v, --verbose
>           explain what is being done
>
> +  -d, --debug
> +        show debug info, repeat for more debug info
> +
>   EOF
>   }
>
> @@ -215,13 +219,16 @@ remove_duplicated () {
>             for arch in $ava_archs; do
>                 grep -h "/$fn-$arch-" $list_suffix >>$fn_tmp
>             done
> +          [ $debug -gt 1 ] && echo "Available files for $fn with suffix $suffix:" && cat $fn_tmp
>             # Use the modification time
>             to_del=$(ls -t $(cat $fn_tmp) | sed -n '1!p')
> +          [ $debug -gt 2 ] && echo "Considering to delete: $to_del"
>             # The sstate file which is downloaded from the SSTATE_MIRROR is
>             # put in SSTATE_DIR, and there is a symlink in SSTATE_DIR/??/ to
>             # it, so filter it out from the remove list if it should not be
>             # removed.
>             to_keep=$(ls -t $(cat $fn_tmp) | sed -n '1p')
> +          [ $debug -gt 2 ] && echo "Considering to keep: $to_keep"
>             for k in $to_keep; do
>                 if [ -L "$k" ]; then
>                     # The symlink's destination
> @@ -235,9 +242,11 @@ remove_duplicated () {
>                 fi
>             done
>             rm -f $fn_tmp
> +          [ $debug -gt 2 ] && echo "Decided to delete: $to_del"
>             gen_rmlist $rm_list "$to_del"
>         done
>         [ ! -s "$rm_list" ] || deleted=`cat $rm_list | wc -l`
> +      [ -s "$rm_list" -a $debug -gt 0 ] && cat $rm_list
>         echo "($deleted files will be removed)"
>         let total_deleted=$total_deleted+$deleted
>     done
> @@ -306,6 +315,7 @@ rm_by_stamps (){
>         gen_rmlist $rm_list "$to_del"
>         let total_deleted=(`cat $rm_list | wc -w`)
>         if [ $total_deleted -gt 0 ]; then
> +          [ $debug -gt 0 ] && cat $rm_list
>             read_confirm
>             if [ "$confirm" = "y" -o "$confirm" = "Y" ]; then
>                 echo "Removing sstate cache files ... ($total_deleted files)"
> @@ -374,6 +384,11 @@ while [ -n "$1" ]; do
>         verbose="-v"
>         shift
>           ;;
> +    --debug)
> +      debug=`expr $debug + 1`
> +      echo "Debug level $debug"
> +      shift
> +        ;;
>       --help|-h)
>         usage
>         exit 0
>
Merged into OE-Core

Thanks
	Sau!

[PATCH 6/9] sstate-cache-management.sh: fix remove_duplicated when multiple archs were built

[Martin Jansa]

* The problem is that you're looking for duplicates across all available archs, so
  armv4t and armv7a populate_sysroot are considered duplicate and only last one is
  kept, but to rebuild from sstate-cache you need both!

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-cache-management.sh | 52 ++++++++++++++++++++------------------
 1 file changed, 27 insertions(+), 25 deletions(-)

diff --git a/scripts/sstate-cache-management.sh b/scripts/sstate-cache-management.sh
index 0cfff77..f519555 100755
--- a/scripts/sstate-cache-management.sh
+++ b/scripts/sstate-cache-management.sh
@@ -217,33 +217,35 @@ remove_duplicated () {
       for fn in $file_names; do
           [ -z "$verbose" ] || echo "Analyzing $fn-xxx_$suffix.tgz"
           for arch in $ava_archs; do
-              grep -h "/$fn-$arch-" $list_suffix >>$fn_tmp
-          done
-          [ $debug -gt 1 ] && echo "Available files for $fn with suffix $suffix:" && cat $fn_tmp
-          # Use the modification time
-          to_del=$(ls -t $(cat $fn_tmp) | sed -n '1!p')
-          [ $debug -gt 2 ] && echo "Considering to delete: $to_del"
-          # The sstate file which is downloaded from the SSTATE_MIRROR is
-          # put in SSTATE_DIR, and there is a symlink in SSTATE_DIR/??/ to
-          # it, so filter it out from the remove list if it should not be
-          # removed.
-          to_keep=$(ls -t $(cat $fn_tmp) | sed -n '1p')
-          [ $debug -gt 2 ] && echo "Considering to keep: $to_keep"
-          for k in $to_keep; do
-              if [ -L "$k" ]; then
-                  # The symlink's destination
-                  k_dest="`readlink -e $k`"
-                  # Maybe it is the one in cache_dir
-                  k_maybe="$cache_dir/${k##/*/}"
-                  # Remove it from the remove list if they are the same.
-                  if [ "$k_dest" = "$k_maybe" ]; then
-                      to_del="`echo $to_del | sed 's#'\"$k_maybe\"'##g'`"
-                  fi
+              grep -h "/$fn-$arch-" $list_suffix >$fn_tmp
+              if [ -s $fn_tmp ] ; then
+                  [ $debug -gt 1 ] && echo "Available files for $fn-$arch- with suffix $suffix:" && cat $fn_tmp
+                  # Use the modification time
+                  to_del=$(ls -t $(cat $fn_tmp) | sed -n '1!p')
+                  [ $debug -gt 2 ] && echo "Considering to delete: $to_del"
+                  # The sstate file which is downloaded from the SSTATE_MIRROR is
+                  # put in SSTATE_DIR, and there is a symlink in SSTATE_DIR/??/ to
+                  # it, so filter it out from the remove list if it should not be
+                  # removed.
+                  to_keep=$(ls -t $(cat $fn_tmp) | sed -n '1p')
+                  [ $debug -gt 2 ] && echo "Considering to keep: $to_keep"
+                  for k in $to_keep; do
+                      if [ -L "$k" ]; then
+                          # The symlink's destination
+                          k_dest="`readlink -e $k`"
+                          # Maybe it is the one in cache_dir
+                          k_maybe="$cache_dir/${k##/*/}"
+                          # Remove it from the remove list if they are the same.
+                          if [ "$k_dest" = "$k_maybe" ]; then
+                              to_del="`echo $to_del | sed 's#'\"$k_maybe\"'##g'`"
+                          fi
+                      fi
+                  done
+                  rm -f $fn_tmp
+                  [ $debug -gt 2 ] && echo "Decided to delete: $to_del"
+                  gen_rmlist $rm_list "$to_del"
               fi
           done
-          rm -f $fn_tmp
-          [ $debug -gt 2 ] && echo "Decided to delete: $to_del"
-          gen_rmlist $rm_list "$to_del"
       done
       [ ! -s "$rm_list" ] || deleted=`cat $rm_list | wc -l`
       [ -s "$rm_list" -a $debug -gt 0 ] && cat $rm_list
-- 
1.7.12

Re: [PATCH 0/4] sstate-cache-management.sh fixes [YOCTO #2897]

[Richard Purdie]

On Sun, 2012-09-02 at 13:13 +0200, Martin Jansa wrote:
> Fixes some issues reported in YOCTO #2897
> 
> The following changes since commit c2109b765b24a7ffe4781257ad3fe4641a3b2a49:
> 
>   sstate: Ensure master.list exists if it doesn't already (2012-08-30 22:44:52 -0700)
> 
> are available in the git repository at:
> 
>   git://git.openembedded.org/openembedded-core-contrib jansa/pull
>   http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/pull
> 
> Martin Jansa (4):
>   sstate-cache-management.sh: fix regexp in checksum grep
>   sstate-cache-management.sh: fix regexp in AVAILTUNES grep
>   sstate-cache-management.sh: look in meta* above oe-core dir when
>     looking for available tunes
>   sstate-cache-management.sh: don't hardcode available machines only to
>     qemu*

Merged to master, thanks (along with 5/4 and 6/4 :).

I do agree with Paul, we need to find a better way to do some of this
but that is something for the future...

Cheers,

Richard