[PATCH] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[PATCH] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

* it's not very universal, but works with default oe-core setup and
  shows basic HOW-TO. It can be improved later.

* eglibc-initial should be fixed, but for now better to add that
  work around then to show a lot of false positives

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 scripts/sstate-sysroot-cruft.sh | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100755 scripts/sstate-sysroot-cruft.sh

diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh
new file mode 100755
index 0000000..ea5fdf8
--- /dev/null
+++ b/scripts/sstate-sysroot-cruft.sh
@@ -0,0 +1,27 @@
+#!/bin/sh
+
+# Used to find files installed in sysroot which are not tracked by sstate manifest
+# Update BASE
+
+BASE="/OE/oe-core"
+
+OUTPUT=${BASE}/sysroot.cruft.`date "+%s"`
+WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$"
+
+mkdir ${OUTPUT}
+find ${BASE}/tmp-eglibc/sstate-control -name \*populate-sysroot -o -name \*package | xargs cat | grep sysroots | \
+  sed 's#tcbootstrapusr#tcbootstrap/usr#g; s#/$##g; s#///*#/#g' | \
+  # work around for missing / in manifest for eglibc-initial, paths ending with / for directories and multiplied // (e.g. paths to native sysroot)
+  sort -u > ${OUTPUT}/sstate-control.master.list # -u because some directories are listed for more recipes
+find ${BASE}/tmp-eglibc/sysroots/ | \
+  sort > ${OUTPUT}/sstate-control.sysroot.list
+
+diff ${OUTPUT}/sstate-control.master.list ${OUTPUT}/sstate-control.sysroot.list > ${OUTPUT}/sstate-control.diff.all
+
+cp ${OUTPUT}/sstate-control.diff.all ${OUTPUT}/sstate-control.diff
+for item in ${WHITELIST}; do 
+  sed -i "/${item}/d" ${OUTPUT}/sstate-control.diff;
+done
+
+echo "Following files are installed in sysroot, but not tracked by sstate"
+cat ${OUTPUT}/sstate-control.diff
-- 
1.7.12.4

[PATCHv2] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

* it's not very universal, but works with default oe-core setup and
  shows basic HOW-TO. It can be improved later.

* eglibc-initial should be fixed, but for now better to add that
  work around then to show a lot of false positives

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 V2: added .pyo to WHITELIST
     shorter filenames
     TMPDIR
     added duplicates but not shown

 scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100755 scripts/sstate-sysroot-cruft.sh

diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh
new file mode 100755
index 0000000..7bfb8f4
--- /dev/null
+++ b/scripts/sstate-sysroot-cruft.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+
+# Used to find files installed in sysroot which are not tracked by sstate manifest
+# Update BASE
+
+BASE="/OE/oe-core"
+TMPDIR="${BASE}/tmp-eglibc"
+
+OUTPUT=${BASE}/sysroot.cruft.`date "+%s"`
+WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$"
+
+mkdir ${OUTPUT}
+find ${TMPDIR}/sstate-control -name \*populate-sysroot -o -name \*package | xargs cat | grep sysroots | \
+  sed 's#tcbootstrapusr#tcbootstrap/usr#g; s#/$##g; s#///*#/#g' | \
+  # work around for missing / in manifest for eglibc-initial, paths ending with / for directories and multiplied // (e.g. paths to native sysroot)
+  sort > ${OUTPUT}/master.list.all
+sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes
+find ${TMPDIR}/sysroots/ | \
+  sort > ${OUTPUT}/sysroot.list
+
+diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates
+diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all
+
+cp ${OUTPUT}/diff.all ${OUTPUT}/diff
+for item in ${WHITELIST}; do 
+  sed -i "/${item}/d" ${OUTPUT}/diff;
+done
+
+# too many false positives for directories
+# echo "Following files are installed in sysroot at least twice"
+# cat ${OUTPUT}/duplicates
+
+echo "Following files are installed in sysroot, but not tracked by sstate"
+cat ${OUTPUT}/diff
-- 
1.7.12.4

[PATCHv3] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

* it's not very universal, but works with default oe-core setup and
  shows basic HOW-TO. It can be improved later.

* eglibc-initial should be fixed, but for now better to add that
  work around then to show a lot of false positives

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 V2: added .pyo to WHITELIST                                                                                            
     shorter filenames                                                                                        
     TMPDIR                                                                                                                                                   
     added duplicates but not shown                                                                                                   

 V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot 
     was changed in febeaf3d1b8917b660c7279b008d8b03337568e9

 scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100755 scripts/sstate-sysroot-cruft.sh

diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh
new file mode 100755
index 0000000..b68de42
--- /dev/null
+++ b/scripts/sstate-sysroot-cruft.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+
+# Used to find files installed in sysroot which are not tracked by sstate manifest
+# Update BASE
+
+BASE="/OE/oe-core"
+TMPDIR="${BASE}/tmp-eglibc"
+
+OUTPUT=${BASE}/sysroot.cruft.`date "+%s"`
+WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$"
+
+mkdir ${OUTPUT}
+find ${TMPDIR}/sstate-control -name \*.populate-sysroot\* -o -name \*.package\* | xargs cat | grep sysroots | \
+  sed 's#tcbootstrapusr#tcbootstrap/usr#g; s#/$##g; s#///*#/#g' | \
+  # work around for missing / in manifest for eglibc-initial, paths ending with / for directories and multiplied // (e.g. paths to native sysroot)
+  sort > ${OUTPUT}/master.list.all
+sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes
+find ${TMPDIR}/sysroots/ | \
+  sort > ${OUTPUT}/sysroot.list
+
+diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates
+diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all
+
+cp ${OUTPUT}/diff.all ${OUTPUT}/diff
+for item in ${WHITELIST}; do 
+  sed -i "/${item}/d" ${OUTPUT}/diff;
+done
+
+# too many false positives for directories
+# echo "Following files are installed in sysroot at least twice"
+# cat ${OUTPUT}/duplicates
+
+echo "Following files are installed in sysroot, but not tracked by sstate"
+cat ${OUTPUT}/diff
-- 
1.7.12.4

[PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

* it's not very universal, but works with default oe-core setup and
  shows basic HOW-TO. It can be improved later.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 V2: added .pyo to WHITELIST
     shorter filenames
     TMPDIR
     added duplicates but not shown

 V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot
     was changed in febeaf3d1b8917b660c7279b008d8b03337568e9

 V4: dropped eglibc-initial work around, it was fixed in oe-core
                                                                                                                                                              
 scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100755 scripts/sstate-sysroot-cruft.sh

diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh
new file mode 100755
index 0000000..ca23dcf
--- /dev/null
+++ b/scripts/sstate-sysroot-cruft.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+
+# Used to find files installed in sysroot which are not tracked by sstate manifest
+# Update BASE
+
+BASE="/OE/oe-core"
+TMPDIR="${BASE}/tmp-eglibc"
+
+OUTPUT=${BASE}/sysroot.cruft.`date "+%s"`
+WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$"
+
+mkdir ${OUTPUT}
+find ${TMPDIR}/sstate-control -name \*.populate-sysroot\* -o -name \*.package\* | xargs cat | grep sysroots | \
+  sed 's#/$##g; s#///*#/#g' | \
+  # work around for paths ending with / for directories and multiplied // (e.g. paths to native sysroot)
+  sort > ${OUTPUT}/master.list.all
+sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes
+find ${TMPDIR}/sysroots/ | \
+  sort > ${OUTPUT}/sysroot.list
+
+diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates
+diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all
+
+cp ${OUTPUT}/diff.all ${OUTPUT}/diff
+for item in ${WHITELIST}; do 
+  sed -i "/${item}/d" ${OUTPUT}/diff;
+done
+
+# too many false positives for directories
+# echo "Following files are installed in sysroot at least twice"
+# cat ${OUTPUT}/duplicates
+
+echo "Following files are installed in sysroot, but not tracked by sstate"
+cat ${OUTPUT}/diff
-- 
1.8.0

Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 1758 bytes --]

On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> * it's not very universal, but works with default oe-core setup and
>   shows basic HOW-TO. It can be improved later.
>
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  V2: added .pyo to WHITELIST
>      shorter filenames
>      TMPDIR
>      added duplicates but not shown
>
>  V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot
>      was changed in febeaf3d1b8917b660c7279b008d8b03337568e9
>
>  V4: dropped eglibc-initial work around, it was fixed in oe-core
>
>  scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++
>  1 file changed, 34 insertions(+)
>  create mode 100755 scripts/sstate-sysroot-cruft.sh
>
> diff --git a/scripts/sstate-sysroot-cruft.sh
> b/scripts/sstate-sysroot-cruft.sh
> new file mode 100755
> index 0000000..ca23dcf
> --- /dev/null
> +++ b/scripts/sstate-sysroot-cruft.sh
> @@ -0,0 +1,34 @@
> +#!/bin/sh
> +
> +# Used to find files installed in sysroot which are not tracked by sstate
> manifest
> +# Update BASE
> +
> +BASE="/OE/oe-core"
>

This seems interesting, but I have a few comments/concerns.

1) don't hardcode BASE, figure out the path relative to the script's
location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)"
2) output files shouldn't go into oe-core directly, as oe-core isn't
guaranteed to be writable, and it's more common to expect output from a
script like this to go relative to the current directory, or a temp
directory
3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that
breaks for any distros or users which separate their tmpdirs by distro, or
set TCLIBCAPPEND = ""
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 2518 bytes --]

Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2387 bytes --]

On Thu, Nov 15, 2012 at 08:25:20AM -0700, Chris Larson wrote:
> On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com>wrote:
> 
> > * it's not very universal, but works with default oe-core setup and
> >   shows basic HOW-TO. It can be improved later.
> >
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  V2: added .pyo to WHITELIST
> >      shorter filenames
> >      TMPDIR
> >      added duplicates but not shown
> >
> >  V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot
> >      was changed in febeaf3d1b8917b660c7279b008d8b03337568e9
> >
> >  V4: dropped eglibc-initial work around, it was fixed in oe-core
> >
> >  scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++
> >  1 file changed, 34 insertions(+)
> >  create mode 100755 scripts/sstate-sysroot-cruft.sh
> >
> > diff --git a/scripts/sstate-sysroot-cruft.sh
> > b/scripts/sstate-sysroot-cruft.sh
> > new file mode 100755
> > index 0000000..ca23dcf
> > --- /dev/null
> > +++ b/scripts/sstate-sysroot-cruft.sh
> > @@ -0,0 +1,34 @@
> > +#!/bin/sh
> > +
> > +# Used to find files installed in sysroot which are not tracked by sstate
> > manifest
> > +# Update BASE
> > +
> > +BASE="/OE/oe-core"
> >
> 
> This seems interesting, but I have a few comments/concerns.
> 
> 1) don't hardcode BASE, figure out the path relative to the script's
> location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)"
> 2) output files shouldn't go into oe-core directly, as oe-core isn't
> guaranteed to be writable, and it's more common to expect output from a
> script like this to go relative to the current directory, or a temp
> directory
> 3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that
> breaks for any distros or users which separate their tmpdirs by distro, or
> set TCLIBCAPPEND = ""

Ah in this case BASE is not directory with oe-core layer but one above.

I didn't want to use bitbake -e, so that this script can be executed
when some other bitbake process is running.

As said it's not really universal (I wrote it just to confirm something
and then shared it to find if someone else finds it also usefull to
improve it).

What about moving output to TMPDIR and setting TMPDIR by parameter?

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 3024 bytes --]

On Thu, Nov 15, 2012 at 9:19 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> On Thu, Nov 15, 2012 at 08:25:20AM -0700, Chris Larson wrote:
> > On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com
> >wrote:
> >
> > > * it's not very universal, but works with default oe-core setup and
> > >   shows basic HOW-TO. It can be improved later.
> > >
> > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > > ---
> > >  V2: added .pyo to WHITELIST
> > >      shorter filenames
> > >      TMPDIR
> > >      added duplicates but not shown
> > >
> > >  V3: use also populate-sysroot.MACHINE, manifest name for
> populate-sysroot
> > >      was changed in febeaf3d1b8917b660c7279b008d8b03337568e9
> > >
> > >  V4: dropped eglibc-initial work around, it was fixed in oe-core
> > >
> > >  scripts/sstate-sysroot-cruft.sh | 34
> ++++++++++++++++++++++++++++++++++
> > >  1 file changed, 34 insertions(+)
> > >  create mode 100755 scripts/sstate-sysroot-cruft.sh
> > >
> > > diff --git a/scripts/sstate-sysroot-cruft.sh
> > > b/scripts/sstate-sysroot-cruft.sh
> > > new file mode 100755
> > > index 0000000..ca23dcf
> > > --- /dev/null
> > > +++ b/scripts/sstate-sysroot-cruft.sh
> > > @@ -0,0 +1,34 @@
> > > +#!/bin/sh
> > > +
> > > +# Used to find files installed in sysroot which are not tracked by
> sstate
> > > manifest
> > > +# Update BASE
> > > +
> > > +BASE="/OE/oe-core"
> > >
> >
> > This seems interesting, but I have a few comments/concerns.
> >
> > 1) don't hardcode BASE, figure out the path relative to the script's
> > location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)"
> > 2) output files shouldn't go into oe-core directly, as oe-core isn't
> > guaranteed to be writable, and it's more common to expect output from a
> > script like this to go relative to the current directory, or a temp
> > directory
> > 3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that
> > breaks for any distros or users which separate their tmpdirs by distro,
> or
> > set TCLIBCAPPEND = ""
>
> Ah in this case BASE is not directory with oe-core layer but one above.
>

Still trivial to determine relative to the script's location. Adding an
extra call to dirname, or an extra '/..' isn't particularly tough.

I didn't want to use bitbake -e, so that this script can be executed
> when some other bitbake process is running.
>

I don't see having to hardcode assumptions about the environment as a net
win, personally. At least make them arguments, or add an argument to
optionally use bitbake -e, or let it take vars from the environment.

As said it's not really universal (I wrote it just to confirm something
> and then shared it to find if someone else finds it also usefull to
> improve it).
>

If it can't be used in a wide variety of circumstances, I'd argue against
its inclusion in oe-core until such time that it's ready. The subject of
this thread didn't include "RFC' ;)
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 4262 bytes --]

[PATCHv5] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 V2: added .pyo to WHITELIST                                                                                                                                  
     shorter filenames                                                                                                                                        
     TMPDIR                                                                                                                                                   
     added duplicates but not shown                                                                                                                           
                                                                                                                                                              
 V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot                                                                                    
     was changed in febeaf3d1b8917b660c7279b008d8b03337568e9                                                                                                  
                                                                                                                                                              
 V4: dropped eglibc-initial work around, it was fixed in oe-core                                                                                              

 V5: set tmpdir from param or env variable and show usage

 scripts/sstate-sysroot-cruft.sh | 78 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
 create mode 100755 scripts/sstate-sysroot-cruft.sh

diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh
new file mode 100755
index 0000000..6caa252
--- /dev/null
+++ b/scripts/sstate-sysroot-cruft.sh
@@ -0,0 +1,78 @@
+#!/bin/sh
+
+# Used to find files installed in sysroot which are not tracked by sstate manifest
+
+# Global vars
+tmpdir=
+
+usage () {
+  cat << EOF
+Welcome to sysroot cruft finding utility.
+$0 <OPTION>
+
+Options:
+  -h, --help
+        Display this help and exit.
+
+  --tmpdir=<tmpdir>
+        Specify tmpdir, will use the environment variable TMPDIR if it is not specified.
+	Something like /OE/oe-core/tmp-eglibc (no / at the end).
+EOF
+}
+
+# Print error information and exit.
+echo_error () {
+  echo "ERROR: $1" >&2
+  exit 1
+}
+
+while [ -n "$1" ]; do
+  case $1 in
+    --tmpdir=*)
+      tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e`
+      [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir"
+      shift
+        ;;
+    --help|-h)
+      usage
+      exit 0
+        ;;
+    *)
+      echo "Invalid arguments $*"
+      echo_error "Try '$0 -h' for more information."
+        ;;
+  esac
+done
+
+# sstate cache directory, use environment variable TMPDIR
+# if it was not specified, otherwise, error.
+[ -n "$tmpdir" ] || tmpdir=$TMPDIR
+[ -n "$tmpdir" ] || echo_error "No tmpdir found!"
+[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\""
+
+OUTPUT=${tmpdir}/sysroot.cruft.`date "+%s"`
+WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$"
+
+mkdir ${OUTPUT}
+find ${tmpdir}/sstate-control -name \*.populate-sysroot\* -o -name \*.package\* | xargs cat | grep sysroots | \
+  sed 's#/$##g; s#///*#/#g' | \
+  # work around for paths ending with / for directories and multiplied // (e.g. paths to native sysroot)
+  sort > ${OUTPUT}/master.list.all
+sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes
+find ${tmpdir}/sysroots/ | \
+  sort > ${OUTPUT}/sysroot.list
+
+diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates
+diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all
+
+cp ${OUTPUT}/diff.all ${OUTPUT}/diff
+for item in ${WHITELIST}; do 
+  sed -i "/${item}/d" ${OUTPUT}/diff;
+done
+
+# too many false positives for directories
+# echo "Following files are installed in sysroot at least twice"
+# cat ${OUTPUT}/duplicates
+
+echo "Following files are installed in sysroot, but not tracked by sstate"
+cat ${OUTPUT}/diff
-- 
1.8.0

Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Paul Eggleton]

On Thursday 15 November 2012 17:19:07 Martin Jansa wrote:
> I didn't want to use bitbake -e, so that this script can be executed
> when some other bitbake process is running.

You really should not be trying to touch anything under TMPDIR let alone 
sstate files while bitbake is actually running against that same directory. I'm 
not sure why you would want to though...

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

On Thu, Nov 15, 2012 at 04:49:01PM +0000, Paul Eggleton wrote:
> On Thursday 15 November 2012 17:19:07 Martin Jansa wrote:
> > I didn't want to use bitbake -e, so that this script can be executed
> > when some other bitbake process is running.
> 
> You really should not be trying to touch anything under TMPDIR let alone 
> sstate files while bitbake is actually running against that same directory. I'm 
> not sure why you would want to though...

While running builds for 8 MACHINEs in for loop I find it usefull to
test one sysroot (from 1st MACHINE already built) while keeping
build for remaining machines running for rest of night and sometimes
following days..

At least that was my use-case when I wrote this script..

Yes I could have execute build only for 1st machine, wait till it's
finished, then run script wait till it's finished and then start for
loop for remainig 7, but that does not allow me to fall asleep while
it's still building 1st :) or I have to plan it in advance and build
with much longer one-liner.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3610 bytes --]

On Thu, Nov 15, 2012 at 09:27:52AM -0700, Chris Larson wrote:
> On Thu, Nov 15, 2012 at 9:19 AM, Martin Jansa <martin.jansa@gmail.com>wrote:
> 
> > On Thu, Nov 15, 2012 at 08:25:20AM -0700, Chris Larson wrote:
> > > On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com
> > >wrote:
> > >
> > > > * it's not very universal, but works with default oe-core setup and
> > > >   shows basic HOW-TO. It can be improved later.
> > > >
> > > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > > > ---
> > > >  V2: added .pyo to WHITELIST
> > > >      shorter filenames
> > > >      TMPDIR
> > > >      added duplicates but not shown
> > > >
> > > >  V3: use also populate-sysroot.MACHINE, manifest name for
> > populate-sysroot
> > > >      was changed in febeaf3d1b8917b660c7279b008d8b03337568e9
> > > >
> > > >  V4: dropped eglibc-initial work around, it was fixed in oe-core
> > > >
> > > >  scripts/sstate-sysroot-cruft.sh | 34
> > ++++++++++++++++++++++++++++++++++
> > > >  1 file changed, 34 insertions(+)
> > > >  create mode 100755 scripts/sstate-sysroot-cruft.sh
> > > >
> > > > diff --git a/scripts/sstate-sysroot-cruft.sh
> > > > b/scripts/sstate-sysroot-cruft.sh
> > > > new file mode 100755
> > > > index 0000000..ca23dcf
> > > > --- /dev/null
> > > > +++ b/scripts/sstate-sysroot-cruft.sh
> > > > @@ -0,0 +1,34 @@
> > > > +#!/bin/sh
> > > > +
> > > > +# Used to find files installed in sysroot which are not tracked by
> > sstate
> > > > manifest
> > > > +# Update BASE
> > > > +
> > > > +BASE="/OE/oe-core"
> > > >
> > >
> > > This seems interesting, but I have a few comments/concerns.
> > >
> > > 1) don't hardcode BASE, figure out the path relative to the script's
> > > location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)"
> > > 2) output files shouldn't go into oe-core directly, as oe-core isn't
> > > guaranteed to be writable, and it's more common to expect output from a
> > > script like this to go relative to the current directory, or a temp
> > > directory
> > > 3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that
> > > breaks for any distros or users which separate their tmpdirs by distro,
> > or
> > > set TCLIBCAPPEND = ""
> >
> > Ah in this case BASE is not directory with oe-core layer but one above.
> >
> 
> Still trivial to determine relative to the script's location. Adding an
> extra call to dirname, or an extra '/..' isn't particularly tough.

Yes, but making another assumption that TMPDIR is on same level as
oe-core checkout it.

> I didn't want to use bitbake -e, so that this script can be executed
> > when some other bitbake process is running.
> >
> 
> I don't see having to hardcode assumptions about the environment as a net
> win, personally. At least make them arguments, or add an argument to
> optionally use bitbake -e, or let it take vars from the environment.

v5 is using argument or env variable (like sstate-cache-management.sh)

> As said it's not really universal (I wrote it just to confirm something
> > and then shared it to find if someone else finds it also usefull to
> > improve it).
> >
> 
> If it can't be used in a wide variety of circumstances, I'd argue against
> its inclusion in oe-core until such time that it's ready. The subject of
> this thread didn't include "RFC' ;)

I was using it in wide variety of circumstances just by updating 1
variable in script, but I agree that argument + shell history can make
it easier.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]