* [PATCH] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate @ 2012-10-23 22:51 Martin Jansa 2012-10-24 8:10 ` [PATCHv2] " Martin Jansa 0 siblings, 1 reply; 11+ messages in thread From: Martin Jansa @ 2012-10-23 22:51 UTC (permalink / raw) To: openembedded-core * it's not very universal, but works with default oe-core setup and shows basic HOW-TO. It can be improved later. * eglibc-initial should be fixed, but for now better to add that work around then to show a lot of false positives Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- scripts/sstate-sysroot-cruft.sh | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100755 scripts/sstate-sysroot-cruft.sh diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh new file mode 100755 index 0000000..ea5fdf8 --- /dev/null +++ b/scripts/sstate-sysroot-cruft.sh @@ -0,0 +1,27 @@ +#!/bin/sh + +# Used to find files installed in sysroot which are not tracked by sstate manifest +# Update BASE + +BASE="/OE/oe-core" + +OUTPUT=${BASE}/sysroot.cruft.`date "+%s"` +WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$" + +mkdir ${OUTPUT} +find ${BASE}/tmp-eglibc/sstate-control -name \*populate-sysroot -o -name \*package | xargs cat | grep sysroots | \ + sed 's#tcbootstrapusr#tcbootstrap/usr#g; s#/$##g; s#///*#/#g' | \ + # work around for missing / in manifest for eglibc-initial, paths ending with / for directories and multiplied // (e.g. paths to native sysroot) + sort -u > ${OUTPUT}/sstate-control.master.list # -u because some directories are listed for more recipes +find ${BASE}/tmp-eglibc/sysroots/ | \ + sort > ${OUTPUT}/sstate-control.sysroot.list + +diff ${OUTPUT}/sstate-control.master.list ${OUTPUT}/sstate-control.sysroot.list > ${OUTPUT}/sstate-control.diff.all + +cp ${OUTPUT}/sstate-control.diff.all ${OUTPUT}/sstate-control.diff +for item in ${WHITELIST}; do + sed -i "/${item}/d" ${OUTPUT}/sstate-control.diff; +done + +echo "Following files are installed in sysroot, but not tracked by sstate" +cat ${OUTPUT}/sstate-control.diff -- 1.7.12.4 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCHv2] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-10-23 22:51 [PATCH] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate Martin Jansa @ 2012-10-24 8:10 ` Martin Jansa 2012-10-24 11:15 ` [PATCHv3] " Martin Jansa 0 siblings, 1 reply; 11+ messages in thread From: Martin Jansa @ 2012-10-24 8:10 UTC (permalink / raw) To: openembedded-core * it's not very universal, but works with default oe-core setup and shows basic HOW-TO. It can be improved later. * eglibc-initial should be fixed, but for now better to add that work around then to show a lot of false positives Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- V2: added .pyo to WHITELIST shorter filenames TMPDIR added duplicates but not shown scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 scripts/sstate-sysroot-cruft.sh diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh new file mode 100755 index 0000000..7bfb8f4 --- /dev/null +++ b/scripts/sstate-sysroot-cruft.sh @@ -0,0 +1,34 @@ +#!/bin/sh + +# Used to find files installed in sysroot which are not tracked by sstate manifest +# Update BASE + +BASE="/OE/oe-core" +TMPDIR="${BASE}/tmp-eglibc" + +OUTPUT=${BASE}/sysroot.cruft.`date "+%s"` +WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$" + +mkdir ${OUTPUT} +find ${TMPDIR}/sstate-control -name \*populate-sysroot -o -name \*package | xargs cat | grep sysroots | \ + sed 's#tcbootstrapusr#tcbootstrap/usr#g; s#/$##g; s#///*#/#g' | \ + # work around for missing / in manifest for eglibc-initial, paths ending with / for directories and multiplied // (e.g. paths to native sysroot) + sort > ${OUTPUT}/master.list.all +sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes +find ${TMPDIR}/sysroots/ | \ + sort > ${OUTPUT}/sysroot.list + +diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates +diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all + +cp ${OUTPUT}/diff.all ${OUTPUT}/diff +for item in ${WHITELIST}; do + sed -i "/${item}/d" ${OUTPUT}/diff; +done + +# too many false positives for directories +# echo "Following files are installed in sysroot at least twice" +# cat ${OUTPUT}/duplicates + +echo "Following files are installed in sysroot, but not tracked by sstate" +cat ${OUTPUT}/diff -- 1.7.12.4 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCHv3] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-10-24 8:10 ` [PATCHv2] " Martin Jansa @ 2012-10-24 11:15 ` Martin Jansa 2012-11-15 7:30 ` [PATCHv4] " Martin Jansa 0 siblings, 1 reply; 11+ messages in thread From: Martin Jansa @ 2012-10-24 11:15 UTC (permalink / raw) To: openembedded-core * it's not very universal, but works with default oe-core setup and shows basic HOW-TO. It can be improved later. * eglibc-initial should be fixed, but for now better to add that work around then to show a lot of false positives Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- V2: added .pyo to WHITELIST shorter filenames TMPDIR added duplicates but not shown V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot was changed in febeaf3d1b8917b660c7279b008d8b03337568e9 scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 scripts/sstate-sysroot-cruft.sh diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh new file mode 100755 index 0000000..b68de42 --- /dev/null +++ b/scripts/sstate-sysroot-cruft.sh @@ -0,0 +1,34 @@ +#!/bin/sh + +# Used to find files installed in sysroot which are not tracked by sstate manifest +# Update BASE + +BASE="/OE/oe-core" +TMPDIR="${BASE}/tmp-eglibc" + +OUTPUT=${BASE}/sysroot.cruft.`date "+%s"` +WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$" + +mkdir ${OUTPUT} +find ${TMPDIR}/sstate-control -name \*.populate-sysroot\* -o -name \*.package\* | xargs cat | grep sysroots | \ + sed 's#tcbootstrapusr#tcbootstrap/usr#g; s#/$##g; s#///*#/#g' | \ + # work around for missing / in manifest for eglibc-initial, paths ending with / for directories and multiplied // (e.g. paths to native sysroot) + sort > ${OUTPUT}/master.list.all +sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes +find ${TMPDIR}/sysroots/ | \ + sort > ${OUTPUT}/sysroot.list + +diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates +diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all + +cp ${OUTPUT}/diff.all ${OUTPUT}/diff +for item in ${WHITELIST}; do + sed -i "/${item}/d" ${OUTPUT}/diff; +done + +# too many false positives for directories +# echo "Following files are installed in sysroot at least twice" +# cat ${OUTPUT}/duplicates + +echo "Following files are installed in sysroot, but not tracked by sstate" +cat ${OUTPUT}/diff -- 1.7.12.4 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-10-24 11:15 ` [PATCHv3] " Martin Jansa @ 2012-11-15 7:30 ` Martin Jansa 2012-11-15 15:25 ` Chris Larson 2012-11-15 16:34 ` [PATCHv5] " Martin Jansa 0 siblings, 2 replies; 11+ messages in thread From: Martin Jansa @ 2012-11-15 7:30 UTC (permalink / raw) To: openembedded-core * it's not very universal, but works with default oe-core setup and shows basic HOW-TO. It can be improved later. Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- V2: added .pyo to WHITELIST shorter filenames TMPDIR added duplicates but not shown V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot was changed in febeaf3d1b8917b660c7279b008d8b03337568e9 V4: dropped eglibc-initial work around, it was fixed in oe-core scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 scripts/sstate-sysroot-cruft.sh diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh new file mode 100755 index 0000000..ca23dcf --- /dev/null +++ b/scripts/sstate-sysroot-cruft.sh @@ -0,0 +1,34 @@ +#!/bin/sh + +# Used to find files installed in sysroot which are not tracked by sstate manifest +# Update BASE + +BASE="/OE/oe-core" +TMPDIR="${BASE}/tmp-eglibc" + +OUTPUT=${BASE}/sysroot.cruft.`date "+%s"` +WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$" + +mkdir ${OUTPUT} +find ${TMPDIR}/sstate-control -name \*.populate-sysroot\* -o -name \*.package\* | xargs cat | grep sysroots | \ + sed 's#/$##g; s#///*#/#g' | \ + # work around for paths ending with / for directories and multiplied // (e.g. paths to native sysroot) + sort > ${OUTPUT}/master.list.all +sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes +find ${TMPDIR}/sysroots/ | \ + sort > ${OUTPUT}/sysroot.list + +diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates +diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all + +cp ${OUTPUT}/diff.all ${OUTPUT}/diff +for item in ${WHITELIST}; do + sed -i "/${item}/d" ${OUTPUT}/diff; +done + +# too many false positives for directories +# echo "Following files are installed in sysroot at least twice" +# cat ${OUTPUT}/duplicates + +echo "Following files are installed in sysroot, but not tracked by sstate" +cat ${OUTPUT}/diff -- 1.8.0 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-11-15 7:30 ` [PATCHv4] " Martin Jansa @ 2012-11-15 15:25 ` Chris Larson 2012-11-15 16:19 ` Martin Jansa 2012-11-15 16:34 ` [PATCHv5] " Martin Jansa 1 sibling, 1 reply; 11+ messages in thread From: Chris Larson @ 2012-11-15 15:25 UTC (permalink / raw) To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 1758 bytes --] On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > * it's not very universal, but works with default oe-core setup and > shows basic HOW-TO. It can be improved later. > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > --- > V2: added .pyo to WHITELIST > shorter filenames > TMPDIR > added duplicates but not shown > > V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot > was changed in febeaf3d1b8917b660c7279b008d8b03337568e9 > > V4: dropped eglibc-initial work around, it was fixed in oe-core > > scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++ > 1 file changed, 34 insertions(+) > create mode 100755 scripts/sstate-sysroot-cruft.sh > > diff --git a/scripts/sstate-sysroot-cruft.sh > b/scripts/sstate-sysroot-cruft.sh > new file mode 100755 > index 0000000..ca23dcf > --- /dev/null > +++ b/scripts/sstate-sysroot-cruft.sh > @@ -0,0 +1,34 @@ > +#!/bin/sh > + > +# Used to find files installed in sysroot which are not tracked by sstate > manifest > +# Update BASE > + > +BASE="/OE/oe-core" > This seems interesting, but I have a few comments/concerns. 1) don't hardcode BASE, figure out the path relative to the script's location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)" 2) output files shouldn't go into oe-core directly, as oe-core isn't guaranteed to be writable, and it's more common to expect output from a script like this to go relative to the current directory, or a temp directory 3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that breaks for any distros or users which separate their tmpdirs by distro, or set TCLIBCAPPEND = "" -- Christopher Larson [-- Attachment #2: Type: text/html, Size: 2518 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-11-15 15:25 ` Chris Larson @ 2012-11-15 16:19 ` Martin Jansa 2012-11-15 16:27 ` Chris Larson 2012-11-15 16:49 ` Paul Eggleton 0 siblings, 2 replies; 11+ messages in thread From: Martin Jansa @ 2012-11-15 16:19 UTC (permalink / raw) To: Chris Larson; +Cc: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 2387 bytes --] On Thu, Nov 15, 2012 at 08:25:20AM -0700, Chris Larson wrote: > On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > > > * it's not very universal, but works with default oe-core setup and > > shows basic HOW-TO. It can be improved later. > > > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > > --- > > V2: added .pyo to WHITELIST > > shorter filenames > > TMPDIR > > added duplicates but not shown > > > > V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot > > was changed in febeaf3d1b8917b660c7279b008d8b03337568e9 > > > > V4: dropped eglibc-initial work around, it was fixed in oe-core > > > > scripts/sstate-sysroot-cruft.sh | 34 ++++++++++++++++++++++++++++++++++ > > 1 file changed, 34 insertions(+) > > create mode 100755 scripts/sstate-sysroot-cruft.sh > > > > diff --git a/scripts/sstate-sysroot-cruft.sh > > b/scripts/sstate-sysroot-cruft.sh > > new file mode 100755 > > index 0000000..ca23dcf > > --- /dev/null > > +++ b/scripts/sstate-sysroot-cruft.sh > > @@ -0,0 +1,34 @@ > > +#!/bin/sh > > + > > +# Used to find files installed in sysroot which are not tracked by sstate > > manifest > > +# Update BASE > > + > > +BASE="/OE/oe-core" > > > > This seems interesting, but I have a few comments/concerns. > > 1) don't hardcode BASE, figure out the path relative to the script's > location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)" > 2) output files shouldn't go into oe-core directly, as oe-core isn't > guaranteed to be writable, and it's more common to expect output from a > script like this to go relative to the current directory, or a temp > directory > 3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that > breaks for any distros or users which separate their tmpdirs by distro, or > set TCLIBCAPPEND = "" Ah in this case BASE is not directory with oe-core layer but one above. I didn't want to use bitbake -e, so that this script can be executed when some other bitbake process is running. As said it's not really universal (I wrote it just to confirm something and then shared it to find if someone else finds it also usefull to improve it). What about moving output to TMPDIR and setting TMPDIR by parameter? Cheers, -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-11-15 16:19 ` Martin Jansa @ 2012-11-15 16:27 ` Chris Larson 2012-11-15 17:09 ` Martin Jansa 2012-11-15 16:49 ` Paul Eggleton 1 sibling, 1 reply; 11+ messages in thread From: Chris Larson @ 2012-11-15 16:27 UTC (permalink / raw) To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 3024 bytes --] On Thu, Nov 15, 2012 at 9:19 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > On Thu, Nov 15, 2012 at 08:25:20AM -0700, Chris Larson wrote: > > On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com > >wrote: > > > > > * it's not very universal, but works with default oe-core setup and > > > shows basic HOW-TO. It can be improved later. > > > > > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > > > --- > > > V2: added .pyo to WHITELIST > > > shorter filenames > > > TMPDIR > > > added duplicates but not shown > > > > > > V3: use also populate-sysroot.MACHINE, manifest name for > populate-sysroot > > > was changed in febeaf3d1b8917b660c7279b008d8b03337568e9 > > > > > > V4: dropped eglibc-initial work around, it was fixed in oe-core > > > > > > scripts/sstate-sysroot-cruft.sh | 34 > ++++++++++++++++++++++++++++++++++ > > > 1 file changed, 34 insertions(+) > > > create mode 100755 scripts/sstate-sysroot-cruft.sh > > > > > > diff --git a/scripts/sstate-sysroot-cruft.sh > > > b/scripts/sstate-sysroot-cruft.sh > > > new file mode 100755 > > > index 0000000..ca23dcf > > > --- /dev/null > > > +++ b/scripts/sstate-sysroot-cruft.sh > > > @@ -0,0 +1,34 @@ > > > +#!/bin/sh > > > + > > > +# Used to find files installed in sysroot which are not tracked by > sstate > > > manifest > > > +# Update BASE > > > + > > > +BASE="/OE/oe-core" > > > > > > > This seems interesting, but I have a few comments/concerns. > > > > 1) don't hardcode BASE, figure out the path relative to the script's > > location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)" > > 2) output files shouldn't go into oe-core directly, as oe-core isn't > > guaranteed to be writable, and it's more common to expect output from a > > script like this to go relative to the current directory, or a temp > > directory > > 3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that > > breaks for any distros or users which separate their tmpdirs by distro, > or > > set TCLIBCAPPEND = "" > > Ah in this case BASE is not directory with oe-core layer but one above. > Still trivial to determine relative to the script's location. Adding an extra call to dirname, or an extra '/..' isn't particularly tough. I didn't want to use bitbake -e, so that this script can be executed > when some other bitbake process is running. > I don't see having to hardcode assumptions about the environment as a net win, personally. At least make them arguments, or add an argument to optionally use bitbake -e, or let it take vars from the environment. As said it's not really universal (I wrote it just to confirm something > and then shared it to find if someone else finds it also usefull to > improve it). > If it can't be used in a wide variety of circumstances, I'd argue against its inclusion in oe-core until such time that it's ready. The subject of this thread didn't include "RFC' ;) -- Christopher Larson [-- Attachment #2: Type: text/html, Size: 4262 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-11-15 16:27 ` Chris Larson @ 2012-11-15 17:09 ` Martin Jansa 0 siblings, 0 replies; 11+ messages in thread From: Martin Jansa @ 2012-11-15 17:09 UTC (permalink / raw) To: Chris Larson; +Cc: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 3610 bytes --] On Thu, Nov 15, 2012 at 09:27:52AM -0700, Chris Larson wrote: > On Thu, Nov 15, 2012 at 9:19 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > > > On Thu, Nov 15, 2012 at 08:25:20AM -0700, Chris Larson wrote: > > > On Thu, Nov 15, 2012 at 12:30 AM, Martin Jansa <martin.jansa@gmail.com > > >wrote: > > > > > > > * it's not very universal, but works with default oe-core setup and > > > > shows basic HOW-TO. It can be improved later. > > > > > > > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > > > > --- > > > > V2: added .pyo to WHITELIST > > > > shorter filenames > > > > TMPDIR > > > > added duplicates but not shown > > > > > > > > V3: use also populate-sysroot.MACHINE, manifest name for > > populate-sysroot > > > > was changed in febeaf3d1b8917b660c7279b008d8b03337568e9 > > > > > > > > V4: dropped eglibc-initial work around, it was fixed in oe-core > > > > > > > > scripts/sstate-sysroot-cruft.sh | 34 > > ++++++++++++++++++++++++++++++++++ > > > > 1 file changed, 34 insertions(+) > > > > create mode 100755 scripts/sstate-sysroot-cruft.sh > > > > > > > > diff --git a/scripts/sstate-sysroot-cruft.sh > > > > b/scripts/sstate-sysroot-cruft.sh > > > > new file mode 100755 > > > > index 0000000..ca23dcf > > > > --- /dev/null > > > > +++ b/scripts/sstate-sysroot-cruft.sh > > > > @@ -0,0 +1,34 @@ > > > > +#!/bin/sh > > > > + > > > > +# Used to find files installed in sysroot which are not tracked by > > sstate > > > > manifest > > > > +# Update BASE > > > > + > > > > +BASE="/OE/oe-core" > > > > > > > > > > This seems interesting, but I have a few comments/concerns. > > > > > > 1) don't hardcode BASE, figure out the path relative to the script's > > > location, e.g. BASE="$(cd $(dirname $(dirname $0)) && pwd)" > > > 2) output files shouldn't go into oe-core directly, as oe-core isn't > > > guaranteed to be writable, and it's more common to expect output from a > > > script like this to go relative to the current directory, or a temp > > > directory > > > 3) extract TMPDIR from bitbake -e, rather than hardcoding that, as that > > > breaks for any distros or users which separate their tmpdirs by distro, > > or > > > set TCLIBCAPPEND = "" > > > > Ah in this case BASE is not directory with oe-core layer but one above. > > > > Still trivial to determine relative to the script's location. Adding an > extra call to dirname, or an extra '/..' isn't particularly tough. Yes, but making another assumption that TMPDIR is on same level as oe-core checkout it. > I didn't want to use bitbake -e, so that this script can be executed > > when some other bitbake process is running. > > > > I don't see having to hardcode assumptions about the environment as a net > win, personally. At least make them arguments, or add an argument to > optionally use bitbake -e, or let it take vars from the environment. v5 is using argument or env variable (like sstate-cache-management.sh) > As said it's not really universal (I wrote it just to confirm something > > and then shared it to find if someone else finds it also usefull to > > improve it). > > > > If it can't be used in a wide variety of circumstances, I'd argue against > its inclusion in oe-core until such time that it's ready. The subject of > this thread didn't include "RFC' ;) I was using it in wide variety of circumstances just by updating 1 variable in script, but I agree that argument + shell history can make it easier. Cheers, -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-11-15 16:19 ` Martin Jansa 2012-11-15 16:27 ` Chris Larson @ 2012-11-15 16:49 ` Paul Eggleton 2012-11-15 17:03 ` Martin Jansa 1 sibling, 1 reply; 11+ messages in thread From: Paul Eggleton @ 2012-11-15 16:49 UTC (permalink / raw) To: Martin Jansa; +Cc: Chris Larson, openembedded-core On Thursday 15 November 2012 17:19:07 Martin Jansa wrote: > I didn't want to use bitbake -e, so that this script can be executed > when some other bitbake process is running. You really should not be trying to touch anything under TMPDIR let alone sstate files while bitbake is actually running against that same directory. I'm not sure why you would want to though... Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCHv4] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-11-15 16:49 ` Paul Eggleton @ 2012-11-15 17:03 ` Martin Jansa 0 siblings, 0 replies; 11+ messages in thread From: Martin Jansa @ 2012-11-15 17:03 UTC (permalink / raw) To: Paul Eggleton; +Cc: Chris Larson, openembedded-core [-- Attachment #1: Type: text/plain, Size: 1126 bytes --] On Thu, Nov 15, 2012 at 04:49:01PM +0000, Paul Eggleton wrote: > On Thursday 15 November 2012 17:19:07 Martin Jansa wrote: > > I didn't want to use bitbake -e, so that this script can be executed > > when some other bitbake process is running. > > You really should not be trying to touch anything under TMPDIR let alone > sstate files while bitbake is actually running against that same directory. I'm > not sure why you would want to though... While running builds for 8 MACHINEs in for loop I find it usefull to test one sysroot (from 1st MACHINE already built) while keeping build for remaining machines running for rest of night and sometimes following days.. At least that was my use-case when I wrote this script.. Yes I could have execute build only for 1st machine, wait till it's finished, then run script wait till it's finished and then start for loop for remainig 7, but that does not allow me to fall asleep while it's still building 1st :) or I have to plan it in advance and build with much longer one-liner. Cheers, -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCHv5] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate 2012-11-15 7:30 ` [PATCHv4] " Martin Jansa 2012-11-15 15:25 ` Chris Larson @ 2012-11-15 16:34 ` Martin Jansa 1 sibling, 0 replies; 11+ messages in thread From: Martin Jansa @ 2012-11-15 16:34 UTC (permalink / raw) To: openembedded-core Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- V2: added .pyo to WHITELIST shorter filenames TMPDIR added duplicates but not shown V3: use also populate-sysroot.MACHINE, manifest name for populate-sysroot was changed in febeaf3d1b8917b660c7279b008d8b03337568e9 V4: dropped eglibc-initial work around, it was fixed in oe-core V5: set tmpdir from param or env variable and show usage scripts/sstate-sysroot-cruft.sh | 78 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 scripts/sstate-sysroot-cruft.sh diff --git a/scripts/sstate-sysroot-cruft.sh b/scripts/sstate-sysroot-cruft.sh new file mode 100755 index 0000000..6caa252 --- /dev/null +++ b/scripts/sstate-sysroot-cruft.sh @@ -0,0 +1,78 @@ +#!/bin/sh + +# Used to find files installed in sysroot which are not tracked by sstate manifest + +# Global vars +tmpdir= + +usage () { + cat << EOF +Welcome to sysroot cruft finding utility. +$0 <OPTION> + +Options: + -h, --help + Display this help and exit. + + --tmpdir=<tmpdir> + Specify tmpdir, will use the environment variable TMPDIR if it is not specified. + Something like /OE/oe-core/tmp-eglibc (no / at the end). +EOF +} + +# Print error information and exit. +echo_error () { + echo "ERROR: $1" >&2 + exit 1 +} + +while [ -n "$1" ]; do + case $1 in + --tmpdir=*) + tmpdir=`echo $1 | sed -e 's#^--tmpdir=##' | xargs readlink -e` + [ -d "$tmpdir" ] || echo_error "Invalid argument to --tmpdir" + shift + ;; + --help|-h) + usage + exit 0 + ;; + *) + echo "Invalid arguments $*" + echo_error "Try '$0 -h' for more information." + ;; + esac +done + +# sstate cache directory, use environment variable TMPDIR +# if it was not specified, otherwise, error. +[ -n "$tmpdir" ] || tmpdir=$TMPDIR +[ -n "$tmpdir" ] || echo_error "No tmpdir found!" +[ -d "$tmpdir" ] || echo_error "Invalid tmpdir \"$tmpdir\"" + +OUTPUT=${tmpdir}/sysroot.cruft.`date "+%s"` +WHITELIST="\/var\/pseudo\/*[^\/]*$ \/shlibs$ \.pyc$ \.pyo$" + +mkdir ${OUTPUT} +find ${tmpdir}/sstate-control -name \*.populate-sysroot\* -o -name \*.package\* | xargs cat | grep sysroots | \ + sed 's#/$##g; s#///*#/#g' | \ + # work around for paths ending with / for directories and multiplied // (e.g. paths to native sysroot) + sort > ${OUTPUT}/master.list.all +sort -u ${OUTPUT}/master.list.all > ${OUTPUT}/master.list # -u because some directories are listed for more recipes +find ${tmpdir}/sysroots/ | \ + sort > ${OUTPUT}/sysroot.list + +diff ${OUTPUT}/master.list.all ${OUTPUT}/master.list > ${OUTPUT}/duplicates +diff ${OUTPUT}/master.list ${OUTPUT}/sysroot.list > ${OUTPUT}/diff.all + +cp ${OUTPUT}/diff.all ${OUTPUT}/diff +for item in ${WHITELIST}; do + sed -i "/${item}/d" ${OUTPUT}/diff; +done + +# too many false positives for directories +# echo "Following files are installed in sysroot at least twice" +# cat ${OUTPUT}/duplicates + +echo "Following files are installed in sysroot, but not tracked by sstate" +cat ${OUTPUT}/diff -- 1.8.0 ^ permalink raw reply related [flat|nested] 11+ messages in thread
end of thread, other threads:[~2012-11-15 17:23 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-10-23 22:51 [PATCH] scripts/sstate-sysroot-cruft.sh: add simple script to find files in sysroots not tracked by sstate Martin Jansa 2012-10-24 8:10 ` [PATCHv2] " Martin Jansa 2012-10-24 11:15 ` [PATCHv3] " Martin Jansa 2012-11-15 7:30 ` [PATCHv4] " Martin Jansa 2012-11-15 15:25 ` Chris Larson 2012-11-15 16:19 ` Martin Jansa 2012-11-15 16:27 ` Chris Larson 2012-11-15 17:09 ` Martin Jansa 2012-11-15 16:49 ` Paul Eggleton 2012-11-15 17:03 ` Martin Jansa 2012-11-15 16:34 ` [PATCHv5] " Martin Jansa
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox