* Re: [oe-commits] Richard Purdie : sstate.bbclass: Detect overwriting files in sstate controlled directories
[not found] <20120831003845.485CE10335@opal>
@ 2012-09-27 20:46 ` Andreas Müller
2012-09-28 6:03 ` Andreas Müller
1 sibling, 0 replies; 3+ messages in thread
From: Andreas Müller @ 2012-09-27 20:46 UTC (permalink / raw)
To: openembedded-devel
On Fri, Aug 31, 2012 at 2:38 AM, <git@git.openembedded.org> wrote:
> Module: openembedded-core.git
> Branch: master
> Commit: 1f1b6f93d6b7aa8c9bd9bb5b1826997812e36932
> URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=1f1b6f93d6b7aa8c9bd9bb5b1826997812e36932
>
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date: Sat Aug 25 15:48:14 2012 +0000
>
> sstate.bbclass: Detect overwriting files in sstate controlled directories
>
> We have a potential problem where two sstate packages try and touch the same
> file. This adds code which will print a warning whenever this happens.
>
> The implementation does but by maintaining a master file list and comparing
> file accesses against this. There are a number of places we have duplicate
> accesses which are harmless, mostly in the deploy directory so these
> are whitelisted.
>
> For now the code prints warnings, this could be strengthened in future to
> become error messages. Whilst working on this code on and off over the past
> few months various issues were uncovered, some serious.
>
> [YOCTO #238]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> ---
>
> meta/classes/sstate.bbclass | 45 ++++++++++++++++++++++++++++++++++++++++++-
> 1 files changed, 44 insertions(+), 1 deletions(-)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 6762e32..02594a7 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -3,6 +3,7 @@ SSTATE_VERSION = "2"
> SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
> SSTATE_MANFILEBASE = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-"
> SSTATE_MANFILEPREFIX = "${SSTATE_MANFILEBASE}${PN}"
> +SSTATE_MASTERMANIFEST = "${SSTATE_MANIFESTS}/master.list"
>
> def generate_sstatefn(spec, hash, d):
> if not hash:
> @@ -17,6 +18,7 @@ SSTATE_EXTRAPATH = ""
> SSTATE_EXTRAPATHWILDCARD = ""
> SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/${SSTATE_PKGSPEC}"
>
> +SSTATE_DUPWHITELIST = "${DEPLOY_DIR_IMAGE}/ ${DEPLOY_DIR}/licenses/ ${DEPLOY_DIR_IPK}/all/ ${DEPLOY_DIR_RPM}/all ${DEPLOY_DIR_DEB}/all/"
>
> SSTATE_SCAN_FILES ?= "*.la *-config *_config"
> SSTATE_SCAN_CMD ?= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES", True).split())}" \) -type f'
> @@ -125,7 +127,6 @@ def sstate_install(ss, d):
> locks.append(bb.utils.lockfile(lock))
>
> for state in ss['dirs']:
> - oe.path.copytree(state[1], state[2])
> for walkroot, dirs, files in os.walk(state[1]):
> bb.debug(2, "Staging files from %s to %s" % (state[1], state[2]))
> for file in files:
> @@ -140,9 +141,35 @@ def sstate_install(ss, d):
> if not dstdir.endswith("/"):
> dstdir = dstdir + "/"
> shareddirs.append(dstdir)
> +
> + # Check the file list for conflicts against the master manifest
> + mastermanifest = d.getVar("SSTATE_MASTERMANIFEST", True)
> + whitelist = d.getVar("SSTATE_DUPWHITELIST", True)
> + lock = bb.utils.lockfile(mastermanifest + ".lock")
> + fileslist = [line.strip() for line in open(mastermanifest)]
> + bb.utils.unlockfile(lock)
> + match = []
> + for f in sharedfiles:
> + if f in fileslist:
> + realmatch = True
> + for w in whitelist:
> + if f.startswith(w):
> + realmatch = False
> + break
> + if realmatch:
> + match.append(f)
> + if match:
> + bb.warn("The recipe is trying to install files into a shared area when those files already exist. Those files are:\n %s" % "\n ".join(match))
> +
> + # Write out the manifest and add to the task's manifest file
> + lock = bb.utils.lockfile(mastermanifest + ".lock")
> + mf = open(mastermanifest, "a")
> f = open(manifest, "w")
> for file in sharedfiles:
> + mf.write(file + "\n")
> f.write(file + "\n")
> + bb.utils.unlockfile(lock)
> +
> # We want to ensure that directories appear at the end of the manifest
> # so that when we test to see if they should be deleted any contents
> # added by the task will have been removed first.
> @@ -152,6 +179,10 @@ def sstate_install(ss, d):
> f.write(di + "\n")
> f.close()
>
> + # Run the actual file install
> + for state in ss['dirs']:
> + oe.path.copytree(state[1], state[2])
> +
> for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
> bb.build.exec_func(postinst, d)
>
> @@ -268,6 +299,18 @@ def sstate_clean_manifest(manifest, d):
> except OSError:
> pass
>
> + # Remove the entries from the master manifest
> + mastermanifest = d.getVar("SSTATE_MASTERMANIFEST", True)
> + lock = bb.utils.lockfile(mastermanifest + ".lock")
> + mf = open(mastermanifest + ".new", "w")
> + for line in open(mastermanifest, "r"):
> + if not line or line in entries:
> + continue
> + mf.write(line)
> + mf.close()
> + os.rename(mastermanifest + ".new", mastermanifest)
> + bb.utils.unlockfile(lock)
> +
> oe.path.remove(manifest)
>
> def sstate_clean(ss, d):
>
>
I think it started with this one: I often do play around with
different kernel versions/sources to bisect when a bug was born. Doing
so I now get wallpapers with
WARNING: The recipe is trying to install files into a shared area when
those files already exist. Those files are:
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/Module.symvers
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/CREDITS
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/kernel-abiversion
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/Makefile
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/kernel-image-name
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/uImage
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/README
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/COPYING
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/REPORTING-BUGS
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/System.map-3.5.0
...
(log.do_populate_sysroot is 2.8 MB)
Would it cause huge effort to add a variable which is in scope of
recipe avoiding this warning because it scrolls out all other
interesting messages?
Andreas
^ permalink raw reply [flat|nested] 3+ messages in thread* Re: [oe-commits] Richard Purdie : sstate.bbclass: Detect overwriting files in sstate controlled directories
[not found] <20120831003845.485CE10335@opal>
2012-09-27 20:46 ` [oe-commits] Richard Purdie : sstate.bbclass: Detect overwriting files in sstate controlled directories Andreas Müller
@ 2012-09-28 6:03 ` Andreas Müller
2012-09-28 10:22 ` Richard Purdie
1 sibling, 1 reply; 3+ messages in thread
From: Andreas Müller @ 2012-09-28 6:03 UTC (permalink / raw)
To: openembedded-core
On Fri, Aug 31, 2012 at 2:38 AM, <git@git.openembedded.org> wrote:
> Module: openembedded-core.git
> Branch: master
> Commit: 1f1b6f93d6b7aa8c9bd9bb5b1826997812e36932
> URL: http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=1f1b6f93d6b7aa8c9bd9bb5b1826997812e36932
>
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date: Sat Aug 25 15:48:14 2012 +0000
>
> sstate.bbclass: Detect overwriting files in sstate controlled directories
>
> We have a potential problem where two sstate packages try and touch the same
> file. This adds code which will print a warning whenever this happens.
>
> The implementation does but by maintaining a master file list and comparing
> file accesses against this. There are a number of places we have duplicate
> accesses which are harmless, mostly in the deploy directory so these
> are whitelisted.
>
> For now the code prints warnings, this could be strengthened in future to
> become error messages. Whilst working on this code on and off over the past
> few months various issues were uncovered, some serious.
>
> [YOCTO #238]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> ---
>
> meta/classes/sstate.bbclass | 45 ++++++++++++++++++++++++++++++++++++++++++-
> 1 files changed, 44 insertions(+), 1 deletions(-)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 6762e32..02594a7 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -3,6 +3,7 @@ SSTATE_VERSION = "2"
> SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
> SSTATE_MANFILEBASE = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-"
> SSTATE_MANFILEPREFIX = "${SSTATE_MANFILEBASE}${PN}"
> +SSTATE_MASTERMANIFEST = "${SSTATE_MANIFESTS}/master.list"
>
> def generate_sstatefn(spec, hash, d):
> if not hash:
> @@ -17,6 +18,7 @@ SSTATE_EXTRAPATH = ""
> SSTATE_EXTRAPATHWILDCARD = ""
> SSTATE_PATHSPEC = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/${SSTATE_PKGSPEC}"
>
> +SSTATE_DUPWHITELIST = "${DEPLOY_DIR_IMAGE}/ ${DEPLOY_DIR}/licenses/ ${DEPLOY_DIR_IPK}/all/ ${DEPLOY_DIR_RPM}/all ${DEPLOY_DIR_DEB}/all/"
>
> SSTATE_SCAN_FILES ?= "*.la *-config *_config"
> SSTATE_SCAN_CMD ?= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES", True).split())}" \) -type f'
> @@ -125,7 +127,6 @@ def sstate_install(ss, d):
> locks.append(bb.utils.lockfile(lock))
>
> for state in ss['dirs']:
> - oe.path.copytree(state[1], state[2])
> for walkroot, dirs, files in os.walk(state[1]):
> bb.debug(2, "Staging files from %s to %s" % (state[1], state[2]))
> for file in files:
> @@ -140,9 +141,35 @@ def sstate_install(ss, d):
> if not dstdir.endswith("/"):
> dstdir = dstdir + "/"
> shareddirs.append(dstdir)
> +
> + # Check the file list for conflicts against the master manifest
> + mastermanifest = d.getVar("SSTATE_MASTERMANIFEST", True)
> + whitelist = d.getVar("SSTATE_DUPWHITELIST", True)
> + lock = bb.utils.lockfile(mastermanifest + ".lock")
> + fileslist = [line.strip() for line in open(mastermanifest)]
> + bb.utils.unlockfile(lock)
> + match = []
> + for f in sharedfiles:
> + if f in fileslist:
> + realmatch = True
> + for w in whitelist:
> + if f.startswith(w):
> + realmatch = False
> + break
> + if realmatch:
> + match.append(f)
> + if match:
> + bb.warn("The recipe is trying to install files into a shared area when those files already exist. Those files are:\n %s" % "\n ".join(match))
> +
> + # Write out the manifest and add to the task's manifest file
> + lock = bb.utils.lockfile(mastermanifest + ".lock")
> + mf = open(mastermanifest, "a")
> f = open(manifest, "w")
> for file in sharedfiles:
> + mf.write(file + "\n")
> f.write(file + "\n")
> + bb.utils.unlockfile(lock)
> +
> # We want to ensure that directories appear at the end of the manifest
> # so that when we test to see if they should be deleted any contents
> # added by the task will have been removed first.
> @@ -152,6 +179,10 @@ def sstate_install(ss, d):
> f.write(di + "\n")
> f.close()
>
> + # Run the actual file install
> + for state in ss['dirs']:
> + oe.path.copytree(state[1], state[2])
> +
> for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
> bb.build.exec_func(postinst, d)
>
> @@ -268,6 +299,18 @@ def sstate_clean_manifest(manifest, d):
> except OSError:
> pass
>
> + # Remove the entries from the master manifest
> + mastermanifest = d.getVar("SSTATE_MASTERMANIFEST", True)
> + lock = bb.utils.lockfile(mastermanifest + ".lock")
> + mf = open(mastermanifest + ".new", "w")
> + for line in open(mastermanifest, "r"):
> + if not line or line in entries:
> + continue
> + mf.write(line)
> + mf.close()
> + os.rename(mastermanifest + ".new", mastermanifest)
> + bb.utils.unlockfile(lock)
> +
> oe.path.remove(manifest)
>
> def sstate_clean(ss, d):
>
This previous addressed the wrong list - so again:
I think it started with this one: I often do play around with
different kernel versions/sources to bisect when a bug was born. Doing
so I now get wallpapers with
WARNING: The recipe is trying to install files into a shared area when
those files already exist. Those files are:
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/Module.symvers
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/CREDITS
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/kernel-abiversion
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/Makefile
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/kernel-image-name
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/uImage
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/README
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/COPYING
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/REPORTING-BUGS
/home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/System.map-3.5.0
...
(log.do_populate_sysroot is 2.8 MB)
Would it cause huge effort to add a variable which is in scope of
recipe avoiding this warning because it scrolls out all other
interesting messages?
Andreas
^ permalink raw reply [flat|nested] 3+ messages in thread