[PATCH] sstate: add manifest info for shared file matches

[PATCH] sstate: add manifest info for shared file matches

[Saul Wold]

Present the manifest file that contains the matches for
files being installed to a location that already contains
that file. This will help to determine which is the correct
recipe to fix when this occurs.

[YOCTO #3191]

Signed-off-by: Saul Wold <sgw@linux.intel.com>
---
 meta/classes/sstate.bbclass |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index d2a120b..140dd59 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -27,7 +27,7 @@ SSTATE_DUPWHITELIST += "${STAGING_LIBDIR_NATIVE}/${MULTIMACH_TARGET_SYS} ${STAGI
 # Also avoid python issues until we fix the python recipe
 SSTATE_DUPWHITELIST += "${STAGING_LIBDIR}/python2.7/config/Makefile ${STAGING_LIBDIR}/libpython2.7 ${STAGING_INCDIR}/python2.7/pyconfig.h"
 # Avoid docbook/sgml catalog warnings for now
-SSTATE_DUPWHITELIST += "${STAGING_ETCDIR_NATIVE}/sgml ${STAGING_DATADIR_NATIVE}/sgml"
+#SSTATE_DUPWHITELIST += "${STAGING_ETCDIR_NATIVE}/sgml ${STAGING_DATADIR_NATIVE}/sgml"
 
 SSTATE_SCAN_FILES ?= "*.la *-config *_config"
 SSTATE_SCAN_CMD ?= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES", True).split())}" \) -type f'
@@ -120,6 +120,7 @@ def sstate_add(ss, source, dest, d):
 
 def sstate_install(ss, d):
     import oe.path
+    import subprocess
 
     sharedfiles = []
     shareddirs = []
@@ -164,8 +165,12 @@ def sstate_install(ss, d):
                     break
             if realmatch:
                 match.append(f)
+                sstate_search_cmd = "grep -rl %s %s | sed -e 's:^.*/::' -e 's:\.populate-sysroot::'" % (f, d.expand("${SSTATE_MANIFESTS}"))
+                search_output = subprocess.check_output(sstate_search_cmd, stderr=subprocess.STDOUT, shell=True)
+                if search_output != None:
+                    match.append("Matched in %s" % search_output.rstrip())
     if match:
-        bb.warn("The recipe %s is trying to install files into a shared area when those files already exist. Those files are:\n   %s" % (d.getVar("PN", True), "\n   ".join(match)))
+        bb.warn("The recipe %s is trying to install files into a shared area when those files already exist. Those files and their manifest location are:\n   %s\nPlease verify which package should provide the above files." % (d.getVar('PN', True), "\n   ".join(match)))
 
     # Write out the manifest
     f = open(manifest, "w")
-- 
1.7.9.5

Re: [PATCH] sstate: add manifest info for shared file matches

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

On Thu, Oct 18, 2012 at 12:25:05PM -0700, Saul Wold wrote:
> Present the manifest file that contains the matches for
> files being installed to a location that already contains
> that file. This will help to determine which is the correct
> recipe to fix when this occurs.
> 
> [YOCTO #3191]

Can we do the same when removing files from sysroot?

To catch errors like this:
http://git.openembedded.org/openembedded-core/commit/?id=6b12d4cd39bacb087654b59e25f5052a4e839b26

12:40:15 < JaMa> it depends on order of tasks
12:41:27 < JaMa> gettext-native depends on gettext-minimal-native, so gettext-minimal-native installs own config.rpath to sysroot (when gettext-native "owns"
                 it too)
12:41:55 < JaMa> then gettext-native is built and before populting sysroot it removes own files there (including config.rpath)
12:43:45 < JaMa> and because sstate assumes that every file in sysroot is provided by exactly one recipe, gettext-minimal-native looses his config.rpath
12:44:49 < JaMa> now there is at least warning when 2 recipes are installing the same file to sysroot, but that doesn't resolve this issues when some file is
                 "moved" between recipes
12:47:30 < JaMa> because gettext-minimal-native is only recipe installing it
12:47:39 < JaMa> and gettext-native is only removing it

If we check manifests from other recipes before removing files we can show warning that
gettext-native is trying to remove config.rpath now provided by gettext-minimal-native
and keep config.rpath in sysroot.

Now everybody doing incremental builds needs to call
bitbake -c cleansstate gettext-minimal-native

Cheers,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] sstate: add manifest info for shared file matches

[Richard Purdie]

On Fri, 2012-10-19 at 12:55 +0200, Martin Jansa wrote:
> On Thu, Oct 18, 2012 at 12:25:05PM -0700, Saul Wold wrote:
> > Present the manifest file that contains the matches for
> > files being installed to a location that already contains
> > that file. This will help to determine which is the correct
> > recipe to fix when this occurs.
> > 
> > [YOCTO #3191]
> 
> Can we do the same when removing files from sysroot?
> 
> To catch errors like this:
> http://git.openembedded.org/openembedded-core/commit/?id=6b12d4cd39bacb087654b59e25f5052a4e839b26
> 
> 12:40:15 < JaMa> it depends on order of tasks
> 12:41:27 < JaMa> gettext-native depends on gettext-minimal-native, so gettext-minimal-native installs own config.rpath to sysroot (when gettext-native "owns"
>                  it too)
> 12:41:55 < JaMa> then gettext-native is built and before populting sysroot it removes own files there (including config.rpath)
> 12:43:45 < JaMa> and because sstate assumes that every file in sysroot is provided by exactly one recipe, gettext-minimal-native looses his config.rpath
> 12:44:49 < JaMa> now there is at least warning when 2 recipes are installing the same file to sysroot, but that doesn't resolve this issues when some file is
>                  "moved" between recipes
> 12:47:30 < JaMa> because gettext-minimal-native is only recipe installing it
> 12:47:39 < JaMa> and gettext-native is only removing it
> 
> If we check manifests from other recipes before removing files we can show warning that
> gettext-native is trying to remove config.rpath now provided by gettext-minimal-native
> and keep config.rpath in sysroot.

The plan is to make it a hard error if something installs a file that
already exists. This issue will be resolved when that happens and I want
to get there sooner than later.

Running the check at sysroot clean time is therefore going to complicate
the code a lot and damage performance with no real end need...

> Now everybody doing incremental builds needs to call
> bitbake -c cleansstate gettext-minimal-native

-c clean should be sufficient. I wish people would use appropriately
sized hammers and this is why I never wanted cleansstate in the first
place :(.

Cheers,

Richard

Re: [PATCH] sstate: add manifest info for shared file matches

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2560 bytes --]

On Fri, Oct 19, 2012 at 12:07:05PM +0100, Richard Purdie wrote:
> On Fri, 2012-10-19 at 12:55 +0200, Martin Jansa wrote:
> > On Thu, Oct 18, 2012 at 12:25:05PM -0700, Saul Wold wrote:
> > > Present the manifest file that contains the matches for
> > > files being installed to a location that already contains
> > > that file. This will help to determine which is the correct
> > > recipe to fix when this occurs.
> > > 
> > > [YOCTO #3191]
> > 
> > Can we do the same when removing files from sysroot?
> > 
> > To catch errors like this:
> > http://git.openembedded.org/openembedded-core/commit/?id=6b12d4cd39bacb087654b59e25f5052a4e839b26
> > 
> > 12:40:15 < JaMa> it depends on order of tasks
> > 12:41:27 < JaMa> gettext-native depends on gettext-minimal-native, so gettext-minimal-native installs own config.rpath to sysroot (when gettext-native "owns"
> >                  it too)
> > 12:41:55 < JaMa> then gettext-native is built and before populting sysroot it removes own files there (including config.rpath)
> > 12:43:45 < JaMa> and because sstate assumes that every file in sysroot is provided by exactly one recipe, gettext-minimal-native looses his config.rpath
> > 12:44:49 < JaMa> now there is at least warning when 2 recipes are installing the same file to sysroot, but that doesn't resolve this issues when some file is
> >                  "moved" between recipes
> > 12:47:30 < JaMa> because gettext-minimal-native is only recipe installing it
> > 12:47:39 < JaMa> and gettext-native is only removing it
> > 
> > If we check manifests from other recipes before removing files we can show warning that
> > gettext-native is trying to remove config.rpath now provided by gettext-minimal-native
> > and keep config.rpath in sysroot.
> 
> The plan is to make it a hard error if something installs a file that
> already exists. This issue will be resolved when that happens and I want
> to get there sooner than later.

For new recipes yes, but that does not resolve this case when one file
is moved from one recipe to another.
 
> Running the check at sysroot clean time is therefore going to complicate
> the code a lot and damage performance with no real end need...
> 
> > Now everybody doing incremental builds needs to call
> > bitbake -c cleansstate gettext-minimal-native
> 
> -c clean should be sufficient. I wish people would use appropriately
> sized hammers and this is why I never wanted cleansstate in the first
> place :(.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] sstate: add manifest info for shared file matches

[Chris Larson]

On Thu, Oct 18, 2012 at 12:25 PM, Saul Wold <sgw@linux.intel.com> wrote:
>  # Avoid docbook/sgml catalog warnings for now
> -SSTATE_DUPWHITELIST += "${STAGING_ETCDIR_NATIVE}/sgml ${STAGING_DATADIR_NATIVE}/sgml"
> +#SSTATE_DUPWHITELIST += "${STAGING_ETCDIR_NATIVE}/sgml ${STAGING_DATADIR_NATIVE}/sgml"

Was this block of the patch intended? It's not mentioned in the commit
message, that I can see.
-- 
Christopher Larson