[PATCH 0/2] Store a list of parsed configuration files

[PATCH 0/2] Store a list of parsed configuration files

[Joshua Lock]

This series addresses Yocto #1246 "Configuration changes could be saved to a
file the cooker isn't using" by having the parser store a list of
configuration files it has parsed in the data object and then switching the
cooker to verify against this list that the found file was parsed before
emitting the ConfigFilePathFound event.

The following changes since commit 1009ca570a750a00b0e60afcc30ead070c7b310a:

  hob: remove temporary directory on program shutdown (2011-07-30 12:21:18 -0700)

are available in the git repository at:
  git://github.com/incandescant/bitbake configfiles
  https://github.com/incandescant/bitbake/tree/configfiles

Joshua Lock (2):
  bb/parse: track parsed configuration files in __BB_PARSED_CONF
  bb/cooker: only emit ConfigFilePathFound for files which were parsed

 lib/bb/cooker.py                     |   18 +++++++++++++++++-
 lib/bb/parse/__init__.py             |   12 +++++++++++-
 lib/bb/parse/parse_py/ConfHandler.py |    2 ++
 3 files changed, 30 insertions(+), 2 deletions(-)

-- 
1.7.6

[PATCH 1/2] bb/parse: track parsed configuration files in __BB_PARSED_CONF

[Joshua Lock]

Keep a list of each configuration (.conf) file which is handled by the
ConfHandler in the internal __BB_PARSED_CONF variable.

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 lib/bb/parse/__init__.py             |   12 +++++++++++-
 lib/bb/parse/parse_py/ConfHandler.py |    2 ++
 2 files changed, 13 insertions(+), 1 deletions(-)

diff --git a/lib/bb/parse/__init__.py b/lib/bb/parse/__init__.py
index eee8d9c..afe7681 100644
--- a/lib/bb/parse/__init__.py
+++ b/lib/bb/parse/__init__.py
@@ -59,13 +59,23 @@ def update_mtime(f):
     __mtime_cache[f] = os.stat(f)[stat.ST_MTIME]
     return __mtime_cache[f]
 
-def mark_dependency(d, f):
+def full_path(f):
     if f.startswith('./'):
         f = "%s/%s" % (os.getcwd(), f[2:])
+    return f
+
+def mark_dependency(d, f):
+    full_path(f)
     deps = bb.data.getVar('__depends', d) or set()
     deps.update([(f, cached_mtime(f))])
     bb.data.setVar('__depends', deps, d)
 
+def track_conf(d, f):
+    full_path(f)
+    conf = bb.data.getVar('__BB_PARSED_CONF', d) or []
+    conf.append(f)
+    bb.data.setVar('__BB_PARSED_CONF', conf, d)
+
 def supports(fn, data):
     """Returns true if we have a handler for this file, false otherwise"""
     for h in handlers:
diff --git a/lib/bb/parse/parse_py/ConfHandler.py b/lib/bb/parse/parse_py/ConfHandler.py
index 102c0e9..953ef87 100644
--- a/lib/bb/parse/parse_py/ConfHandler.py
+++ b/lib/bb/parse/parse_py/ConfHandler.py
@@ -85,6 +85,8 @@ def handle(fn, data, include):
     if include:
         bb.parse.mark_dependency(data, abs_fn)
 
+    bb.parse.track_conf(data, abs_fn)
+
     statements = ast.StatementGroup()
     lineno = 0
     while True:
-- 
1.7.6

[PATCH 2/2] bb/cooker: only emit ConfigFilePathFound for files which were parsed

[Joshua Lock]

When the requested configuration file is found on disk check the
configuration object to ensure the file was parsed before emitting the
ConfigFilePathFound event. If it wasn't just return (and don't emit).

Fixes [YOCTO #1246]

Signed-off-by: Joshua Lock <josh@linux.intel.com>
---
 lib/bb/cooker.py |   18 +++++++++++++++++-
 1 files changed, 17 insertions(+), 1 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 025dfe4..a3456c3 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -609,9 +609,25 @@ class BBCooker:
                 collectlog.warn("No bb files matched BBFILE_PATTERN_%s '%s'" % (collection, pattern))
 
     def findConfigFilePath(self, configfile):
+        """
+        Find the location on disk of configfile and if it exists and was parsed by BitBake
+        emit the ConfigFilePathFound event with the path to the file.
+        """
         path = self._findConfigFile(configfile)
-        if path:
+        if not path:
+            return
+
+        parsed_confs = bb.data.getVar('__BB_PARSED_CONF', self.configuration.data) or []
+        if path and path in parsed_confs:
             bb.event.fire(bb.event.ConfigFilePathFound(path), self.configuration.data)
+        elif path:
+            _, conf, conffile = path.rpartition("conf/")
+            match = os.path.join(conf, conffile)
+            # Try and find matches for conf/conffilename.conf
+            for cfg in parsed_confs:
+                if cfg.endswith(match):
+                    bb.event.fire(bb.event.ConfigFilePathFound(path), self.configuration.data)
+                    break
 
     def findFilesMatchingInDir(self, filepattern, directory):
         """
-- 
1.7.6

Re: [PATCH 0/2] Store a list of parsed configuration files

[Richard Purdie]

On Tue, 2011-08-02 at 11:51 -0700, Joshua Lock wrote:
> This series addresses Yocto #1246 "Configuration changes could be saved to a
> file the cooker isn't using" by having the parser store a list of
> configuration files it has parsed in the data object and then switching the
> cooker to verify against this list that the found file was parsed before
> emitting the ConfigFilePathFound event.
> 
> The following changes since commit 1009ca570a750a00b0e60afcc30ead070c7b310a:
> 
>   hob: remove temporary directory on program shutdown (2011-07-30 12:21:18 -0700)
> 
> are available in the git repository at:
>   git://github.com/incandescant/bitbake configfiles
>   https://github.com/incandescant/bitbake/tree/configfiles
> 
> Joshua Lock (2):
>   bb/parse: track parsed configuration files in __BB_PARSED_CONF
>   bb/cooker: only emit ConfigFilePathFound for files which were parsed

Am I right in thinking this series is now superseded by the tweak to
include the information in __base_depends?

Cheers,

Richard

Re: [PATCH 0/2] Store a list of parsed configuration files

[Joshua Lock]

On Fri, 2011-08-05 at 18:12 +0100, Richard Purdie wrote:
> On Tue, 2011-08-02 at 11:51 -0700, Joshua Lock wrote:
> > This series addresses Yocto #1246 "Configuration changes could be saved to a
> > file the cooker isn't using" by having the parser store a list of
> > configuration files it has parsed in the data object and then switching the
> > cooker to verify against this list that the found file was parsed before
> > emitting the ConfigFilePathFound event.
> > 
> > The following changes since commit 1009ca570a750a00b0e60afcc30ead070c7b310a:
> > 
> >   hob: remove temporary directory on program shutdown (2011-07-30 12:21:18 -0700)
> > 
> > are available in the git repository at:
> >   git://github.com/incandescant/bitbake configfiles
> >   https://github.com/incandescant/bitbake/tree/configfiles
> > 
> > Joshua Lock (2):
> >   bb/parse: track parsed configuration files in __BB_PARSED_CONF
> >   bb/cooker: only emit ConfigFilePathFound for files which were parsed
> 
> Am I right in thinking this series is now superseded by the tweak to
> include the information in __base_depends?

That's right. The only reason I haven't sent that patch out yet is
because I'm in two minds about which is the cleanest way.

Because __base_depends a) contains more than just .conf files and b)
gets renamed I'm thinking the cleanest thing to do would be to iterate
__depends before the call to renameVar, pull out all conf files and
store them in self.configuration.config_files (or similar).

I wasn't sure adding an extra list to the configuration object would be
appreciated though...

Cheers,
Joshua
-- 
Joshua Lock
        Yocto Project "Johannes factotum"
        Intel Open Source Technology Centre

Re: [PATCH 0/2] Store a list of parsed configuration files

[Richard Purdie]

On Fri, 2011-08-05 at 11:07 -0700, Joshua Lock wrote:
> On Fri, 2011-08-05 at 18:12 +0100, Richard Purdie wrote:
> > On Tue, 2011-08-02 at 11:51 -0700, Joshua Lock wrote:
> > >   git://github.com/incandescant/bitbake configfiles
> > >   https://github.com/incandescant/bitbake/tree/configfiles
> > > 
> > > Joshua Lock (2):
> > >   bb/parse: track parsed configuration files in __BB_PARSED_CONF
> > >   bb/cooker: only emit ConfigFilePathFound for files which were parsed
> > 
> > Am I right in thinking this series is now superseded by the tweak to
> > include the information in __base_depends?
> 
> That's right. The only reason I haven't sent that patch out yet is
> because I'm in two minds about which is the cleanest way.
> 
> Because __base_depends a) contains more than just .conf files and b)
> gets renamed I'm thinking the cleanest thing to do would be to iterate
> __depends before the call to renameVar, pull out all conf files and
> store them in self.configuration.config_files (or similar).
> 
> I wasn't sure adding an extra list to the configuration object would be
> appreciated though...

I'm not sure we need a variable just for .conf files. Anyone wanting
conf files can generate that information easily enough...

Cheers,

Richard