v3 or so [PATCH 0/1] File inclusion tracking (cleaned up)

v3 or so [PATCH 0/1] File inclusion tracking (cleaned up)

[Peter Seebach]

This is *just* the file inclusion tracking. Spent a while looking
at it, improved it, got some great advice from a friend who is
better at Python, improved it more, and ended up with something I think
is much more maintainable and much clearer. The include tracking
cost is minimal (the expensive stuff went away), so it's currently
done unconditionally.

As before, the main implication is that bitbake -e output starts out:

        #
        # INCLUDE HISTORY:
        #
        # /home/seebs/poky/build/conf/bblayers.conf
        # /home/seebs/poky/meta/conf/layer.conf
        # /home/seebs/poky/meta-yocto/conf/layer.conf
        # /home/seebs/poky/meta-yocto-bsp/conf/layer.conf
        # conf/bitbake.conf includes:
        #   /home/seebs/poky/meta/conf/abi_version.conf
        [...]

The idea is that if you are getting syntax errors in a file, and you
don't even know *why*, this makes it possible to see. For a smallish
build, the include history is only maybe 50-60 lines, so it's not a
big impact on the size of the bitbake -e output.

The following changes since commit ac75b06744e73399ca1fbda322ef851ae5754b0a:
  Valentin Popa (1):
        Implement 'settings' dialog as designed

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib seebs/include
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=seebs/include

Peter Seebach (1):
  data_smart.py and friends: Track file inclusions for bitbake -e

 lib/bb/cooker.py         |    5 +++++
 lib/bb/data_smart.py     |   44 ++++++++++++++++++++++++++++++++++++++++++++
 lib/bb/parse/__init__.py |    3 ++-
 3 files changed, 51 insertions(+), 1 deletions(-)

[PATCH 1/1] data_smart.py and friends: Track file inclusions for bitbake -e

[Peter Seebach]

This code adds inclusion history to bitbake -e output, showing
which files were included, in what order. This doesn't completely
resolve timing questions, because it doesn't show you which lines
of a file were processed before or after a given include, but it
does let you figure out what the path was by which a particular
file ended up in your build at all.

How it works: data_smart acquires a .history member, which is an
IncludeHistory; this represents the inclusion of a file and all its
inclusions, recursively. It provides methods for including files,
for finishing inclusion (done as an __exit__), and for
dumping the whole tree.

The parser is modified to run includes inside a with() to push
and pop the include filename.

Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
---
 lib/bb/cooker.py         |    5 +++++
 lib/bb/data_smart.py     |   44 ++++++++++++++++++++++++++++++++++++++++++++
 lib/bb/parse/__init__.py |    3 ++-
 3 files changed, 51 insertions(+), 1 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 237019c..4642753 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -332,6 +332,11 @@ class BBCooker:
                 parselog.exception("Unable to read %s", fn)
                 raise
 
+        # Display history
+        with closing(StringIO()) as env:
+            self.configuration.data.history.emit(env)
+            logger.plain(env.getvalue())
+
         # emit variables and shell functions
         data.update_data(envdata)
         with closing(StringIO()) as env:
diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index f5f3b13..7cb53d8 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -111,6 +111,48 @@ class ExpansionError(Exception):
     def __str__(self):
         return self.msg
 
+class IncludeHistory(object):
+    def __init__(self, parent = None, filename = None):
+        self.parent = parent
+        if parent:
+            self.top = parent.top
+        else:
+            self.top = self
+        self.filename = filename or '[TOP LEVEL]'
+        self.children = []
+        self.current = self
+
+    def include(self, filename):
+        newfile = IncludeHistory(self.current, filename)
+        self.current.children.append(newfile)
+        self.current = newfile
+        return self
+
+    def __enter__(self):
+        pass
+
+    def __exit__(self, a, b, c):
+        if self.current.parent:
+            self.current = self.current.parent
+        else:
+            bb.warn("Include log: Tried to finish '%s' at top level." % filename)
+        return False
+
+    def emit(self, o, level = 0):
+        """Emit an include history file, and its children."""
+        if self != self.top:
+            spaces = "  " * level
+            o.write("# %s%s" % (spaces, self.filename))
+            if len(self.children) > 0:
+                o.write(" includes:")
+            o.write("\n")
+            level = level + 1
+        else:
+            o.write("#\n# INCLUDE HISTORY:\n#\n")
+        for child in self.children:
+            child.emit(o, level)
+
+
 class DataSmart(MutableMapping):
     def __init__(self, special = COWDictBase.copy(), seen = COWDictBase.copy() ):
         self.dict = {}
@@ -118,6 +160,7 @@ class DataSmart(MutableMapping):
         # cookie monster tribute
         self._special_values = special
         self._seen_overrides = seen
+        self.history = IncludeHistory()
 
         self.expand_cache = {}
 
@@ -411,6 +454,7 @@ class DataSmart(MutableMapping):
         # we really want this to be a DataSmart...
         data = DataSmart(seen=self._seen_overrides.copy(), special=self._special_values.copy())
         data.dict["_data"] = self.dict
+        data.history = copy.deepcopy(self.history)
 
         return data
 
diff --git a/lib/bb/parse/__init__.py b/lib/bb/parse/__init__.py
index 7b9c47e..b24673b 100644
--- a/lib/bb/parse/__init__.py
+++ b/lib/bb/parse/__init__.py
@@ -88,7 +88,8 @@ def handle(fn, data, include = 0):
     """Call the handler that is appropriate for this file"""
     for h in handlers:
         if h['supports'](fn, data):
-            return h['handle'](fn, data, include)
+            with data.history.include(fn):
+                return h['handle'](fn, data, include)
     raise ParseError("not a BitBake file", fn)
 
 def init(fn, data):
-- 
1.7.0.4