All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 01/16] cooker: Allow profiling of the parser in profile mode
@ 2013-09-16 21:53 Richard Purdie
  2013-09-16 21:53 ` [PATCH 02/16] data: Be explicit in data_db check Richard Purdie
                   ` (14 more replies)
  0 siblings, 15 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index a07615b..2c8d4dc 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1480,7 +1480,7 @@ class Feeder(multiprocessing.Process):
                 continue
 
 class Parser(multiprocessing.Process):
-    def __init__(self, jobs, results, quit, init):
+    def __init__(self, jobs, results, quit, init, profile):
         self.jobs = jobs
         self.results = results
         self.quit = quit
@@ -1488,8 +1488,28 @@ class Parser(multiprocessing.Process):
         multiprocessing.Process.__init__(self)
         self.context = bb.utils.get_context().copy()
         self.handlers = bb.event.get_class_handlers().copy()
+        self.profile = profile
 
     def run(self):
+
+        if not self.profile:
+            self.realrun()
+            return
+
+        try:
+            import cProfile as profile
+        except:
+            import profile
+        prof = profile.Profile()
+        try:
+            profile.Profile.runcall(prof, self.realrun)
+        finally:
+            logfile = "profile-parse-%s.log" % multiprocessing.current_process().name
+            prof.dump_stats(logfile)
+            bb.utils.process_profilelog(logfile)
+            print("Raw profiling information saved to %s and processed statistics to %s.processed" % (logfile, logfile))
+
+    def realrun(self):
         if self.init:
             self.init()
 
@@ -1590,7 +1610,7 @@ class CookerParser(object):
             self.feeder = Feeder(self.willparse, self.jobs, self.feeder_quit)
             self.feeder.start()
             for i in range(0, self.num_processes):
-                parser = Parser(self.jobs, self.result_queue, self.parser_quit, init)
+                parser = Parser(self.jobs, self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
                 parser.start()
                 self.processes.append(parser)
 
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 02/16] data: Be explicit in data_db check
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 03/16] siggen: Use lookup cache exclusively Richard Purdie
                   ` (13 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

The if statement current causes the size of parent to be calcuated which
is like a len() operation on a datastore. Since we're only interested
whether the value is none, checking explictly for this gives a
small performance gain.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index 8c9cb0f..e6d5232 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -59,7 +59,7 @@ def init():
 def init_db(parent = None):
     """Return a new object representing the Bitbake data,
     optionally based on an existing object"""
-    if parent:
+    if parent is not None:
         return parent.createCopy()
     else:
         return _dict_type()
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 03/16] siggen: Use lookup cache exclusively
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
  2013-09-16 21:53 ` [PATCH 02/16] data: Be explicit in data_db check Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 04/16] data_smart: Improve variable expansion regexp Richard Purdie
                   ` (12 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

All the values we need are already guaranteed to be in the lookupcache
so rather than fetch variables again, just use the cache. This gives a
small performance improvement and simplifies the code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/siggen.py | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index fb8b678..c15ba28 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -91,8 +91,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
         basehash = {}
 
         for task in tasklist:
-            data = d.getVar(task, False)
-            lookupcache[task] = data
+            data = lookupcache[task]
 
             if data is None:
                 bb.error("Task %s from %s seems to be empty?!" % (task, fn))
@@ -115,16 +114,8 @@ class SignatureGeneratorBasic(SignatureGenerator):
             alldeps = sorted(seen)
             for dep in alldeps:
                 data = data + dep
-                if dep in lookupcache:
-                    var = lookupcache[dep]
-                elif dep[-1] == ']':
-                    vf = dep[:-1].split('[')
-                    var = d.getVarFlag(vf[0], vf[1], False)
-                    lookupcache[dep] = var
-                else:
-                    var = d.getVar(dep, False)
-                    lookupcache[dep] = var
-                if var:
+                var = lookupcache[dep]
+                if var is not None:
                     data = data + str(var)
             self.basehash[fn + "." + task] = hashlib.md5(data).hexdigest()
             taskdeps[task] = alldeps
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 04/16] data_smart: Improve variable expansion regexp
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
  2013-09-16 21:53 ` [PATCH 02/16] data: Be explicit in data_db check Richard Purdie
  2013-09-16 21:53 ` [PATCH 03/16] siggen: Use lookup cache exclusively Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 05/16] data_smart: use the expand_cache in VariableParse Richard Purdie
                   ` (11 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Debugging showed the variable expansion regexp was catching python
expressions (starting with @). Since these are caught by their own
dedicated regexp, stop matching these for the plain variable expansion
for small performance improvements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 635b259..9be5d5e 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -40,7 +40,7 @@ logger = logging.getLogger("BitBake.Data")
 
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
 __setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>.*))?$')
-__expand_var_regexp__ = re.compile(r"\${[^{}]+}")
+__expand_var_regexp__ = re.compile(r"\${[^{}@]+}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
 
 def infer_caller_details(loginfo, parent = False, varval = True):
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 05/16] data_smart: use the expand_cache in VariableParse
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (2 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 04/16] data_smart: Improve variable expansion regexp Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 06/16] data: Use direct iteration, not keys() Richard Purdie
                   ` (10 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

When in VariableParse, use the expand_cache if possible rather than looking
up data. Ultimately it would come from the same place but this short cuts
a heavily used code block for speed improvements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 9be5d5e..a6a4b6c 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -94,6 +94,11 @@ class VariableParse:
             if self.varname and key:
                 if self.varname == key:
                     raise Exception("variable %s references itself!" % self.varname)
+            if key in self.d.expand_cache:
+                varparse = self.d.expand_cache[key]
+                self.references |= varparse.references
+                self.execs |= varparse.execs
+                return varparse.value
             var = self.d.getVar(key, True)
             if var is not None:
                 self.references.add(key)
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 06/16] data: Use direct iteration, not keys()
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (3 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 05/16] data_smart: use the expand_cache in VariableParse Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 07/16] data: Cache an list of export variables Richard Purdie
                   ` (9 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Profiling shows the creation of keys() has overhead and we're better using
an iterator rather than the memory associated with the huge list of keys
when iterating the whoe datastore. We minimise the number of times
we do this to twice only per recipe.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index e6d5232..beaf089 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -148,7 +148,7 @@ def expandKeys(alterdata, readdata = None):
         readdata = alterdata
 
     todolist = {}
-    for key in keys(alterdata):
+    for key in alterdata:
         if not '${' in key:
             continue
 
@@ -341,7 +341,7 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
 
 def generate_dependencies(d):
 
-    keys = set(key for key in d.keys() if not key.startswith("__"))
+    keys = set(key for key in d if not key.startswith("__"))
     shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
     vardepvals = set(key for key in keys if d.getVarFlag(key, "vardepvalue"))
 
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 07/16] data: Cache an list of export variables
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (4 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 06/16] data: Use direct iteration, not keys() Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 08/16] data_smart: Add explict None checks Richard Purdie
                   ` (8 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Compute a cache of the list of potential export variables so
that we don't have to compute the list from scratch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data.py       | 2 +-
 lib/bb/data_smart.py | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index beaf089..ecac66c 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -342,7 +342,7 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
 def generate_dependencies(d):
 
     keys = set(key for key in d if not key.startswith("__"))
-    shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
+    shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
     vardepvals = set(key for key in keys if d.getVarFlag(key, "vardepvalue"))
 
     deps = {}
diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index a6a4b6c..1bb186e 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -578,6 +578,13 @@ class DataSmart(MutableMapping):
         if flag == "defaultval" and '_' in var:
             self._setvar_update_overrides(var)
 
+        if flag == "unexport" or flag == "export":
+            if not "__exportlist" in self.dict:
+                self._makeShadowCopy("__exportlist")
+            if not "_content" in self.dict["__exportlist"]:
+                self.dict["__exportlist"]["_content"] = set()
+            self.dict["__exportlist"]["_content"].add(var)
+
     def getVarFlag(self, var, flag, expand=False, noweakdefault=False):
         local_var = self._findVar(var)
         value = None
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 08/16] data_smart: Add explict None checks
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (5 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 07/16] data: Cache an list of export variables Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 09/16] data_smart: Allow expansion of flags in getVarFlags Richard Purdie
                   ` (7 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel; +Cc: Alexandru DAMIAN

From: Alexandru DAMIAN <alexandru.damian@intel.com>

Simple if xxx checks end up calling len(xxx). We're interested in the specific case
of None which means we can break out the iterator much earlier after the first
item. This adds in the specific tests for None in what is a hot path in the
data store code which gives small performance gains.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 1bb186e..970e404 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -588,7 +588,7 @@ class DataSmart(MutableMapping):
     def getVarFlag(self, var, flag, expand=False, noweakdefault=False):
         local_var = self._findVar(var)
         value = None
-        if local_var:
+        if local_var is not None:
             if flag in local_var:
                 value = copy.copy(local_var[flag])
             elif flag == "_content" and "defaultval" in local_var and not noweakdefault:
@@ -599,7 +599,7 @@ class DataSmart(MutableMapping):
             if flag == "_content":
                 cachename = var
             value = self.expand(value, cachename)
-        if value and flag == "_content" and local_var and "_removeactive" in local_var:
+        if value is not None and flag == "_content" and local_var is not None and "_removeactive" in local_var:
             filtered = filter(lambda v: v not in local_var["_removeactive"],
                               value.split(" "))
             value = " ".join(filtered)
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 09/16] data_smart: Allow expansion of flags in getVarFlags
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (6 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 08/16] data_smart: Add explict None checks Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 10/16] data_smart: Allow flags to use the expand cache Richard Purdie
                   ` (6 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Allow a list of flags to expand to be passed into getVarFlags. This
is useful within bitbake itself to optimise performance of the
dependency generation code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 970e404..bc4ad54 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -647,7 +647,7 @@ class DataSmart(MutableMapping):
             self.varhistory.record(**loginfo)
             self.dict[var][i] = flags[i]
 
-    def getVarFlags(self, var, internalflags=False):
+    def getVarFlags(self, var, expand = False, internalflags=False):
         local_var = self._findVar(var)
         flags = {}
 
@@ -656,7 +656,8 @@ class DataSmart(MutableMapping):
                 if i.startswith("_") and not internalflags:
                     continue
                 flags[i] = local_var[i]
-
+                if expand and i in expand:
+                    flags[i] = self.expand(flags[i], None)
         if len(flags) == 0:
             return None
         return flags
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 10/16] data_smart: Allow flags to use the expand cache
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (7 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 09/16] data_smart: Allow expansion of flags in getVarFlags Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 11/16] data_smart: Cache the fact a variable accesses another even if its unset Richard Purdie
                   ` (5 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index bc4ad54..054b5cb 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -598,6 +598,8 @@ class DataSmart(MutableMapping):
             cachename = None
             if flag == "_content":
                 cachename = var
+            else:
+                cachename = var + "[" + flag + "]"
             value = self.expand(value, cachename)
         if value is not None and flag == "_content" and local_var is not None and "_removeactive" in local_var:
             filtered = filter(lambda v: v not in local_var["_removeactive"],
@@ -657,7 +659,7 @@ class DataSmart(MutableMapping):
                     continue
                 flags[i] = local_var[i]
                 if expand and i in expand:
-                    flags[i] = self.expand(flags[i], None)
+                    flags[i] = self.expand(flags[i], var + "[" + i + "]")
         if len(flags) == 0:
             return None
         return flags
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 11/16] data_smart: Cache the fact a variable accesses another even if its unset
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (8 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 10/16] data_smart: Allow flags to use the expand cache Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 12/16] data: Optimise build_dependencies a little Richard Purdie
                   ` (4 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

If a variable references another but it isn't set at present, the
reference wasn't stored. It really should be marked as a reference
and the higher level dependency code can handle as appropriate.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 054b5cb..79bf331 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -100,8 +100,8 @@ class VariableParse:
                 self.execs |= varparse.execs
                 return varparse.value
             var = self.d.getVar(key, True)
+            self.references.add(key)
             if var is not None:
-                self.references.add(key)
                 return var
             else:
                 return match.group()
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 12/16] data: Optimise build_dependencies a little
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (9 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 11/16] data_smart: Cache the fact a variable accesses another even if its unset Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 13/16] data: Optimise flag lookup in build_dependencies Richard Purdie
                   ` (3 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Instead of multiple calls to getVarFlag, make one call to getVarFlags, only expanding
the flags that need to be expanded. This improves performance.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data.py | 21 ++++++++++-----------
 1 file changed, 10 insertions(+), 11 deletions(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index ecac66c..876f6ca 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -287,18 +287,19 @@ def update_data(d):
 
 def build_dependencies(key, keys, shelldeps, vardepvals, d):
     deps = set()
-    vardeps = d.getVarFlag(key, "vardeps", True)
     try:
         if key[-1] == ']':
             vf = key[:-1].split('[')
             value = d.getVarFlag(vf[0], vf[1], False)
         else:
             value = d.getVar(key, False)
+        varflags = d.getVarFlags(key, ["vardeps", "vardepvalue", "vardepsexclude"]) or {}
+        vardeps = varflags.get("vardeps")
 
-        if key in vardepvals:
-           value =  d.getVarFlag(key, "vardepvalue", True)
-        elif d.getVarFlag(key, "func"):
-            if d.getVarFlag(key, "python"):
+        if "vardepvalue" in varflags:
+           value = varflags.get("vardepvalue")
+        elif varflags.get("func"):
+            if varflags.get("python"):
                 parsedvar = d.expandWithRefs(value, key)
                 parser = bb.codeparser.PythonParser(key, logger)
                 if parsedvar.value and "\t" in parsedvar.value:
@@ -323,16 +324,14 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
         varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
         if varflagsexcl:
             varfdeps = []
-            varflags = d.getVarFlags(key)
-            if varflags:
-                for f in varflags:
-                    if f not in varflagsexcl:
-                        varfdeps.append('%s[%s]' % (key, f))
+            for f in varflags:
+                if f not in varflagsexcl:
+                    varfdeps.append('%s[%s]' % (key, f))
             if varfdeps:
                 deps |= set(varfdeps)
 
         deps |= set((vardeps or "").split())
-        deps -= set((d.getVarFlag(key, "vardepsexclude", True) or "").split())
+        deps -= set(varflags.get("vardepsexclude", "").split())
     except Exception as e:
         raise bb.data_smart.ExpansionError(key, None, e)
     return deps, value
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 13/16] data: Optimise flag lookup in build_dependencies
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (10 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 12/16] data: Optimise build_dependencies a little Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 14/16] data: Optimise flag exclusion list handling Richard Purdie
                   ` (2 subsequent siblings)
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

When looking up flag variable dependencies, large chunks of the function
aren't needed. Optimise the function flow accordingly for speed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index 876f6ca..49bb5e9 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -291,10 +291,13 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
         if key[-1] == ']':
             vf = key[:-1].split('[')
             value = d.getVarFlag(vf[0], vf[1], False)
-        else:
-            value = d.getVar(key, False)
+            parser = d.expandWithRefs(value, key)
+            deps |= parser.references
+            deps = deps | (keys & parser.execs)
+            return deps, value
         varflags = d.getVarFlags(key, ["vardeps", "vardepvalue", "vardepsexclude"]) or {}
         vardeps = varflags.get("vardeps")
+        value = d.getVar(key, False)
 
         if "vardepvalue" in varflags:
            value = varflags.get("vardepvalue")
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 14/16] data: Optimise flag exclusion list handling
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (11 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 13/16] data: Optimise flag lookup in build_dependencies Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 15/16] bitbake: bb.fatal: Raise a BBHandledException instead of exiting Richard Purdie
  2013-09-16 21:53 ` [PATCH 16/16] bitbake: bb: Drop deprecated functions Richard Purdie
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

Move the variable lookup to the outer loop for performance, replacing
a now unneeded parameter (after the previous changes).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data.py | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index 49bb5e9..349fcfe 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -285,7 +285,7 @@ def update_data(d):
     """Performs final steps upon the datastore, including application of overrides"""
     d.finalize(parent = True)
 
-def build_dependencies(key, keys, shelldeps, vardepvals, d):
+def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
     deps = set()
     try:
         if key[-1] == ']':
@@ -324,7 +324,6 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
             deps = deps | (keys & parser.execs)
 
         # Add varflags, assuming an exclusion list is set
-        varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
         if varflagsexcl:
             varfdeps = []
             for f in varflags:
@@ -345,14 +344,14 @@ def generate_dependencies(d):
 
     keys = set(key for key in d if not key.startswith("__"))
     shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
-    vardepvals = set(key for key in keys if d.getVarFlag(key, "vardepvalue"))
+    varflagsexcl = d.getVar('BB_SIGNATURE_EXCLUDE_FLAGS', True)
 
     deps = {}
     values = {}
 
     tasklist = d.getVar('__BBTASKS') or []
     for task in tasklist:
-        deps[task], values[task] = build_dependencies(task, keys, shelldeps, vardepvals, d)
+        deps[task], values[task] = build_dependencies(task, keys, shelldeps, varflagsexcl, d)
         newdeps = deps[task]
         seen = set()
         while newdeps:
@@ -361,7 +360,7 @@ def generate_dependencies(d):
             newdeps = set()
             for dep in nextdeps:
                 if dep not in deps:
-                    deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, vardepvals, d)
+                    deps[dep], values[dep] = build_dependencies(dep, keys, shelldeps, varflagsexcl, d)
                 newdeps |=  deps[dep]
             newdeps -= seen
         #print "For %s: %s" % (task, str(deps[task]))
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 15/16] bitbake: bb.fatal: Raise a BBHandledException instead of exiting
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (12 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 14/16] data: Optimise flag exclusion list handling Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-16 21:53 ` [PATCH 16/16] bitbake: bb: Drop deprecated functions Richard Purdie
  14 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

With new bitbake UIs having the cooker exit at 'random' points
in the codebase is problematic. This patch raises an exception
which matches the siutation instead.

(Bitbake rev: 4effcdf21f42bf0c82891d998934065ca9f0097e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/__init__.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py
index 64491ff..018b744 100644
--- a/lib/bb/__init__.py
+++ b/lib/bb/__init__.py
@@ -99,8 +99,7 @@ def error(*args):
 
 def fatal(*args):
     logger.critical(''.join(args))
-    sys.exit(1)
-
+    raise BBHandledException()
 
 def deprecated(func, name=None, advice=""):
     """This is a decorator which can be used to mark functions
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* [PATCH 16/16] bitbake: bb: Drop deprecated functions
  2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
                   ` (13 preceding siblings ...)
  2013-09-16 21:53 ` [PATCH 15/16] bitbake: bb.fatal: Raise a BBHandledException instead of exiting Richard Purdie
@ 2013-09-16 21:53 ` Richard Purdie
  2013-09-17  9:42   ` Richard Purdie
  14 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2013-09-16 21:53 UTC (permalink / raw)
  To: bitbake-devel

These functions in the main bb module have long been deprecated
and moved to other modules. Finally remove the compatibility links.

(Bitbake rev: ccd181c3ed4852e2b9169cf19aaf18aeacddcc18)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/__init__.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py
index 018b744..8a23a0f 100644
--- a/lib/bb/__init__.py
+++ b/lib/bb/__init__.py
@@ -140,6 +140,3 @@ def deprecate_import(current, modulename, fromlist, renames = None):
 
         setattr(sys.modules[current], newname, newobj)
 
-deprecate_import(__name__, "bb.fetch", ("MalformedUrl", "encodeurl", "decodeurl"))
-deprecate_import(__name__, "bb.utils", ("mkdirhier", "movefile", "copyfile", "which"))
-deprecate_import(__name__, "bb.utils", ["vercmp_string"], ["vercmp"])
-- 
1.8.1.2



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* Re: [PATCH 16/16] bitbake: bb: Drop deprecated functions
  2013-09-16 21:53 ` [PATCH 16/16] bitbake: bb: Drop deprecated functions Richard Purdie
@ 2013-09-17  9:42   ` Richard Purdie
  0 siblings, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2013-09-17  9:42 UTC (permalink / raw)
  To: bitbake-devel

On Mon, 2013-09-16 at 22:53 +0100, Richard Purdie wrote:
> These functions in the main bb module have long been deprecated
> and moved to other modules. Finally remove the compatibility links.
> 
> (Bitbake rev: ccd181c3ed4852e2b9169cf19aaf18aeacddcc18)
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/__init__.py | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py
> index 018b744..8a23a0f 100644
> --- a/lib/bb/__init__.py
> +++ b/lib/bb/__init__.py
> @@ -140,6 +140,3 @@ def deprecate_import(current, modulename, fromlist, renames = None):
>  
>          setattr(sys.modules[current], newname, newobj)
>  
> -deprecate_import(__name__, "bb.fetch", ("MalformedUrl", "encodeurl", "decodeurl"))
> -deprecate_import(__name__, "bb.utils", ("mkdirhier", "movefile", "copyfile", "which"))
> -deprecate_import(__name__, "bb.utils", ["vercmp_string"], ["vercmp"])

I should be clear with this one that now isn't the ideal time to merge
this and that isn't my intention. 

It is a reminder to people that this API *will* go away soon in the 1.6
cycle though.

Cheers,

Richard




^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2013-09-17  9:42 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-09-16 21:53 [PATCH 01/16] cooker: Allow profiling of the parser in profile mode Richard Purdie
2013-09-16 21:53 ` [PATCH 02/16] data: Be explicit in data_db check Richard Purdie
2013-09-16 21:53 ` [PATCH 03/16] siggen: Use lookup cache exclusively Richard Purdie
2013-09-16 21:53 ` [PATCH 04/16] data_smart: Improve variable expansion regexp Richard Purdie
2013-09-16 21:53 ` [PATCH 05/16] data_smart: use the expand_cache in VariableParse Richard Purdie
2013-09-16 21:53 ` [PATCH 06/16] data: Use direct iteration, not keys() Richard Purdie
2013-09-16 21:53 ` [PATCH 07/16] data: Cache an list of export variables Richard Purdie
2013-09-16 21:53 ` [PATCH 08/16] data_smart: Add explict None checks Richard Purdie
2013-09-16 21:53 ` [PATCH 09/16] data_smart: Allow expansion of flags in getVarFlags Richard Purdie
2013-09-16 21:53 ` [PATCH 10/16] data_smart: Allow flags to use the expand cache Richard Purdie
2013-09-16 21:53 ` [PATCH 11/16] data_smart: Cache the fact a variable accesses another even if its unset Richard Purdie
2013-09-16 21:53 ` [PATCH 12/16] data: Optimise build_dependencies a little Richard Purdie
2013-09-16 21:53 ` [PATCH 13/16] data: Optimise flag lookup in build_dependencies Richard Purdie
2013-09-16 21:53 ` [PATCH 14/16] data: Optimise flag exclusion list handling Richard Purdie
2013-09-16 21:53 ` [PATCH 15/16] bitbake: bb.fatal: Raise a BBHandledException instead of exiting Richard Purdie
2013-09-16 21:53 ` [PATCH 16/16] bitbake: bb: Drop deprecated functions Richard Purdie
2013-09-17  9:42   ` Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.