[PATCH] data_smart/utils: Add 'd' to the context used for better_eval in python expansion

[PATCH] data_smart/utils: Add 'd' to the context used for better_eval in python expansion

[Richard Purdie]

If a line like:

foo=${@' '.join([d.getVar('D', True) + x for x in (' '.join([d.getVar('FILES_bash-' + p, True) or '' for p in ['lib', 'dev', 'staticdev', 'doc', 'locale', 'ptest']])).split()])}

is added to a function like do_install, it fails with Exception name 'd'
is not defined. This is due to a change of behaviour in python 3 compared
to python 2. Generator expressions, dict comprehensions  and set comprehensions
are executed in a new scope but list comprehensions in python 2.x are not. In
python 3 they all use a new scope.

To allow these kinds of expressions to work, the easiest approach is
to add 'd' to the global context. To do this, an extra optional parameter
is added to better_eval and we use that to add 'd'.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/data_smart.py b/bitbake/lib/bb/data_smart.py
index 25c412c..f100446 100644
--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -135,7 +135,7 @@ class VariableParse:
                     self.contains[k] = parser.contains[k].copy()
                 else:
                     self.contains[k].update(parser.contains[k])
-            value = utils.better_eval(codeobj, DataContext(self.d))
+            value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d})
             return str(value)
 
 
diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 8f75871..0a1bf68 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -37,6 +37,7 @@ import errno
 import signal
 import ast
 import collections
+import copy
 from subprocess import getstatusoutput
 from contextlib import contextmanager
 from ctypes import cdll
@@ -407,8 +408,13 @@ def better_exec(code, context, text = None, realfile = "<code>", pythonexception
 def simple_exec(code, context):
     exec(code, get_context(), context)
 
-def better_eval(source, locals):
-    return eval(source, get_context(), locals)
+def better_eval(source, locals, extraglobals = None):
+    ctx = get_context()
+    if extraglobals:
+        ctx = copy.copy(ctx)
+        for g in extraglobals:
+            ctx[g] = extraglobals[g]
+    return eval(source, ctx, locals)
 
 @contextmanager
 def fileslocked(files):

Re: [PATCH] data_smart/utils: Add 'd' to the context used for better_eval in python expansion

[Patrick Ohly]

On Thu, 2016-06-09 at 23:38 +0100, Richard Purdie wrote:
> If a line like:
> 
> foo=${@' '.join([d.getVar('D', True) + x for x in (' '.join([d.getVar('FILES_bash-' + p, True) or '' for p in ['lib', 'dev', 'staticdev', 'doc', 'locale', 'ptest']])).split()])}
> 
> is added to a function like do_install, it fails with Exception name 'd'
> is not defined. This is due to a change of behaviour in python 3 compared
> to python 2. Generator expressions, dict comprehensions  and set comprehensions
> are executed in a new scope but list comprehensions in python 2.x are not. In
> python 3 they all use a new scope.
> 
> To allow these kinds of expressions to work, the easiest approach is
> to add 'd' to the global context. To do this, an extra optional parameter
> is added to better_eval and we use that to add 'd'.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

I'm building Ostro OS with this patch, which is where the scoping
problem was first found. Seems to work for me, so please include it.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.