[PATCH 0/4] Developer workflow related enhancements / fixes

[PATCH 0/4] Developer workflow related enhancements / fixes

[Paul Eggleton]

The following series is the set of BitBake changes that are part of the
developer workflow improvements currently being worked on [1].

Of particular note is the final patch for resetting the class handlers
object - this was discovered incidentally and fixes a rather nasty bug.
It should probably be considered separately from the rest of the series.

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=6662

The following changes since commit 94d9590a4310f96396e8e782bcf65918f4dcdb36:

  bitbake: Update to 1.25.0 as development version after release (2014-12-03 12:14:43 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/devtool-bb
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/devtool-bb

Paul Eggleton (4):
  utils: add exec_flat_python_func()
  fetch2: add means of disabling SRC_URI checksums
  event: add a means of filtering events internally
  event: fix resetting class handlers object

 lib/bb/event.py           |  9 +++++++++
 lib/bb/fetch2/__init__.py |  3 ++-
 lib/bb/utils.py           | 22 ++++++++++++++++++++++
 3 files changed, 33 insertions(+), 1 deletion(-)

-- 
1.9.3

[PATCH 1/4] utils: add exec_flat_python_func()

[Paul Eggleton]

Add a function that allows executing a flat python function (defined
with def funcname(args): ...).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/utils.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index c6f4071..d7c5067 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -893,3 +893,25 @@ def multiprocessingpool(*args, **kwargs):
 
     return multiprocessing.Pool(*args, **kwargs)
 
+def exec_flat_python_func(func, *args, **kwargs):
+    """Execute a flat python function (defined with def funcname(args):...)"""
+    # Prepare a small piece of python code which calls the requested function
+    # To do this we need to prepare two things - a set of variables we can use to pass
+    # the values of arguments into the calling function, and the list of arguments for
+    # the function being called
+    context = {}
+    funcargs = []
+    # Handle unnamed arguments
+    aidx = 1
+    for arg in args:
+        argname = 'arg_%s' % aidx
+        context[argname] = arg
+        funcargs.append(argname)
+        aidx += 1
+    # Handle keyword arguments
+    context.update(kwargs)
+    funcargs.extend(['%s=%s' % (arg, arg) for arg in kwargs.iterkeys()])
+    code = 'retval = %s(%s)' % (func, ', '.join(funcargs))
+    comp = bb.utils.better_compile(code, '<string>', '<string>')
+    bb.utils.better_exec(comp, context, code, '<string>')
+    return context['retval']
-- 
1.9.3

[PATCH 2/4] fetch2: add means of disabling SRC_URI checksums

[Paul Eggleton]

If we're fetching outside of the context of a recipe, it's handy to be
able to disable checksum functionality so you don't get a meaningless
warning about the signatures being missing.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/fetch2/__init__.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 3696e24..5b26524 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -535,7 +535,7 @@ def verify_checksum(ud, d):
 
     """
 
-    if not ud.method.supports_checksum(ud):
+    if ud.ignore_checksums or not ud.method.supports_checksum(ud):
         return
 
     md5data = bb.utils.md5_file(ud.localpath)
@@ -1041,6 +1041,7 @@ class FetchData(object):
             self.sha256_expected = None
         else:
             self.sha256_expected = d.getVarFlag("SRC_URI", self.sha256_name)
+        self.ignore_checksums = False
 
         self.names = self.parm.get("name",'default').split(',')
 
-- 
1.9.3

[PATCH 3/4] event: add a means of filtering events internally

[Paul Eggleton]

When using external tinfoil-based utilities, it is useful to be able to
turn off most of the event handlers; for example sstate_eventhandler
doesn't like being sent events for any recipe which has been skipped.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/event.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/bb/event.py b/lib/bb/event.py
index 32df779..9645476 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -67,6 +67,7 @@ _ui_logfilters = {}
 _ui_handler_seq = 0
 _event_handler_map = {}
 _catchall_handlers = {}
+_eventfilter = None
 
 def execute_handler(name, handler, event, d):
     event.data = d
@@ -94,6 +95,9 @@ def fire_class_handlers(event, d):
     evt_hmap = _event_handler_map.get(eid, {})
     for name, handler in _handlers.iteritems():
         if name in _catchall_handlers or name in evt_hmap:
+            if _eventfilter:
+                if not _eventfilter(name, handler, event, d):
+                    continue
             execute_handler(name, handler, event, d)
 
 ui_queue = []
@@ -204,6 +208,10 @@ def remove(name, handler):
     """Remove an Event handler"""
     _handlers.pop(name)
 
+def set_eventfilter(func):
+    global _eventfilter
+    _eventfilter = func
+
 def register_UIHhandler(handler):
     bb.event._ui_handler_seq = bb.event._ui_handler_seq + 1
     _ui_handlers[_ui_handler_seq] = handler
-- 
1.9.3

[PATCH 4/4] event: fix resetting class handlers object

[Paul Eggleton]

If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/event.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/lib/bb/event.py b/lib/bb/event.py
index 9645476..8bc6b80 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -55,6 +55,7 @@ def get_class_handlers():
     return _handlers
 
 def set_class_handlers(h):
+    global handlers
     _handlers = h
 
 def clean_class_handlers():
-- 
1.9.3

Re: [PATCH 4/4] event: fix resetting class handlers object

[Richard Purdie]

On Mon, 2014-12-08 at 10:50 +0000, Paul Eggleton wrote:
> If you don't explicitly specify to use a global variable when doing an
> assignment, you will be setting a local variable instead, which means
> this function wasn't working at all. It explains some odd behaviour we
> have seen in the layer index where event handlers were sometimes
> bleeding into other contexts where they should not have been.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  lib/bb/event.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/lib/bb/event.py b/lib/bb/event.py
> index 9645476..8bc6b80 100644
> --- a/lib/bb/event.py
> +++ b/lib/bb/event.py
> @@ -55,6 +55,7 @@ def get_class_handlers():
>      return _handlers
>  
>  def set_class_handlers(h):
> +    global handlers
>      _handlers = h

Er, shouldn't this be _handlers?

Cheers,

Richard

>  def clean_class_handlers():
> -- 
> 1.9.3
> 

Re: [PATCH 4/4] event: fix resetting class handlers object

[Paul Eggleton]

On Monday 08 December 2014 17:42:19 Richard Purdie wrote:
> On Mon, 2014-12-08 at 10:50 +0000, Paul Eggleton wrote:
> > If you don't explicitly specify to use a global variable when doing an
> > assignment, you will be setting a local variable instead, which means
> > this function wasn't working at all. It explains some odd behaviour we
> > have seen in the layer index where event handlers were sometimes
> > bleeding into other contexts where they should not have been.
> > 
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> > ---
> > 
> >  lib/bb/event.py | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/lib/bb/event.py b/lib/bb/event.py
> > index 9645476..8bc6b80 100644
> > --- a/lib/bb/event.py
> > +++ b/lib/bb/event.py
> > 
> > @@ -55,6 +55,7 @@ def get_class_handlers():
> >      return _handlers
> >  
> >  def set_class_handlers(h):
> > +    global handlers
> > 
> >      _handlers = h
> 
> Er, shouldn't this be _handlers?

Oh dear. Yes, it should. I've fixed that on the branch, but I'll do a bit more 
testing just to make sure it isn't breaking anything since the previous change 
wouldn't have made any impact at all.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 4/4] event: fix resetting class handlers object

[Richard Purdie]

On Mon, 2014-12-08 at 17:42 +0000, Richard Purdie wrote:
> On Mon, 2014-12-08 at 10:50 +0000, Paul Eggleton wrote:
> > If you don't explicitly specify to use a global variable when doing an
> > assignment, you will be setting a local variable instead, which means
> > this function wasn't working at all. It explains some odd behaviour we
> > have seen in the layer index where event handlers were sometimes
> > bleeding into other contexts where they should not have been.
> > 
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> > ---
> >  lib/bb/event.py | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/lib/bb/event.py b/lib/bb/event.py
> > index 9645476..8bc6b80 100644
> > --- a/lib/bb/event.py
> > +++ b/lib/bb/event.py
> > @@ -55,6 +55,7 @@ def get_class_handlers():
> >      return _handlers
> >  
> >  def set_class_handlers(h):
> > +    global handlers
> >      _handlers = h
> 
> Er, shouldn't this be _handlers?

I put the patch in master-next and tweaked this.

Cheers,

Richard