[PATCH 0/5] Variable typing & reworked imports handling

[PATCH 0/5] Variable typing & reworked imports handling

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Pull URL: git://git.openembedded.org/openembedded-core-contrib
  Branch: kergoth/oe-sync-base
  Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/oe-sync-base

Thanks,
    Chris Larson <chris_larson@mentor.com>
---


Chris Larson (5):
  Implement variable typing (sync from OE)
  oe.data: expand the flags
  base.bbclass: switch to current OE's imports handling
  Shift oe import logic out of the event handler
  base.bbclass: use oe.data for OE_IMPORTS

 meta/classes/base.bbclass      |   46 +++++++++--------
 meta/classes/typecheck.bbclass |   12 +++++
 meta/lib/oe/data.py            |   18 +++++++
 meta/lib/oe/maketype.py        |  100 ++++++++++++++++++++++++++++++++++++++
 meta/lib/oe/test_types.py      |   62 ++++++++++++++++++++++++
 meta/lib/oe/types.py           |  104 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 320 insertions(+), 22 deletions(-)
 create mode 100644 meta/classes/typecheck.bbclass
 create mode 100644 meta/lib/oe/data.py
 create mode 100644 meta/lib/oe/maketype.py
 create mode 100644 meta/lib/oe/test_types.py
 create mode 100644 meta/lib/oe/types.py

[PATCH 1/5] Implement variable typing (sync from OE)

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

This implementation consists of two components:

- Type creation python modules, whose job it is to construct objects of the
  defined type for a given variable in the metadata
- typecheck.bbclass, which iterates over all configuration variables with a
  type defined and uses oe.types to check the validity of the values

This gives us a few benefits:

- Automatic sanity checking of all configuration variables with a defined type
- Avoid duplicating the "how do I make use of the value of this variable"
  logic between its users.  For variables like PATH, this is simply a split(),
  for boolean variables, the duplication can result in confusing, or even
  mismatched semantics (is this 0/1, empty/nonempty, what?)
- Make it easier to create a configuration UI, as the type information could
  be used to provide a better interface than a text edit box (e.g checkbox for
  'boolean', dropdown for 'choice')

This functionality is entirely opt-in right now.  To enable the configuration
variable type checking, simply INHERIT += "typecheck".  Example of a failing
type check:

BAZ = "foo"
BAZ[type] = "boolean"

$ bitbake -p
FATAL: BAZ: Invalid boolean value 'foo'
$

Examples of leveraging oe.types in a python snippet:

PACKAGES[type] = "list"

python () {
    import oe.data
    for pkg in oe.data.typed_value("PACKAGES", d):
        bb.note("package: %s" % pkg)
}

LIBTOOL_HAS_SYSROOT = "yes"
LIBTOOL_HAS_SYSROOT[type] = "boolean"

python () {
    import oe.data
    assert(oe.data.typed_value("LIBTOOL_HAS_SYSROOT", d) == True)
}

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass      |    1 +
 meta/classes/typecheck.bbclass |   12 +++++
 meta/lib/oe/data.py            |   13 +++++
 meta/lib/oe/maketype.py        |  100 ++++++++++++++++++++++++++++++++++++++
 meta/lib/oe/test_types.py      |   62 ++++++++++++++++++++++++
 meta/lib/oe/types.py           |  104 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 292 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/typecheck.bbclass
 create mode 100644 meta/lib/oe/data.py
 create mode 100644 meta/lib/oe/maketype.py
 create mode 100644 meta/lib/oe/test_types.py
 create mode 100644 meta/lib/oe/types.py

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 23095ec..b552de2 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -28,6 +28,7 @@ python sys_path_eh () {
 
         import oe.path
         import oe.utils
+        import oe.data
         inject("bb", bb)
         inject("sys", sys)
         inject("time", time)
diff --git a/meta/classes/typecheck.bbclass b/meta/classes/typecheck.bbclass
new file mode 100644
index 0000000..646cd4e
--- /dev/null
+++ b/meta/classes/typecheck.bbclass
@@ -0,0 +1,12 @@
+# Check types of bitbake configuration variables
+#
+# See oe.types for details.
+
+python check_types() {
+    import oe.types
+    if isinstance(e, bb.event.ConfigParsed):
+        for key in e.data.keys():
+            if e.data.getVarFlag(key, "type"):
+                oe.types.value(key, e.data)
+}
+addhandler check_types
diff --git a/meta/lib/oe/data.py b/meta/lib/oe/data.py
new file mode 100644
index 0000000..8b7c3cd
--- /dev/null
+++ b/meta/lib/oe/data.py
@@ -0,0 +1,13 @@
+import oe.maketype
+import bb.msg
+
+def typed_value(key, d):
+    """Construct a value for the specified metadata variable, using its flags
+    to determine the type and parameters for construction."""
+    var_type = d.getVarFlag(key, 'type')
+    flags = d.getVarFlags(key)
+
+    try:
+        return oe.maketype.create(d.getVar(key, True) or '', var_type, **flags)
+    except (TypeError, ValueError), exc:
+        bb.msg.fatal(bb.msg.domain.Data, "%s: %s" % (key, str(exc)))
diff --git a/meta/lib/oe/maketype.py b/meta/lib/oe/maketype.py
new file mode 100644
index 0000000..0e9dbc6
--- /dev/null
+++ b/meta/lib/oe/maketype.py
@@ -0,0 +1,100 @@
+"""OpenEmbedded variable typing support
+
+Types are defined in the metadata by name, using the 'type' flag on a
+variable.  Other flags may be utilized in the construction of the types.  See
+the arguments of the type's factory for details.
+"""
+
+import bb
+import inspect
+import types
+
+available_types = {}
+
+class MissingFlag(TypeError):
+    """A particular flag is required to construct the type, but has not been
+    provided."""
+    def __init__(self, flag, type):
+        self.flag = flag
+        self.type = type
+        TypeError.__init__(self)
+
+    def __str__(self):
+        return "Type '%s' requires flag '%s'" % (self.type, self.flag)
+
+def factory(var_type):
+    """Return the factory for a specified type."""
+    if var_type is None:
+        raise TypeError("No type specified. Valid types: %s" %
+                        ', '.join(available_types))
+    try:
+        return available_types[var_type]
+    except KeyError:
+        raise TypeError("Invalid type '%s':\n  Valid types: %s" %
+                        (var_type, ', '.join(available_types)))
+
+def create(value, var_type, **flags):
+    """Create an object of the specified type, given the specified flags and
+    string value."""
+    obj = factory(var_type)
+    objflags = {}
+    for flag in obj.flags:
+        if flag not in flags:
+            if flag not in obj.optflags:
+                raise MissingFlag(flag, var_type)
+        else:
+            objflags[flag] = flags[flag]
+
+    return obj(value, **objflags)
+
+def get_callable_args(obj):
+    """Grab all but the first argument of the specified callable, returning
+    the list, as well as a list of which of the arguments have default
+    values."""
+    if type(obj) is type:
+        obj = obj.__init__
+
+    args, varargs, keywords, defaults = inspect.getargspec(obj)
+    flaglist = []
+    if args:
+        if len(args) > 1 and args[0] == 'self':
+            args = args[1:]
+        flaglist.extend(args)
+
+    optional = set()
+    if defaults:
+        optional |= set(flaglist[-len(defaults):])
+    return flaglist, optional
+
+def factory_setup(name, obj):
+    """Prepare a factory for use."""
+    args, optional = get_callable_args(obj)
+    extra_args = args[1:]
+    if extra_args:
+        obj.flags, optional = extra_args, optional
+        obj.optflags = set(optional)
+    else:
+        obj.flags = obj.optflags = ()
+
+    if not hasattr(obj, 'name'):
+        obj.name = name
+
+def register(name, factory):
+    """Register a type, given its name and a factory callable.
+
+    Determines the required and optional flags from the factory's
+    arguments."""
+    factory_setup(name, factory)
+    available_types[factory.name] = factory
+
+
+# Register all our included types
+for name in dir(types):
+    if name.startswith('_'):
+        continue
+
+    obj = getattr(types, name)
+    if not callable(obj):
+        continue
+
+    register(name, obj)
diff --git a/meta/lib/oe/test_types.py b/meta/lib/oe/test_types.py
new file mode 100644
index 0000000..367cc30
--- /dev/null
+++ b/meta/lib/oe/test_types.py
@@ -0,0 +1,62 @@
+import unittest
+from oe.maketype import create, factory
+
+class TestTypes(unittest.TestCase):
+    def assertIsInstance(self, obj, cls):
+        return self.assertTrue(isinstance(obj, cls))
+
+    def assertIsNot(self, obj, other):
+        return self.assertFalse(obj is other)
+
+    def assertFactoryCreated(self, value, type, **flags):
+        cls = factory(type)
+        self.assertIsNot(cls, None)
+        self.assertIsInstance(create(value, type, **flags), cls)
+
+class TestBooleanType(TestTypes):
+    def test_invalid(self):
+        self.assertRaises(ValueError, create, '', 'boolean')
+        self.assertRaises(ValueError, create, 'foo', 'boolean')
+        self.assertRaises(TypeError, create, object(), 'boolean')
+
+    def test_true(self):
+        self.assertTrue(create('y', 'boolean'))
+        self.assertTrue(create('yes', 'boolean'))
+        self.assertTrue(create('1', 'boolean'))
+        self.assertTrue(create('t', 'boolean'))
+        self.assertTrue(create('true', 'boolean'))
+        self.assertTrue(create('TRUE', 'boolean'))
+        self.assertTrue(create('truE', 'boolean'))
+
+    def test_false(self):
+        self.assertFalse(create('n', 'boolean'))
+        self.assertFalse(create('no', 'boolean'))
+        self.assertFalse(create('0', 'boolean'))
+        self.assertFalse(create('f', 'boolean'))
+        self.assertFalse(create('false', 'boolean'))
+        self.assertFalse(create('FALSE', 'boolean'))
+        self.assertFalse(create('faLse', 'boolean'))
+
+    def test_bool_equality(self):
+        self.assertEqual(create('n', 'boolean'), False)
+        self.assertNotEqual(create('n', 'boolean'), True)
+        self.assertEqual(create('y', 'boolean'), True)
+        self.assertNotEqual(create('y', 'boolean'), False)
+
+class TestList(TestTypes):
+    def assertListEqual(self, value, valid, sep=None):
+        obj = create(value, 'list', separator=sep)
+        self.assertEqual(obj, valid)
+        if sep is not None:
+            self.assertEqual(obj.separator, sep)
+        self.assertEqual(str(obj), obj.separator.join(obj))
+
+    def test_list_nosep(self):
+        testlist = ['alpha', 'beta', 'theta']
+        self.assertListEqual('alpha beta theta', testlist)
+        self.assertListEqual('alpha  beta\ttheta', testlist)
+        self.assertListEqual('alpha', ['alpha'])
+
+    def test_list_usersep(self):
+        self.assertListEqual('foo:bar', ['foo', 'bar'], ':')
+        self.assertListEqual('foo:bar:baz', ['foo', 'bar', 'baz'], ':')
diff --git a/meta/lib/oe/types.py b/meta/lib/oe/types.py
new file mode 100644
index 0000000..ea31cf4
--- /dev/null
+++ b/meta/lib/oe/types.py
@@ -0,0 +1,104 @@
+import re
+
+class OEList(list):
+    """OpenEmbedded 'list' type
+
+    Acts as an ordinary list, but is constructed from a string value and a
+    separator (optional), and re-joins itself when converted to a string with
+    str().  Set the variable type flag to 'list' to use this type, and the
+    'separator' flag may be specified (defaulting to whitespace)."""
+
+    name = "list"
+
+    def __init__(self, value, separator = None):
+        if value is not None:
+            list.__init__(self, value.split(separator))
+        else:
+            list.__init__(self)
+
+        if separator is None:
+            self.separator = " "
+        else:
+            self.separator = separator
+
+    def __str__(self):
+        return self.separator.join(self)
+
+def choice(value, choices):
+    """OpenEmbedded 'choice' type
+
+    Acts as a multiple choice for the user.  To use this, set the variable
+    type flag to 'choice', and set the 'choices' flag to a space separated
+    list of valid values."""
+    if not isinstance(value, basestring):
+        raise TypeError("choice accepts a string, not '%s'" % type(value))
+
+    value = value.lower()
+    choices = choices.lower()
+    if value not in choices.split():
+        raise ValueError("Invalid choice '%s'.  Valid choices: %s" %
+                         (value, choices))
+    return value
+
+def regex(value, regexflags=None):
+    """OpenEmbedded 'regex' type
+
+    Acts as a regular expression, returning the pre-compiled regular
+    expression pattern object.  To use this type, set the variable type flag
+    to 'regex', and optionally, set the 'regexflags' type to a space separated
+    list of the flags to control the regular expression matching (e.g.
+    FOO[regexflags] += 'ignorecase').  See the python documentation on the
+    're' module for a list of valid flags."""
+
+    flagval = 0
+    if regexflags:
+        for flag in regexflags.split():
+            flag = flag.upper()
+            try:
+                flagval |= getattr(re, flag)
+            except AttributeError:
+                raise ValueError("Invalid regex flag '%s'" % flag)
+
+    try:
+        return re.compile(value, flagval)
+    except re.error, exc:
+        raise ValueError("Invalid regex value '%s': %s" %
+                         (value, exc.args[0]))
+
+def boolean(value):
+    """OpenEmbedded 'boolean' type
+
+    Valid values for true: 'yes', 'y', 'true', 't', '1'
+    Valid values for false: 'no', 'n', 'false', 'f', '0'
+    """
+
+    if not isinstance(value, basestring):
+        raise TypeError("boolean accepts a string, not '%s'" % type(value))
+
+    value = value.lower()
+    if value in ('yes', 'y', 'true', 't', '1'):
+        return True
+    elif value in ('no', 'n', 'false', 'f', '0'):
+        return False
+    raise ValueError("Invalid boolean value '%s'" % value)
+
+def integer(value, numberbase=10):
+    """OpenEmbedded 'integer' type
+
+    Defaults to base 10, but this can be specified using the optional
+    'numberbase' flag."""
+
+    return int(value, int(numberbase))
+
+_float = float
+def float(value, fromhex='false'):
+    """OpenEmbedded floating point type
+
+    To use this type, set the type flag to 'float', and optionally set the
+    'fromhex' flag to a true value (obeying the same rules as for the
+    'boolean' type) if the value is in base 16 rather than base 10."""
+
+    if boolean(fromhex):
+        return _float.fromhex(value)
+    else:
+        return _float(value)
-- 
1.7.1

[PATCH 2/5] oe.data: expand the flags

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/lib/oe/data.py |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)

diff --git a/meta/lib/oe/data.py b/meta/lib/oe/data.py
index 8b7c3cd..4b4d03e 100644
--- a/meta/lib/oe/data.py
+++ b/meta/lib/oe/data.py
@@ -6,6 +6,11 @@ def typed_value(key, d):
     to determine the type and parameters for construction."""
     var_type = d.getVarFlag(key, 'type')
     flags = d.getVarFlags(key)
+    if flags is not None:
+        flags = dict((flag, bb.data.expand(value, d))
+                     for flag, value in flags.iteritems())
+    else:
+        flags = {}
 
     try:
         return oe.maketype.create(d.getVar(key, True) or '', var_type, **flags)
-- 
1.7.1

[PATCH 3/5] base.bbclass: switch to current OE's imports handling

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

The current mechanism makes it easier for classes to add new oe modules to be
automatically imported, and thereby made available to python snippets (${@}).

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass |   22 +++++++++-------------
 1 files changed, 9 insertions(+), 13 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index b552de2..51f5a14 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -10,32 +10,28 @@ inherit metadata_scm
 inherit buildstats
 inherit logging
 
-python sys_path_eh () {
+OE_IMPORTS += "os sys time oe.path oe.utils oe.data"
+
+python oe_import () {
     if isinstance(e, bb.event.ConfigParsed):
-        import sys
-        import os
-        import time
+        import os, sys
 
         bbpath = e.data.getVar("BBPATH", True).split(":")
         sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
 
         def inject(name, value):
-            """Make a python object accessible from everywhere for the metadata"""
+            """Make a python object accessible from the metadata"""
             if hasattr(bb.utils, "_context"):
                 bb.utils._context[name] = value
             else:
                 __builtins__[name] = value
 
-        import oe.path
-        import oe.utils
-        import oe.data
-        inject("bb", bb)
-        inject("sys", sys)
-        inject("time", time)
-        inject("oe", oe)
+        for toimport in e.data.getVar("OE_IMPORTS", True).split():
+            imported = __import__(toimport)
+            inject(toimport.split(".", 1)[0], imported)
 }
 
-addhandler sys_path_eh
+addhandler oe_import
 
 die() {
 	bbfatal "$*"
-- 
1.7.1

[PATCH 4/5] Shift oe import logic out of the event handler

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

This can be useful if we need the imports from another config parsed event
handler, and can't rely upon the base one running before that one.

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass |   33 ++++++++++++++++++---------------
 1 files changed, 18 insertions(+), 15 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 51f5a14..391545c 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -12,26 +12,29 @@ inherit logging
 
 OE_IMPORTS += "os sys time oe.path oe.utils oe.data"
 
-python oe_import () {
-    if isinstance(e, bb.event.ConfigParsed):
-        import os, sys
+def oe_import(d):
+    import os, sys
+
+    bbpath = d.getVar("BBPATH", True).split(":")
+    sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
 
-        bbpath = e.data.getVar("BBPATH", True).split(":")
-        sys.path[0:0] = [os.path.join(dir, "lib") for dir in bbpath]
+    def inject(name, value):
+        """Make a python object accessible from the metadata"""
+        if hasattr(bb.utils, "_context"):
+            bb.utils._context[name] = value
+        else:
+            __builtins__[name] = value
 
-        def inject(name, value):
-            """Make a python object accessible from the metadata"""
-            if hasattr(bb.utils, "_context"):
-                bb.utils._context[name] = value
-            else:
-                __builtins__[name] = value
+    for toimport in d.getVar("OE_IMPORTS", True).split():
+        imported = __import__(toimport)
+        inject(toimport.split(".", 1)[0], imported)
 
-        for toimport in e.data.getVar("OE_IMPORTS", True).split():
-            imported = __import__(toimport)
-            inject(toimport.split(".", 1)[0], imported)
+python oe_import_eh () {
+    if isinstance(e, bb.event.ConfigParsed):
+	oe_import(e.data)
 }
 
-addhandler oe_import
+addhandler oe_import_eh
 
 die() {
 	bbfatal "$*"
-- 
1.7.1

[PATCH 5/5] base.bbclass: use oe.data for OE_IMPORTS

[Chris Larson]

From: Chris Larson <chris_larson@mentor.com>

Signed-off-by: Chris Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 391545c..7950bc3 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -11,6 +11,7 @@ inherit buildstats
 inherit logging
 
 OE_IMPORTS += "os sys time oe.path oe.utils oe.data"
+OE_IMPORTS[type] = "list"
 
 def oe_import(d):
     import os, sys
@@ -25,7 +26,8 @@ def oe_import(d):
         else:
             __builtins__[name] = value
 
-    for toimport in d.getVar("OE_IMPORTS", True).split():
+    import oe.data
+    for toimport in oe.data.typed_value("OE_IMPORTS", d):
         imported = __import__(toimport)
         inject(toimport.split(".", 1)[0], imported)
 
-- 
1.7.1

Re: [PATCH 0/5] Variable typing & reworked imports handling

[Joshua Lock]

On Wed, 2011-05-18 at 14:02 -0700, Chris Larson wrote:
> From: Chris Larson <chris_larson@mentor.com>
> 
> Pull URL: git://git.openembedded.org/openembedded-core-contrib
>   Branch: kergoth/oe-sync-base
>   Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/oe-sync-base
> 
> Thanks,
>     Chris Larson <chris_larson@mentor.com>

I like this, I was thinking about typing in BB about the time you first
showed me this so am pleased to see it submitted.

I think this will be especially useful for the oft discussed
configuration editor UI. Thanks!

Acked-by: Joshua Lock <josh@linux.intel.com>

> ---
> 
> 
> Chris Larson (5):
>   Implement variable typing (sync from OE)
>   oe.data: expand the flags
>   base.bbclass: switch to current OE's imports handling
>   Shift oe import logic out of the event handler
>   base.bbclass: use oe.data for OE_IMPORTS
> 
>  meta/classes/base.bbclass      |   46 +++++++++--------
>  meta/classes/typecheck.bbclass |   12 +++++
>  meta/lib/oe/data.py            |   18 +++++++
>  meta/lib/oe/maketype.py        |  100 ++++++++++++++++++++++++++++++++++++++
>  meta/lib/oe/test_types.py      |   62 ++++++++++++++++++++++++
>  meta/lib/oe/types.py           |  104 ++++++++++++++++++++++++++++++++++++++++
>  6 files changed, 320 insertions(+), 22 deletions(-)
>  create mode 100644 meta/classes/typecheck.bbclass
>  create mode 100644 meta/lib/oe/data.py
>  create mode 100644 meta/lib/oe/maketype.py
>  create mode 100644 meta/lib/oe/test_types.py
>  create mode 100644 meta/lib/oe/types.py
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

-- 
Joshua Lock
        Yocto Project Build Monkey
        Intel Open Source Technology Centre

Re: [PATCH 0/5] Variable typing & reworked imports handling

[Richard Purdie]

On Wed, 2011-05-18 at 14:02 -0700, Chris Larson wrote:
> From: Chris Larson <chris_larson@mentor.com>
> 
> Pull URL: git://git.openembedded.org/openembedded-core-contrib
>   Branch: kergoth/oe-sync-base
>   Browse: http://git.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=kergoth/oe-sync-base
> 
> Thanks,
>     Chris Larson <chris_larson@mentor.com>
> ---
> 
> 
> Chris Larson (5):
>   Implement variable typing (sync from OE)
>   oe.data: expand the flags
>   base.bbclass: switch to current OE's imports handling
>   Shift oe import logic out of the event handler
>   base.bbclass: use oe.data for OE_IMPORTS

Merged to master, thanks.

Richard