[PATCH 0/4] recipeutils fixes for devtool

[PATCH 0/4] recipeutils fixes for devtool

[Paul Eggleton]

A couple of fixes for issues in the recipeutils module (which affect
devtool), a cleanup patch, and some additional related tests for
oe-selftest.


The following changes since commit 21387613fec1a8c142ed48d7a74d587e205b0c98:

  gst-validate: upgrade 1.14.2 -> 1.14.4 (2018-11-27 22:11:22 +0000)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool-recipeutils-fixes
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool-recipeutils-fixes

Paul Eggleton (4):
  lib/oe/recipeutils: patch_recipe(): fix handling of values across
    includes/classes
  lib/oe/recipeutils: patch_recipe(): fix replacing varflag values
  lib/oe/recipeutils: drop obsolete functions
  oe-selftest: add some tests for recipeutils module

 meta/lib/oe/recipeutils.py                  | 104 +++++++++++----
 meta/lib/oeqa/selftest/cases/recipeutils.py | 135 ++++++++++++++++++++
 2 files changed, 211 insertions(+), 28 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py

-- 
2.17.2

[PATCH 1/4] lib/oe/recipeutils: patch_recipe(): fix handling of values across includes/classes

[Paul Eggleton]

If we were setting a variable and part of the variable's value was being
set in a class or a .inc file, we were still just setting the value
outright on the assumption that it was too hard to do otherwise. With
some careful use of the variable history we can do better for certain
situations i.e. when the recipe does not currently set the value
outright.

Additionally, correctly remove _appends for variables we are changing if
we're trying to remove the value added in the _append.

Fixes [YOCTO #12623] and partially fixes [YOCTO #9360].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py | 71 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 68 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index aa64553c06b..9de291f5b59 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -52,6 +52,29 @@ def parse_recipe(cooker, fn, appendfiles):
     return envdata
 
 
+def simplify_history(history, d):
+    """
+    Eliminate any irrelevant events from a variable history
+    """
+    ret_history = []
+    has_set = False
+    # Go backwards through the history and remove any immediate operations
+    # before the most recent set
+    for event in reversed(history):
+        if 'flag' in event or not 'file' in event:
+            continue
+        if event['op'] == 'set':
+            if has_set:
+                continue
+            has_set = True
+        elif event['op'] in ('append', 'prepend', 'postdot', 'predot'):
+            # Reminder: "append" and "prepend" mean += and =+ respectively, NOT _append / _prepend
+            if has_set:
+                continue
+        ret_history.insert(0, event)
+    return ret_history
+
+
 def get_var_files(fn, varlist, d):
     """Find the file in which each of a list of variables is set.
     Note: requires variable history to be enabled when parsing.
@@ -176,7 +199,14 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
     def outputvalue(name, lines, rewindcomments=False):
         if values[name] is None:
             return
-        rawtext = '%s = "%s"%s' % (name, values[name], newline)
+        if isinstance(values[name], tuple):
+            op, value = values[name]
+            if op == '+=' and value.strip() == '':
+                return
+        else:
+            value = values[name]
+            op = '='
+        rawtext = '%s %s "%s"%s' % (name, op, value, newline)
         addlines = []
         nowrap = False
         for nowrap_re in nowrap_vars_res:
@@ -186,10 +216,10 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
         if nowrap:
             addlines.append(rawtext)
         elif name in list_vars:
-            splitvalue = split_var_value(values[name], assignment=False)
+            splitvalue = split_var_value(value, assignment=False)
             if len(splitvalue) > 1:
                 linesplit = ' \\\n' + (' ' * (len(name) + 4))
-                addlines.append('%s = "%s%s"%s' % (name, linesplit.join(splitvalue), linesplit, newline))
+                addlines.append('%s %s "%s%s"%s' % (name, op, linesplit.join(splitvalue), linesplit, newline))
             else:
                 addlines.append(rawtext)
         else:
@@ -321,12 +351,47 @@ def patch_recipe(d, fn, varvalues, patch=False, relpath='', redirect_output=None
     """Modify a list of variable values in the specified recipe. Handles inc files if
     used by the recipe.
     """
+    overrides = d.getVar('OVERRIDES').split(':')
+    def override_applicable(hevent):
+        op = hevent['op']
+        if '[' in op:
+            opoverrides = op.split('[')[1].split(']')[0].split('_')
+            for opoverride in opoverrides:
+                if not opoverride in overrides:
+                    return False
+        return True
+
     varlist = varvalues.keys()
+    fn = os.path.abspath(fn)
     varfiles = get_var_files(fn, varlist, d)
     locs = localise_file_vars(fn, varfiles, varlist)
     patches = []
     for f,v in locs.items():
         vals = {k: varvalues[k] for k in v}
+        f = os.path.abspath(f)
+        if f == fn:
+            extravals = {}
+            for var, value in vals.items():
+                if var in list_vars:
+                    history = simplify_history(d.varhistory.variable(var), d)
+                    recipe_set = False
+                    for event in history:
+                        if os.path.abspath(event['file']) == fn:
+                            if event['op'] == 'set':
+                                recipe_set = True
+                    if not recipe_set:
+                        for event in history:
+                            if event['op'].startswith('_remove'):
+                                continue
+                            if not override_applicable(event):
+                                continue
+                            newvalue = value.replace(event['detail'], '')
+                            if newvalue == value and os.path.abspath(event['file']) == fn and event['op'].startswith('_'):
+                                op = event['op'].replace('[', '_').replace(']', '')
+                                extravals[var + op] = None
+                            value = newvalue
+                            vals[var] = ('+=', value)
+            vals.update(extravals)
         patchdata = patch_recipe_file(f, vals, patch, relpath, redirect_output)
         if patch:
             patches.append(patchdata)
-- 
2.17.2

[PATCH 2/4] lib/oe/recipeutils: patch_recipe(): fix replacing varflag values

[Paul Eggleton]

The code here wasn't correctly getting the variable history for
varflags, so for example if you did a devtool upgrade on a recipe where
the SRC_URI checksums were in the .inc file (typical for python recipes
in order to support both python 2 and 3) then after the upgrade the
new values would be set in the recipe and the old values were left in
the .inc, which is not right. Teach the code here how to get the history
for varflags so it works properly.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 9de291f5b59..886ad26f178 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -81,11 +81,19 @@ def get_var_files(fn, varlist, d):
     """
     varfiles = {}
     for v in varlist:
-        history = d.varhistory.variable(v)
         files = []
-        for event in history:
-            if 'file' in event and not 'flag' in event:
-                files.append(event['file'])
+        if '[' in v:
+            varsplit = v.split('[')
+            varflag = varsplit[1].split(']')[0]
+            history = d.varhistory.variable(varsplit[0])
+            for event in history:
+                if 'file' in event and event.get('flag', '') == varflag:
+                    files.append(event['file'])
+        else:
+            history = d.varhistory.variable(v)
+            for event in history:
+                if 'file' in event and not 'flag' in event:
+                    files.append(event['file'])
         if files:
             actualfile = files[-1]
         else:
-- 
2.17.2

[PATCH 3/4] lib/oe/recipeutils: drop obsolete functions

[Paul Eggleton]

These date from the time before Tinfoil's API covered this functionality
(back when you could actually access cooker from a tinfoil-based
script).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py | 25 -------------------------
 1 file changed, 25 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 886ad26f178..9c99164d248 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -27,31 +27,6 @@ list_vars = ['SRC_URI', 'LIC_FILES_CHKSUM']
 meta_vars = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION']
 
 
-def pn_to_recipe(cooker, pn, mc=''):
-    """Convert a recipe name (PN) to the path to the recipe file"""
-
-    best = cooker.findBestProvider(pn, mc)
-    return best[3]
-
-
-def get_unavailable_reasons(cooker, pn):
-    """If a recipe could not be found, find out why if possible"""
-    import bb.taskdata
-    taskdata = bb.taskdata.TaskData(None, skiplist=cooker.skiplist)
-    return taskdata.get_reasons(pn)
-
-
-def parse_recipe(cooker, fn, appendfiles):
-    """
-    Parse an individual recipe file, optionally with a list of
-    bbappend files.
-    """
-    import bb.cache
-    parser = bb.cache.NoCache(cooker.databuilder)
-    envdata = parser.loadDataFull(fn, appendfiles)
-    return envdata
-
-
 def simplify_history(history, d):
     """
     Eliminate any irrelevant events from a variable history
-- 
2.17.2

[PATCH 4/4] oe-selftest: add some tests for recipeutils module

[Paul Eggleton]

Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/cases/recipeutils.py | 135 ++++++++++++++++++++
 1 file changed, 135 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py

diff --git a/meta/lib/oeqa/selftest/cases/recipeutils.py b/meta/lib/oeqa/selftest/cases/recipeutils.py
new file mode 100644
index 00000000000..c7036c7f04e
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/recipeutils.py
@@ -0,0 +1,135 @@
+import os
+import re
+import time
+import logging
+import bb.tinfoil
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd
+from oeqa.core.decorator.oeid import OETestID
+
+
+def setUpModule():
+    global tinfoil
+    tinfoil = bb.tinfoil.Tinfoil(tracking=True)
+    tinfoil.prepare(config_only=False, quiet=2)
+
+
+def tearDownModule():
+    tinfoil.shutdown()
+
+
+class RecipeUtilsTests(OESelftestTestCase):
+    """ Tests for the recipeutils module functions """
+
+    def test_patch_recipe_varflag(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('python3-async-test')
+        vals = {'SRC_URI[md5sum]': 'woot', 'LICENSE': 'something'}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-devtools/python/python-async-test.inc
++++ b/../meta-selftest/recipes-devtools/python/python-async-test.inc
+@@ -1,14 +1,14 @@
+ SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+ HOMEPAGE = "http://github.com/gitpython-developers/async"
+ SECTION = "devel/python"
+-LICENSE = "BSD"
++LICENSE = "something"
+ LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+ 
+ inherit pypi
+ 
+ PYPI_PACKAGE = "async"
+ 
+-SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
++SRC_URI[md5sum] = "woot"
+ SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+ 
+ RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.maxDiff = None
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_singleappend(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        del val[1]
+        val = ' '.join(val)
+        vals = {'SRC_URI': val}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,4 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+ SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_appends(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        vals = {'SRC_URI': val[0]}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,3 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+-SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_validate_pn(self):
+        import oe.recipeutils
+        expected_results = {
+            'test': '',
+            'glib-2.0': '',
+            'gtk+': '',
+            'forcevariable': 'reserved',
+            'pn-something': 'reserved',
+            'test.bb': 'file',
+            'test_one': 'character',
+            'test!': 'character',
+        }
+
+        for pn, expected in expected_results.items():
+            result = oe.recipeutils.validate_pn(pn)
+            if expected:
+                self.assertIn(expected, result)
+            else:
+                self.assertEqual(result, '')
+
+    def test_split_var_value(self):
+        import oe.recipeutils
+        res = oe.recipeutils.split_var_value('test.1 test.2 ${@call_function("hi there world", false)} test.4')
+        self.assertEqual(res, ['test.1', 'test.2', '${@call_function("hi there world", false)}', 'test.4'])
-- 
2.17.2

[PATCH v2 4/4] oe-selftest: add some tests for recipeutils module

[Paul Eggleton]

Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../python/python-async-test.inc              |  16 +++
 .../python/python3-async-test_0.6.2.bb        |   2 +
 .../recipeutils/recipeutils-test.inc          |   5 +
 .../recipeutils/recipeutils-test/anotherfile  |   0
 .../recipeutils/recipeutils-test/somefile     |   0
 .../recipeutils/recipeutils-test_1.2.bb       |  13 ++
 meta/lib/oeqa/selftest/cases/recipeutils.py   | 135 ++++++++++++++++++
 7 files changed, 171 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python-async-test.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py

diff --git a/meta-selftest/recipes-devtools/python/python-async-test.inc b/meta-selftest/recipes-devtools/python/python-async-test.inc
new file mode 100644
index 00000000000..c9602e8e52d
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python-async-test.inc
@@ -0,0 +1,16 @@
+SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+HOMEPAGE = "http://github.com/gitpython-developers/async"
+SECTION = "devel/python"
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+
+inherit pypi
+
+PYPI_PACKAGE = "async"
+
+SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
+SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+
+RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+
+BBCLASSEXTEND = "nativesdk"
diff --git a/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb b/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
new file mode 100644
index 00000000000..22e241afb3c
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
@@ -0,0 +1,2 @@
+inherit setuptools3
+require python-async-test.inc
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc b/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
new file mode 100644
index 00000000000..8490b902d75
--- /dev/null
+++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
@@ -0,0 +1,5 @@
+SRC_URI = "http://xorg.freedesktop.org/releases/individual/lib/libxshmfence-${PV}.tar.bz2"
+
+SRC_URI[md5sum] = "66662e76899112c0f99e22f2fc775a7e"
+SRC_URI[sha256sum] = "d21b2d1fd78c1efbe1f2c16dae1cb23f8fd231dcf891465b8debe636a9054b0c"
+
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile b/meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile b/meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
new file mode 100644
index 00000000000..f6da97b2d43
--- /dev/null
+++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
@@ -0,0 +1,13 @@
+SUMMARY = "Test recipe for recipeutils.patch_recipe()"
+
+require recipeutils-test.inc
+
+LICENSE = "Proprietary"
+
+DEPENDS += "virtual/libx11"
+
+BBCLASSEXTEND = "native nativesdk"
+
+SRC_URI += "file://somefile"
+
+SRC_URI_append = " file://anotherfile"
diff --git a/meta/lib/oeqa/selftest/cases/recipeutils.py b/meta/lib/oeqa/selftest/cases/recipeutils.py
new file mode 100644
index 00000000000..c7036c7f04e
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/recipeutils.py
@@ -0,0 +1,135 @@
+import os
+import re
+import time
+import logging
+import bb.tinfoil
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd
+from oeqa.core.decorator.oeid import OETestID
+
+
+def setUpModule():
+    global tinfoil
+    tinfoil = bb.tinfoil.Tinfoil(tracking=True)
+    tinfoil.prepare(config_only=False, quiet=2)
+
+
+def tearDownModule():
+    tinfoil.shutdown()
+
+
+class RecipeUtilsTests(OESelftestTestCase):
+    """ Tests for the recipeutils module functions """
+
+    def test_patch_recipe_varflag(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('python3-async-test')
+        vals = {'SRC_URI[md5sum]': 'woot', 'LICENSE': 'something'}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-devtools/python/python-async-test.inc
++++ b/../meta-selftest/recipes-devtools/python/python-async-test.inc
+@@ -1,14 +1,14 @@
+ SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+ HOMEPAGE = "http://github.com/gitpython-developers/async"
+ SECTION = "devel/python"
+-LICENSE = "BSD"
++LICENSE = "something"
+ LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+ 
+ inherit pypi
+ 
+ PYPI_PACKAGE = "async"
+ 
+-SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
++SRC_URI[md5sum] = "woot"
+ SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+ 
+ RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.maxDiff = None
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_singleappend(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        del val[1]
+        val = ' '.join(val)
+        vals = {'SRC_URI': val}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,4 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+ SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_appends(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        vals = {'SRC_URI': val[0]}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True)
+
+        expected_patch = """
+--- a/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/../meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,3 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+-SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_validate_pn(self):
+        import oe.recipeutils
+        expected_results = {
+            'test': '',
+            'glib-2.0': '',
+            'gtk+': '',
+            'forcevariable': 'reserved',
+            'pn-something': 'reserved',
+            'test.bb': 'file',
+            'test_one': 'character',
+            'test!': 'character',
+        }
+
+        for pn, expected in expected_results.items():
+            result = oe.recipeutils.validate_pn(pn)
+            if expected:
+                self.assertIn(expected, result)
+            else:
+                self.assertEqual(result, '')
+
+    def test_split_var_value(self):
+        import oe.recipeutils
+        res = oe.recipeutils.split_var_value('test.1 test.2 ${@call_function("hi there world", false)} test.4')
+        self.assertEqual(res, ['test.1', 'test.2', '${@call_function("hi there world", false)}', 'test.4'])
-- 
2.17.2

Re: [PATCH 0/4] recipeutils fixes for devtool

[Paul Eggleton]

On Wednesday, 28 November 2018 5:16:13 PM NZDT Paul Eggleton wrote:
> A couple of fixes for issues in the recipeutils module (which affect
> devtool), a cleanup patch, and some additional related tests for
> oe-selftest.

Oops, patch 4/4 missed out the actual meta-selftest recipes that a few of the 
tests rely upon. Branch re-pushed and v2 of 4/4 sent.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

[Paul Eggleton]

Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../python/python-async-test.inc              |  16 ++
 .../python/python3-async-test_0.6.2.bb        |   2 +
 .../recipeutils/recipeutils-test.inc          |   5 +
 .../recipeutils/recipeutils-test/anotherfile  |   0
 .../recipeutils/recipeutils-test/somefile     |   0
 .../recipeutils/recipeutils-test_1.2.bb       |  13 ++
 meta/lib/oeqa/selftest/cases/recipeutils.py   | 138 ++++++++++++++++++
 7 files changed, 174 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python-async-test.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py

diff --git a/meta-selftest/recipes-devtools/python/python-async-test.inc b/meta-selftest/recipes-devtools/python/python-async-test.inc
new file mode 100644
index 00000000000..c9602e8e52d
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python-async-test.inc
@@ -0,0 +1,16 @@
+SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+HOMEPAGE = "http://github.com/gitpython-developers/async"
+SECTION = "devel/python"
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+
+inherit pypi
+
+PYPI_PACKAGE = "async"
+
+SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
+SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+
+RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+
+BBCLASSEXTEND = "nativesdk"
diff --git a/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb b/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
new file mode 100644
index 00000000000..22e241afb3c
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
@@ -0,0 +1,2 @@
+inherit setuptools3
+require python-async-test.inc
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc b/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
new file mode 100644
index 00000000000..8490b902d75
--- /dev/null
+++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
@@ -0,0 +1,5 @@
+SRC_URI = "http://xorg.freedesktop.org/releases/individual/lib/libxshmfence-${PV}.tar.bz2"
+
+SRC_URI[md5sum] = "66662e76899112c0f99e22f2fc775a7e"
+SRC_URI[sha256sum] = "d21b2d1fd78c1efbe1f2c16dae1cb23f8fd231dcf891465b8debe636a9054b0c"
+
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile b/meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile b/meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
new file mode 100644
index 00000000000..f6da97b2d43
--- /dev/null
+++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
@@ -0,0 +1,13 @@
+SUMMARY = "Test recipe for recipeutils.patch_recipe()"
+
+require recipeutils-test.inc
+
+LICENSE = "Proprietary"
+
+DEPENDS += "virtual/libx11"
+
+BBCLASSEXTEND = "native nativesdk"
+
+SRC_URI += "file://somefile"
+
+SRC_URI_append = " file://anotherfile"
diff --git a/meta/lib/oeqa/selftest/cases/recipeutils.py b/meta/lib/oeqa/selftest/cases/recipeutils.py
new file mode 100644
index 00000000000..b7cb55baae1
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/recipeutils.py
@@ -0,0 +1,138 @@
+import os
+import re
+import time
+import logging
+import bb.tinfoil
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd
+from oeqa.core.decorator.oeid import OETestID
+
+
+def setUpModule():
+    global tinfoil
+    tinfoil = bb.tinfoil.Tinfoil(tracking=True)
+    tinfoil.prepare(config_only=False, quiet=2)
+
+
+def tearDownModule():
+    tinfoil.shutdown()
+
+
+class RecipeUtilsTests(OESelftestTestCase):
+    """ Tests for the recipeutils module functions """
+
+    def test_patch_recipe_varflag(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('python3-async-test')
+        vals = {'SRC_URI[md5sum]': 'aaaaaa', 'LICENSE': 'something'}
+        corebase = rd.getVar('COREBASE')
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True, relpath=corebase)
+
+        expected_patch = """
+--- a/meta-selftest/recipes-devtools/python/python-async-test.inc
++++ b/meta-selftest/recipes-devtools/python/python-async-test.inc
+@@ -1,14 +1,14 @@
+ SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+ HOMEPAGE = "http://github.com/gitpython-developers/async"
+ SECTION = "devel/python"
+-LICENSE = "BSD"
++LICENSE = "something"
+ LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+ 
+ inherit pypi
+ 
+ PYPI_PACKAGE = "async"
+ 
+-SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
++SRC_URI[md5sum] = "aaaaaa"
+ SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+ 
+ RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.maxDiff = None
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_singleappend(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        del val[1]
+        val = ' '.join(val)
+        vals = {'SRC_URI': val}
+        corebase = rd.getVar('COREBASE')
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True, relpath=corebase)
+
+        expected_patch = """
+--- a/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,4 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+ SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_appends(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        vals = {'SRC_URI': val[0]}
+        corebase = rd.getVar('COREBASE')
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True, relpath=corebase)
+
+        expected_patch = """
+--- a/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,3 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+-SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_validate_pn(self):
+        import oe.recipeutils
+        expected_results = {
+            'test': '',
+            'glib-2.0': '',
+            'gtk+': '',
+            'forcevariable': 'reserved',
+            'pn-something': 'reserved',
+            'test.bb': 'file',
+            'test_one': 'character',
+            'test!': 'character',
+        }
+
+        for pn, expected in expected_results.items():
+            result = oe.recipeutils.validate_pn(pn)
+            if expected:
+                self.assertIn(expected, result)
+            else:
+                self.assertEqual(result, '')
+
+    def test_split_var_value(self):
+        import oe.recipeutils
+        res = oe.recipeutils.split_var_value('test.1 test.2 ${@call_function("hi there world", false)} test.4')
+        self.assertEqual(res, ['test.1', 'test.2', '${@call_function("hi there world", false)}', 'test.4'])
-- 
2.17.2

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

[Richard Purdie]

On Thu, 2018-11-29 at 22:21 +1300, Paul Eggleton wrote:
> Add some tests for functions in meta/lib/oe/recipeutils.py, in
> particular for a few issues I've just fixed. I haven't added tests
> for
> all of the functions - some of them are already being tested via
> devtool
> in any case.

Hate to say it but this still isn't working:

https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d

probably due to it not working with the -j option :/

Cheers,

Richard

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

[Paul Eggleton]

On Friday, 30 November 2018 3:06:32 AM NZDT Richard Purdie wrote:
> On Thu, 2018-11-29 at 22:21 +1300, Paul Eggleton wrote:
> > Add some tests for functions in meta/lib/oe/recipeutils.py, in
> > particular for a few issues I've just fixed. I haven't added tests
> > for
> > all of the functions - some of them are already being tested via
> > devtool
> > in any case.
> 
> Hate to say it but this still isn't working:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d
> 
> probably due to it not working with the -j option :/

Right, so it seems that COREBASE isn't pointing where I'd expect it to 
here - instead of the copy of the core metadata that's made for the test, it's
pointing to the original layer. Is that a bug?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

[richard.purdie]

On Fri, 2018-11-30 at 08:35 +1300, Paul Eggleton wrote:
> On Friday, 30 November 2018 3:06:32 AM NZDT Richard Purdie wrote:
> > On Thu, 2018-11-29 at 22:21 +1300, Paul Eggleton wrote:
> > > Add some tests for functions in meta/lib/oe/recipeutils.py, in
> > > particular for a few issues I've just fixed. I haven't added
> > > tests
> > > for
> > > all of the functions - some of them are already being tested via
> > > devtool
> > > in any case.
> > 
> > Hate to say it but this still isn't working:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d
> > 
> > probably due to it not working with the -j option :/
> 
> Right, so it seems that COREBASE isn't pointing where I'd expect it
> to 
> here - instead of the copy of the core metadata that's made for the
> test, it's
> pointing to the original layer. Is that a bug?

I put 

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=74b7df6ff73764e108bd81e05835c8a7d134c187

together to fix this. Its not a bug since the tests modify meta-
selftest and therefore when running in parallel, we create copies of
that layer per build directory.

I do think we should start using the separate build directory always
for oe-selftest for consistency though. It would also make ensuring
some things like rm_work disabled easier.

Cheers,

Richard

[PATCH v4 4/4] oe-selftest: add some tests for recipeutils module

[Paul Eggleton]

Add some tests for functions in meta/lib/oe/recipeutils.py, in
particular for a few issues I've just fixed. I haven't added tests for
all of the functions - some of them are already being tested via devtool
in any case.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 .../python/python-async-test.inc              |  16 ++
 .../python/python3-async-test_0.6.2.bb        |   2 +
 .../recipeutils/recipeutils-test.inc          |   5 +
 .../recipeutils/recipeutils-test/anotherfile  |   0
 .../recipeutils/recipeutils-test/somefile     |   0
 .../recipeutils/recipeutils-test_1.2.bb       |  13 ++
 meta/lib/oeqa/selftest/cases/recipeutils.py   | 137 ++++++++++++++++++
 7 files changed, 173 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python-async-test.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
 create mode 100644 meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
 create mode 100644 meta/lib/oeqa/selftest/cases/recipeutils.py

diff --git a/meta-selftest/recipes-devtools/python/python-async-test.inc b/meta-selftest/recipes-devtools/python/python-async-test.inc
new file mode 100644
index 00000000000..c9602e8e52d
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python-async-test.inc
@@ -0,0 +1,16 @@
+SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+HOMEPAGE = "http://github.com/gitpython-developers/async"
+SECTION = "devel/python"
+LICENSE = "BSD"
+LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+
+inherit pypi
+
+PYPI_PACKAGE = "async"
+
+SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
+SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+
+RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+
+BBCLASSEXTEND = "nativesdk"
diff --git a/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb b/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
new file mode 100644
index 00000000000..22e241afb3c
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python3-async-test_0.6.2.bb
@@ -0,0 +1,2 @@
+inherit setuptools3
+require python-async-test.inc
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc b/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
new file mode 100644
index 00000000000..8490b902d75
--- /dev/null
+++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test.inc
@@ -0,0 +1,5 @@
+SRC_URI = "http://xorg.freedesktop.org/releases/individual/lib/libxshmfence-${PV}.tar.bz2"
+
+SRC_URI[md5sum] = "66662e76899112c0f99e22f2fc775a7e"
+SRC_URI[sha256sum] = "d21b2d1fd78c1efbe1f2c16dae1cb23f8fd231dcf891465b8debe636a9054b0c"
+
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile b/meta-selftest/recipes-test/recipeutils/recipeutils-test/anotherfile
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile b/meta-selftest/recipes-test/recipeutils/recipeutils-test/somefile
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
new file mode 100644
index 00000000000..f6da97b2d43
--- /dev/null
+++ b/meta-selftest/recipes-test/recipeutils/recipeutils-test_1.2.bb
@@ -0,0 +1,13 @@
+SUMMARY = "Test recipe for recipeutils.patch_recipe()"
+
+require recipeutils-test.inc
+
+LICENSE = "Proprietary"
+
+DEPENDS += "virtual/libx11"
+
+BBCLASSEXTEND = "native nativesdk"
+
+SRC_URI += "file://somefile"
+
+SRC_URI_append = " file://anotherfile"
diff --git a/meta/lib/oeqa/selftest/cases/recipeutils.py b/meta/lib/oeqa/selftest/cases/recipeutils.py
new file mode 100644
index 00000000000..dd2f55839ae
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/recipeutils.py
@@ -0,0 +1,137 @@
+import os
+import re
+import time
+import logging
+import bb.tinfoil
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd, get_test_layer
+from oeqa.core.decorator.oeid import OETestID
+
+
+def setUpModule():
+    global tinfoil
+    global metaselftestpath
+    metaselftestpath = get_test_layer()
+    tinfoil = bb.tinfoil.Tinfoil(tracking=True)
+    tinfoil.prepare(config_only=False, quiet=2)
+
+
+def tearDownModule():
+    tinfoil.shutdown()
+
+
+class RecipeUtilsTests(OESelftestTestCase):
+    """ Tests for the recipeutils module functions """
+
+    def test_patch_recipe_varflag(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('python3-async-test')
+        vals = {'SRC_URI[md5sum]': 'aaaaaa', 'LICENSE': 'something'}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True, relpath=metaselftestpath)
+
+        expected_patch = """
+--- a/recipes-devtools/python/python-async-test.inc
++++ b/recipes-devtools/python/python-async-test.inc
+@@ -1,14 +1,14 @@
+ SUMMARY = "Python framework to process interdependent tasks in a pool of workers"
+ HOMEPAGE = "http://github.com/gitpython-developers/async"
+ SECTION = "devel/python"
+-LICENSE = "BSD"
++LICENSE = "something"
+ LIC_FILES_CHKSUM = "file://PKG-INFO;beginline=8;endline=8;md5=88df8e78b9edfd744953862179f2d14e"
+ 
+ inherit pypi
+ 
+ PYPI_PACKAGE = "async"
+ 
+-SRC_URI[md5sum] = "9b06b5997de2154f3bc0273f80bcef6b"
++SRC_URI[md5sum] = "aaaaaa"
+ SRC_URI[sha256sum] = "ac6894d876e45878faae493b0cf61d0e28ec417334448ac0a6ea2229d8343051"
+ 
+ RDEPENDS_${PN} += "${PYTHON_PN}-threading"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.maxDiff = None
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_singleappend(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        del val[1]
+        val = ' '.join(val)
+        vals = {'SRC_URI': val}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True, relpath=metaselftestpath)
+
+        expected_patch = """
+--- a/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,4 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+ SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_patch_recipe_appends(self):
+        import oe.recipeutils
+        rd = tinfoil.parse_recipe('recipeutils-test')
+        val = rd.getVar('SRC_URI', False).split()
+        vals = {'SRC_URI': val[0]}
+        patches = oe.recipeutils.patch_recipe(rd, rd.getVar('FILE'), vals, patch=True, relpath=metaselftestpath)
+
+        expected_patch = """
+--- a/recipes-test/recipeutils/recipeutils-test_1.2.bb
++++ b/recipes-test/recipeutils/recipeutils-test_1.2.bb
+@@ -8,6 +8,3 @@
+ 
+ BBCLASSEXTEND = "native nativesdk"
+ 
+-SRC_URI += "file://somefile"
+-
+-SRC_URI_append = " file://anotherfile"
+"""
+        patchlines = []
+        for f in patches:
+            for line in f:
+                patchlines.append(line)
+        self.assertEqual(''.join(patchlines).strip(), expected_patch.strip())
+
+
+    def test_validate_pn(self):
+        import oe.recipeutils
+        expected_results = {
+            'test': '',
+            'glib-2.0': '',
+            'gtk+': '',
+            'forcevariable': 'reserved',
+            'pn-something': 'reserved',
+            'test.bb': 'file',
+            'test_one': 'character',
+            'test!': 'character',
+        }
+
+        for pn, expected in expected_results.items():
+            result = oe.recipeutils.validate_pn(pn)
+            if expected:
+                self.assertIn(expected, result)
+            else:
+                self.assertEqual(result, '')
+
+    def test_split_var_value(self):
+        import oe.recipeutils
+        res = oe.recipeutils.split_var_value('test.1 test.2 ${@call_function("hi there world", false)} test.4')
+        self.assertEqual(res, ['test.1', 'test.2', '${@call_function("hi there world", false)}', 'test.4'])
-- 
2.17.2

Re: [PATCH v3 4/4] oe-selftest: add some tests for recipeutils module

[Paul Eggleton]

On Friday, 30 November 2018 12:09:58 PM NZDT richard.purdie@linuxfoundation.org wrote:
> On Fri, 2018-11-30 at 08:35 +1300, Paul Eggleton wrote:
> > On Friday, 30 November 2018 3:06:32 AM NZDT Richard Purdie wrote:
> > > On Thu, 2018-11-29 at 22:21 +1300, Paul Eggleton wrote:
> > > > Add some tests for functions in meta/lib/oe/recipeutils.py, in
> > > > particular for a few issues I've just fixed. I haven't added
> > > > tests
> > > > for
> > > > all of the functions - some of them are already being tested via
> > > > devtool
> > > > in any case.
> > > 
> > > Hate to say it but this still isn't working:
> > > 
> > > https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/28/steps/7/logs/step2d
> > > 
> > > probably due to it not working with the -j option :/
> > 
> > Right, so it seems that COREBASE isn't pointing where I'd expect it
> > to 
> > here - instead of the copy of the core metadata that's made for the
> > test, it's
> > pointing to the original layer. Is that a bug?
> 
> I put 
> 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=74b7df6ff73764e108bd81e05835c8a7d134c187
> 
> together to fix this. Its not a bug since the tests modify meta-
> selftest and therefore when running in parallel, we create copies of
> that layer per build directory.

OK, I've sent a fix - I discussed with RP on IRC but for others' benefit
there's already a function in oeqa.utils.commands that gets the path
to meta-selftest from BBLAYERS so I used that in v4 that I just sent. 
I also tested this one with -j which I hadn't with the previous versions.

> I do think we should start using the separate build directory always
> for oe-selftest for consistency though. It would also make ensuring
> some things like rm_work disabled easier.

Probably since there are gotchas like this one.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre