[PATCH 0/7] OEQA, devtool and YP compatible fixes

[PATCH 0/7] OEQA, devtool and YP compatible fixes

[Aníbal Limón]

The following changes since commit 4a7612c7a12b9a381fb8343ba9586272b889fc15:

  buildhistory: skip tests if GitPython module is missing (2017-06-07 16:00:49 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib alimon/simple_patches
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/simple_patches

Aníbal Limón (7):
  devtool/standard: Fix lock in _prep_extract_operation
  scripts/yocto-compat-layer.py: Return non-zero when layer test fail
  scripts/yocto-compat-layer-wrapper: Use realpath of output_log
  oeqa: Change the order to logDetails and logSummary
  oeqa/core/loader: Allow unittest.TestCase's to be executed
  oeqa/cases/oelib: Change default case class to unittest.case.TestCase
  oeqa/core/loader: Fix filtering on test modules with submodules

 meta/classes/testimage.bbclass                |  2 +-
 meta/classes/testsdk.bbclass                  |  4 +-
 meta/lib/oeqa/core/context.py                 |  2 +-
 meta/lib/oeqa/core/loader.py                  | 82 +++++++++++++--------------
 meta/lib/oeqa/core/runner.py                  | 25 ++++----
 meta/lib/oeqa/selftest/cases/oelib/elf.py     |  4 +-
 meta/lib/oeqa/selftest/cases/oelib/license.py |  6 +-
 meta/lib/oeqa/selftest/cases/oelib/path.py    |  4 +-
 meta/lib/oeqa/selftest/cases/oelib/types.py   |  6 +-
 meta/lib/oeqa/selftest/cases/oelib/utils.py   |  6 +-
 meta/lib/oeqa/selftest/context.py             |  2 +-
 scripts/lib/devtool/standard.py               |  1 +
 scripts/yocto-compat-layer-wrapper            | 18 +++++-
 scripts/yocto-compat-layer.py                 |  5 +-
 14 files changed, 95 insertions(+), 72 deletions(-)

-- 
2.1.4

[PATCH 1/7] devtool/standard: Fix lock in _prep_extract_operation

[Aníbal Limón]

If for any reason the parse_recipe fail in extract command
the process gets locked because Cooker is expecting the
finish event by tinfoil.

For example:

$ devtool extract remake /tmp/remake

ERROR: remake is unavailable:
  remake was skipped: PREFERRED_PROVIDER_virtual/make set to make, not remake

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 2ecef99..7e342e7 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -436,6 +436,7 @@ def _prep_extract_operation(config, basepath, recipename, tinfoil=None):
 
     rd = parse_recipe(config, tinfoil, recipename, True)
     if not rd:
+        tinfoil.shutdown()
         return None
 
     if bb.data.inherits_class('kernel-yocto', rd):
-- 
2.1.4

[PATCH 2/7] scripts/yocto-compat-layer.py: Return non-zero when layer test fail

[Aníbal Limón]

If whatever layer tested fails returns 2 to indicate the
failure.

[YOCTO #11482]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 scripts/yocto-compat-layer.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index 0d5700b..30c55a9 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -178,16 +178,19 @@ def main():
         results_status[layer['name']] = 'PASS' if results[layer['name']].wasSuccessful() else 'FAIL'
         layers_tested = layers_tested + 1
 
+    ret = 0
     if layers_tested:
         logger.info('')
         logger.info('Summary of results:')
         logger.info('')
         for layer_name in results_status:
             logger.info('%s ... %s' % (layer_name, results_status[layer_name]))
+            if not results[layer_name].wasSuccessful():
+                ret = 2 # ret = 1 used for initialization errors
 
     cleanup_bblayers(None, None)
 
-    return 0
+    return ret
 
 if __name__ == '__main__':
     try:
-- 
2.1.4

[PATCH 3/7] scripts/yocto-compat-layer-wrapper: Use realpath of output_log

[Aníbal Limón]

We are using a temp directory, use the realpath for output log
to store the results in the original BUILDDIR.

[YOCTO #11571]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 scripts/yocto-compat-layer-wrapper | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/scripts/yocto-compat-layer-wrapper b/scripts/yocto-compat-layer-wrapper
index db4b687..b6baeb4 100755
--- a/scripts/yocto-compat-layer-wrapper
+++ b/scripts/yocto-compat-layer-wrapper
@@ -13,13 +13,29 @@ if [ -z "$BUILDDIR" ]; then
 	exit 2
 fi
 
+# since we are using a temp directory, use the realpath for output
+# log option
+output_log=''
+while getopts o: name
+do
+	case $name in
+	o) output_log=$(realpath "$OPTARG")
+	esac
+done
+shift $(($OPTIND - 1))
+
+# generate a temp directory to run compat layer script
 base_dir=$(realpath $BUILDDIR/../)
 cd $base_dir
 
 build_dir=$(mktemp -p $base_dir -d -t build-XXXX)
 
 source oe-init-build-env $build_dir
-yocto-compat-layer.py "$@"
+if [[ $output_log != '' ]]; then
+	yocto-compat-layer.py -o "$output_log" "$*"
+else
+	yocto-compat-layer.py "$@"
+fi
 retcode=$?
 
 rm -rf $build_dir
-- 
2.1.4

[PATCH 4/7] oeqa: Change the order to logDetails and logSummary

[Aníbal Limón]

Is better to log the summary at end to see in an easy way
the actual result of the test run.

[YOCTO #11622]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/classes/testimage.bbclass    | 2 +-
 meta/classes/testsdk.bbclass      | 4 ++--
 meta/lib/oeqa/core/context.py     | 2 +-
 meta/lib/oeqa/selftest/context.py | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 1185593..6c33e16 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -292,8 +292,8 @@ def testimage_main(d):
     # Show results (if we have them)
     if not results:
         bb.fatal('%s - FAILED - tests were interrupted during execution' % pn)
-    results.logSummary(pn)
     results.logDetails()
+    results.logSummary(pn)
     if not results.wasSuccessful():
         bb.fatal('%s - FAILED - check the task log and the ssh log' % pn)
 
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 8a9e680..6b51a33 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -72,8 +72,8 @@ def testsdk_main(d):
         component = "%s %s" % (pn, OESDKTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
 
-        result.logSummary(component, context_msg)
         result.logDetails()
+        result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
             fail = True
@@ -176,8 +176,8 @@ def testsdkext_main(d):
         component = "%s %s" % (pn, OESDKExtTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
 
-        result.logSummary(component, context_msg)
         result.logDetails()
+        result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
             fail = True
diff --git a/meta/lib/oeqa/core/context.py b/meta/lib/oeqa/core/context.py
index 0dbf5c3..2d543ff 100644
--- a/meta/lib/oeqa/core/context.py
+++ b/meta/lib/oeqa/core/context.py
@@ -158,8 +158,8 @@ class OETestContextExecutor(object):
         else:
             self._pre_run()
             rc = self.tc.runTests(**self.tc_kwargs['run'])
-            rc.logSummary(self.name)
             rc.logDetails()
+            rc.logSummary(self.name)
 
         output_link = os.path.join(os.path.dirname(args.output_log),
                 "%s-results.log" % self.name)
diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index ca73070..8c8439b 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -179,8 +179,8 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
         else:
             self._pre_run()
             rc = self.tc.runTests(**self.tc_kwargs['run'])
-            rc.logSummary(self.name)
             rc.logDetails()
+            rc.logSummary(self.name)
 
         return rc
     
-- 
2.1.4

[PATCH 5/7] oeqa/core/loader: Allow unittest.TestCase's to be executed

[Aníbal Limón]

Currently there was a restriction to only execute tests that's
inherits from OETestCase but in some circunstancies the features
from the OEQA framework isn't needed so we need to support
basic unittests.

[YOCTO #10828]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/loader.py | 12 +++++++-----
 meta/lib/oeqa/core/runner.py | 25 ++++++++++++++-----------
 2 files changed, 21 insertions(+), 16 deletions(-)

diff --git a/meta/lib/oeqa/core/loader.py b/meta/lib/oeqa/core/loader.py
index 7cc4d4c..b9385ea 100644
--- a/meta/lib/oeqa/core/loader.py
+++ b/meta/lib/oeqa/core/loader.py
@@ -182,7 +182,7 @@ class OETestLoader(unittest.TestLoader):
                         return True
 
         # Decorator filters
-        if self.filters:
+        if self.filters and isinstance(case, OETestCase):
             filters = self.filters.copy()
             case_decorators = [cd for cd in case.decorators
                                if cd.__class__ in self.used_filters]
@@ -200,7 +200,8 @@ class OETestLoader(unittest.TestLoader):
         return False
 
     def _getTestCase(self, testCaseClass, tcName):
-        if not hasattr(testCaseClass, '__oeqa_loader'):
+        if not hasattr(testCaseClass, '__oeqa_loader') and \
+                issubclass(testCaseClass, OETestCase):
             # In order to support data_vars validation
             # monkey patch the default setUp/tearDown{Class} to use
             # the ones provided by OETestCase
@@ -227,7 +228,8 @@ class OETestLoader(unittest.TestLoader):
             setattr(testCaseClass, '__oeqa_loader', True)
 
         case = testCaseClass(tcName)
-        setattr(case, 'decorators', [])
+        if isinstance(case, OETestCase):
+            setattr(case, 'decorators', [])
 
         return case
 
@@ -239,9 +241,9 @@ class OETestLoader(unittest.TestLoader):
             raise TypeError("Test cases should not be derived from TestSuite." \
                                 " Maybe you meant to derive %s from TestCase?" \
                                 % testCaseClass.__name__)
-        if not issubclass(testCaseClass, self.caseClass):
+        if not issubclass(testCaseClass, unittest.case.TestCase):
             raise TypeError("Test %s is not derived from %s" % \
-                    (testCaseClass.__name__, self.caseClass.__name__))
+                    (testCaseClass.__name__, unittest.case.TestCase.__name__))
 
         testCaseNames = self.getTestCaseNames(testCaseClass)
         if not testCaseNames and hasattr(testCaseClass, 'runTest'):
diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
index 7ce718e..532b25b 100644
--- a/meta/lib/oeqa/core/runner.py
+++ b/meta/lib/oeqa/core/runner.py
@@ -121,9 +121,10 @@ class OETestResult(_TestResult):
                     break
 
             oeid = -1
-            for d in case.decorators:
-                if hasattr(d, 'oeid'):
-                    oeid = d.oeid
+            if hasattr(case, 'decorators'):
+                for d in case.decorators:
+                    if hasattr(d, 'oeid'):
+                        oeid = d.oeid
 
             if fail:
                 self.tc.logger.info("RESULTS - %s - Testcase %s: %s" % (case.id(),
@@ -188,9 +189,10 @@ class OETestRunner(_TestRunner):
         def _list_cases_without_id(logger, case):
 
             found_id = False
-            for d in case.decorators:
-                if isinstance(d, OETestID):
-                    found_id = True
+            if hasattr(case, 'decorators'):
+                for d in case.decorators:
+                    if isinstance(d, OETestID):
+                        found_id = True
 
             if not found_id:
                 logger.info('oeid missing for %s' % case.id())
@@ -199,11 +201,12 @@ class OETestRunner(_TestRunner):
             oeid = None
             oetag = None
 
-            for d in case.decorators:
-                if isinstance(d, OETestID):
-                    oeid = d.oeid
-                elif isinstance(d, OETestTag):
-                    oetag = d.oetag
+            if hasattr(case, 'decorators'):
+                for d in case.decorators:
+                    if isinstance(d, OETestID):
+                        oeid = d.oeid
+                    elif isinstance(d, OETestTag):
+                        oetag = d.oetag
 
             logger.info("%s\t%s\t\t%s" % (oeid, oetag, case.id()))
 
-- 
2.1.4

[PATCH 6/7] oeqa/cases/oelib: Change default case class to unittest.case.TestCase

[Aníbal Limón]

Some tests doesn't need call bitbake so it is better to use the
basic unittest case class.

[YOCTO #10828]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/selftest/cases/oelib/elf.py     | 4 ++--
 meta/lib/oeqa/selftest/cases/oelib/license.py | 6 +++---
 meta/lib/oeqa/selftest/cases/oelib/path.py    | 4 ++--
 meta/lib/oeqa/selftest/cases/oelib/types.py   | 6 +++---
 meta/lib/oeqa/selftest/cases/oelib/utils.py   | 6 +++---
 5 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/oelib/elf.py b/meta/lib/oeqa/selftest/cases/oelib/elf.py
index 0451eba..74ee6a1 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/elf.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/elf.py
@@ -1,7 +1,7 @@
-from oeqa.selftest.case import OESelftestTestCase
+from unittest.case import TestCase
 import oe.qa
 
-class TestElf(OESelftestTestCase):
+class TestElf(TestCase):
     def test_machine_name(self):
         """
         Test elf_machine_to_string()
diff --git a/meta/lib/oeqa/selftest/cases/oelib/license.py b/meta/lib/oeqa/selftest/cases/oelib/license.py
index a6d9c9a..bfd9ed9 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/license.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/license.py
@@ -1,4 +1,4 @@
-from oeqa.selftest.case import OESelftestTestCase
+from unittest.case import TestCase
 import oe.license
 
 class SeenVisitor(oe.license.LicenseVisitor):
@@ -9,7 +9,7 @@ class SeenVisitor(oe.license.LicenseVisitor):
     def visit_Str(self, node):
         self.seen.append(node.s)
 
-class TestSingleLicense(OESelftestTestCase):
+class TestSingleLicense(TestCase):
     licenses = [
         "GPLv2",
         "LGPL-2.0",
@@ -37,7 +37,7 @@ class TestSingleLicense(OESelftestTestCase):
                 self.parse(license)
             self.assertEqual(cm.exception.license, license)
 
-class TestSimpleCombinations(OESelftestTestCase):
+class TestSimpleCombinations(TestCase):
     tests = {
         "FOO&BAR": ["FOO", "BAR"],
         "BAZ & MOO": ["BAZ", "MOO"],
diff --git a/meta/lib/oeqa/selftest/cases/oelib/path.py b/meta/lib/oeqa/selftest/cases/oelib/path.py
index 2ae5eaf..75a27c0 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/path.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/path.py
@@ -1,11 +1,11 @@
-from oeqa.selftest.case import OESelftestTestCase
+from unittest.case import TestCase
 import oe, oe.path
 import tempfile
 import os
 import errno
 import shutil
 
-class TestRealPath(OESelftestTestCase):
+class TestRealPath(TestCase):
     DIRS = [ "a", "b", "etc", "sbin", "usr", "usr/bin", "usr/binX", "usr/sbin", "usr/include", "usr/include/gdbm" ]
     FILES = [ "etc/passwd", "b/file" ]
     LINKS = [
diff --git a/meta/lib/oeqa/selftest/cases/oelib/types.py b/meta/lib/oeqa/selftest/cases/oelib/types.py
index 99c8404..6b53aa6 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/types.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/types.py
@@ -1,7 +1,7 @@
-from oeqa.selftest.case import OESelftestTestCase
+from unittest.case import TestCase
 from oe.maketype import create
 
-class TestBooleanType(OESelftestTestCase):
+class TestBooleanType(TestCase):
     def test_invalid(self):
         self.assertRaises(ValueError, create, '', 'boolean')
         self.assertRaises(ValueError, create, 'foo', 'boolean')
@@ -31,7 +31,7 @@ class TestBooleanType(OESelftestTestCase):
         self.assertEqual(create('y', 'boolean'), True)
         self.assertNotEqual(create('y', 'boolean'), False)
 
-class TestList(OESelftestTestCase):
+class TestList(TestCase):
     def assertListEqual(self, value, valid, sep=None):
         obj = create(value, 'list', separator=sep)
         self.assertEqual(obj, valid)
diff --git a/meta/lib/oeqa/selftest/cases/oelib/utils.py b/meta/lib/oeqa/selftest/cases/oelib/utils.py
index 5bc5fff..9fb6c15 100644
--- a/meta/lib/oeqa/selftest/cases/oelib/utils.py
+++ b/meta/lib/oeqa/selftest/cases/oelib/utils.py
@@ -1,7 +1,7 @@
-from oeqa.selftest.case import OESelftestTestCase
+from unittest.case import TestCase
 from oe.utils import packages_filter_out_system, trim_version
 
-class TestPackagesFilterOutSystem(OESelftestTestCase):
+class TestPackagesFilterOutSystem(TestCase):
     def test_filter(self):
         """
         Test that oe.utils.packages_filter_out_system works.
@@ -31,7 +31,7 @@ class TestPackagesFilterOutSystem(OESelftestTestCase):
         self.assertEqual(pkgs, ["foo-data"])
 
 
-class TestTrimVersion(OESelftestTestCase):
+class TestTrimVersion(TestCase):
     def test_version_exception(self):
         with self.assertRaises(TypeError):
             trim_version(None, 2)
-- 
2.1.4

[PATCH 7/7] oeqa/core/loader: Fix filtering on test modules with submodules

[Aníbal Limón]

Our filtering allows to specify which tests to run using,

<module_name>.[test_class].[test_name]

But the module name logic was restricted to only accept one level,
for example: runtime_test vs oelib.types, to support multiple
submodules use only the first part for filtering.

This allows to run the whole tests in a module with more than tree
levels.

Due to the ambiguity on the test filtering options with test cases
with more than tree levels the supported sytnax is,

<module>

or

<module>.[submoduleN].[test_class].[test_name]

[YOCTO #11632]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/loader.py | 70 +++++++++++++++++++++-----------------------
 1 file changed, 34 insertions(+), 36 deletions(-)

diff --git a/meta/lib/oeqa/core/loader.py b/meta/lib/oeqa/core/loader.py
index b9385ea..d110881 100644
--- a/meta/lib/oeqa/core/loader.py
+++ b/meta/lib/oeqa/core/loader.py
@@ -164,8 +164,11 @@ class OETestLoader(unittest.TestLoader):
         """
             Returns True if test case must be filtered, False otherwise.
         """
-        # Filters by module.class.name
-        module_name = case.__module__
+        # XXX; If the module has more than one namespace only use
+        # the first to support run the whole module specifying the
+        # <module_name>.[test_class].[test_name]
+        module_name = case.__module__.split('.')[0]
+
         class_name = case.__class__.__name__
         test_name = case._testMethodName
 
@@ -280,6 +283,33 @@ class OETestLoader(unittest.TestLoader):
 
         return self.suiteClass(cases) if cases else big_suite
 
+    def _filterModule(self, module):
+        if module.__name__ in sys.builtin_module_names:
+            msg = 'Tried to import %s test module but is a built-in'
+            raise ImportError(msg % module.__name__)
+
+        # XXX; If the module has more than one namespace only use
+        # the first to support run the whole module specifying the
+        # <module_name>.[test_class].[test_name]
+        module_name = module.__name__.split('.')[0]
+
+        # Normal test modules are loaded if no modules were specified,
+        # if module is in the specified module list or if 'all' is in
+        # module list.
+        # Underscore modules are loaded only if specified in module list.
+        load_module = True if not module_name.startswith('_') \
+                              and (not self.modules \
+                                   or module_name in self.modules \
+                                   or 'all' in self.modules) \
+                           else False
+
+        load_underscore = True if module_name.startswith('_') \
+                                  and module_name in self.modules \
+                               else False
+
+        return (load_module, load_underscore)
+
+
     # XXX After Python 3.5, remove backward compatibility hacks for
     # use_load_tests deprecation via *args and **kws.  See issue 16662.
     if sys.version_info >= (3,5):
@@ -287,23 +317,7 @@ class OETestLoader(unittest.TestLoader):
             """
                 Returns a suite of all tests cases contained in module.
             """
-            if module.__name__ in sys.builtin_module_names:
-                msg = 'Tried to import %s test module but is a built-in'
-                raise ImportError(msg % module.__name__)
-
-            # Normal test modules are loaded if no modules were specified,
-            # if module is in the specified module list or if 'all' is in
-            # module list.
-            # Underscore modules are loaded only if specified in module list.
-            load_module = True if not module.__name__.startswith('_') \
-                                  and (not self.modules \
-                                       or module.__name__ in self.modules \
-                                       or 'all' in self.modules) \
-                               else False
-
-            load_underscore = True if module.__name__.startswith('_') \
-                                      and module.__name__ in self.modules \
-                                   else False
+            load_module, load_underscore = self._filterModule(module)
 
             if load_module or load_underscore:
                 return super(OETestLoader, self).loadTestsFromModule(
@@ -315,23 +329,7 @@ class OETestLoader(unittest.TestLoader):
             """
                 Returns a suite of all tests cases contained in module.
             """
-            if module.__name__ in sys.builtin_module_names:
-                msg = 'Tried to import %s test module but is a built-in'
-                raise ImportError(msg % module.__name__)
-
-            # Normal test modules are loaded if no modules were specified,
-            # if module is in the specified module list or if 'all' is in
-            # module list.
-            # Underscore modules are loaded only if specified in module list.
-            load_module = True if not module.__name__.startswith('_') \
-                                  and (not self.modules \
-                                       or module.__name__ in self.modules \
-                                       or 'all' in self.modules) \
-                               else False
-
-            load_underscore = True if module.__name__.startswith('_') \
-                                      and module.__name__ in self.modules \
-                                   else False
+            load_module, load_underscore = self._filterModule(module)
 
             if load_module or load_underscore:
                 return super(OETestLoader, self).loadTestsFromModule(
-- 
2.1.4