[PATCH 00/17] OEQA framework add support of threaded mode and enable in SDK

[PATCH 00/17] OEQA framework add support of threaded mode and enable in SDK

[Aníbal Limón]

This series enables support of OEQA framework threaded runs, it implements a new module
oeqa.core.threaded with OETest{Context, Loader, Runner, StreamLogger, Result} versions
supporting parallelized runs.

The {e,}SDK test component was enable to use threaded mode by default.

The following changes since commit 381897c64069ea43d595380a3ae913bcc79cf7e1:

  build-appliance-image: Update to master head revision (2017-05-01 08:56:47 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib alimon/oeqa_threaded
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/oeqa_threaded

Aníbal Limón (17):
  testsdk.bbclass: Remove unused import of OEStreamLogger
  oeqa/sdk/context.py: Import argparse_oe at OESDKTestContext.run method
  oeqa/sdk/context.py: Add return to OESDKTestContext.run() method
  oeqa/core: Don't expose OEStreamLogger in OETestContext
  oeqa/core/runner: OETestResult remove unneeded override of startTest
  oeqa/core: Move OETestContext.log{Summary, Details} into OETestResult
  oeqa/core/threaded: Add new module with OETestLoaderThreaded
  oeqa/core/threaded: Add OEStreamLoggerThreaded class
  oeqa/core/runner: OETestResult add internal _tc_map_results
  oeqa/core/threaded: Add OETestResultThreaded{,Internal} classes
  oeqa/core/threaded: Add support of OETestRunnerThreaded
  oeqa/core/threaded: Add OETestContextThreaded class
  oeqa/core/decorator/depends: Add support for threading mode
  oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode
  oeqa/core/tests: Add tests of OEQA Threaded mode
  oeqa/sdkext/cases: Move sdk_update test into devtool module
  oeqa/sdk: Enable usage of OEQA thread mode

 meta/classes/testimage.bbclass                     |   4 +-
 meta/classes/testsdk.bbclass                       |  13 +-
 meta/lib/oeqa/core/context.py                      |  97 +-------
 meta/lib/oeqa/core/decorator/depends.py            |   8 +-
 meta/lib/oeqa/core/decorator/oetimeout.py          |  40 ++-
 meta/lib/oeqa/core/runner.py                       |  94 ++++++-
 .../core/tests/cases/loader/threaded/threaded.py   |  12 +
 .../tests/cases/loader/threaded/threaded_alone.py  |   8 +
 .../cases/loader/threaded/threaded_depends.py      |  10 +
 .../tests/cases/loader/threaded/threaded_module.py |  12 +
 meta/lib/oeqa/core/tests/common.py                 |  10 +
 meta/lib/oeqa/core/tests/test_decorators.py        |  12 +
 meta/lib/oeqa/core/tests/test_loader.py            |  30 ++-
 meta/lib/oeqa/core/threaded.py                     | 274 +++++++++++++++++++++
 meta/lib/oeqa/sdk/context.py                       |  11 +-
 meta/lib/oeqa/sdkext/cases/devtool.py              |  32 +++
 meta/lib/oeqa/sdkext/cases/sdk_update.py           |  39 ---
 17 files changed, 548 insertions(+), 158 deletions(-)
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded.py
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_alone.py
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_depends.py
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_module.py
 create mode 100644 meta/lib/oeqa/core/threaded.py
 delete mode 100644 meta/lib/oeqa/sdkext/cases/sdk_update.py

-- 
2.1.4

[PATCH 01/17] testsdk.bbclass: Remove unused import of OEStreamLogger

[Aníbal Limón]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/classes/testsdk.bbclass | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 6a201aa..4740233 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -21,7 +21,6 @@ def testsdk_main(d):
     import logging
 
     from bb.utils import export_proxies
-    from oeqa.core.runner import OEStreamLogger
     from oeqa.sdk.context import OESDKTestContext, OESDKTestContextExecutor
     from oeqa.utils import make_logger_bitbake_compatible
 
-- 
2.1.4

[PATCH 02/17] oeqa/sdk/context.py: Import argparse_oe at OESDKTestContext.run method

[Aníbal Limón]

This import was at level of OESDKTestContext.register_commands
but OESDKTestContext.run method need it to raise exceptions.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/sdk/context.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/sdk/context.py b/meta/lib/oeqa/sdk/context.py
index 0189ed8..509d2b4 100644
--- a/meta/lib/oeqa/sdk/context.py
+++ b/meta/lib/oeqa/sdk/context.py
@@ -44,8 +44,6 @@ class OESDKTestContextExecutor(OETestContextExecutor):
     default_test_data = None
 
     def register_commands(self, logger, subparsers):
-        import argparse_oe
-
         super(OESDKTestContextExecutor, self).register_commands(logger, subparsers)
 
         sdk_group = self.parser.add_argument_group('sdk options')
@@ -109,6 +107,8 @@ class OESDKTestContextExecutor(OETestContextExecutor):
             log(env)
 
     def run(self, logger, args):
+        import argparse_oe
+
         if not args.sdk_dir:
             raise argparse_oe.ArgumentUsageError("No SDK directory "\
                    "specified please do, --sdk-dir SDK_DIR", self.name)
-- 
2.1.4

[PATCH 03/17] oeqa/sdk/context.py: Add return to OESDKTestContext.run() method

[Aníbal Limón]

The run() methods of a OETestContext's are expected to return the
results.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/sdk/context.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/sdk/context.py b/meta/lib/oeqa/sdk/context.py
index 509d2b4..82e4c19 100644
--- a/meta/lib/oeqa/sdk/context.py
+++ b/meta/lib/oeqa/sdk/context.py
@@ -128,6 +128,6 @@ class OESDKTestContextExecutor(OETestContextExecutor):
                    "environment (%s) specified" % args.sdk_env, self.name)
 
         self.sdk_env = sdk_envs[args.sdk_env]
-        super(OESDKTestContextExecutor, self).run(logger, args)
+        return super(OESDKTestContextExecutor, self).run(logger, args)
 
 _executor_class = OESDKTestContextExecutor
-- 
2.1.4

[PATCH 04/17] oeqa/core: Don't expose OEStreamLogger in OETestContext

[Aníbal Limón]

The OEStreamLogger class is used for redirect PyUnit output
to a certain logger so there is not need to expose at level
of OETestContext because only OETestRunner needs to know.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/context.py | 6 ++----
 meta/lib/oeqa/core/runner.py  | 3 +++
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/core/context.py b/meta/lib/oeqa/core/context.py
index 4476750..2628651 100644
--- a/meta/lib/oeqa/core/context.py
+++ b/meta/lib/oeqa/core/context.py
@@ -10,12 +10,11 @@ import collections
 import re
 
 from oeqa.core.loader import OETestLoader
-from oeqa.core.runner import OETestRunner, OEStreamLogger, xmlEnabled
+from oeqa.core.runner import OETestRunner, xmlEnabled
 
 class OETestContext(object):
     loaderClass = OETestLoader
     runnerClass = OETestRunner
-    streamLoggerClass = OEStreamLogger
 
     files_dir = os.path.abspath(os.path.join(os.path.dirname(
         os.path.abspath(__file__)), "../files"))
@@ -52,8 +51,7 @@ class OETestContext(object):
         self.suites = self.loader.discover()
 
     def runTests(self):
-        streamLogger = self.streamLoggerClass(self.logger)
-        self.runner = self.runnerClass(self, stream=streamLogger, verbosity=2)
+        self.runner = self.runnerClass(self, verbosity=2)
 
         self._run_start_time = time.time()
         result = self.runner.run(self.suites)
diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
index 44ffecb..40fbf3b 100644
--- a/meta/lib/oeqa/core/runner.py
+++ b/meta/lib/oeqa/core/runner.py
@@ -48,12 +48,15 @@ class OETestResult(_TestResult):
         super(OETestResult, self).startTest(test)
 
 class OETestRunner(_TestRunner):
+    streamLoggerClass = OEStreamLogger
+
     def __init__(self, tc, *args, **kwargs):
         if xmlEnabled:
             if not kwargs.get('output'):
                 kwargs['output'] = os.path.join(os.getcwd(),
                         'TestResults_%s_%s' % (time.strftime("%Y%m%d%H%M%S"), os.getpid()))
 
+        kwargs['stream'] = self.streamLoggerClass(tc.logger)
         super(OETestRunner, self).__init__(*args, **kwargs)
         self.tc = tc
         self.resultclass = OETestResult
-- 
2.1.4

[PATCH 05/17] oeqa/core/runner: OETestResult remove unneeded override of startTest

[Aníbal Limón]

I override this method before for keep track of results and forget
to remove it, now isn't need.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/runner.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
index 40fbf3b..0b04e8d 100644
--- a/meta/lib/oeqa/core/runner.py
+++ b/meta/lib/oeqa/core/runner.py
@@ -44,9 +44,6 @@ class OETestResult(_TestResult):
         self.tc._results['skipped'] = self.skipped
         self.tc._results['expectedFailures'] = self.expectedFailures
 
-    def startTest(self, test):
-        super(OETestResult, self).startTest(test)
-
 class OETestRunner(_TestRunner):
     streamLoggerClass = OEStreamLogger
 
-- 
2.1.4

[PATCH 06/17] oeqa/core: Move OETestContext.log{Summary, Details} into OETestResult

[Aníbal Limón]

Those methods are used to write in the log the results so
it makes sense to have defined at OETestResult because
is a format of the result itself.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/classes/testimage.bbclass |  4 +-
 meta/classes/testsdk.bbclass   |  8 ++--
 meta/lib/oeqa/core/context.py  | 93 ++----------------------------------------
 meta/lib/oeqa/core/runner.py   | 88 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 97 insertions(+), 96 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index fb21460..d42907c 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -292,8 +292,8 @@ def testimage_main(d):
     # Show results (if we have them)
     if not results:
         bb.fatal('%s - FAILED - tests were interrupted during execution' % pn)
-    tc.logSummary(results, pn)
-    tc.logDetails()
+    results.logSummary(pn)
+    results.logDetails()
     if not results.wasSuccessful():
         bb.fatal('%s - FAILED - check the task log and the ssh log' % pn)
 
diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index 4740233..edea89c 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -70,8 +70,8 @@ def testsdk_main(d):
         component = "%s %s" % (pn, OESDKTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
 
-        tc.logSummary(result, component, context_msg)
-        tc.logDetails()
+        result.logSummary(component, context_msg)
+        result.logDetails()
 
         if not result.wasSuccessful():
             fail = True
@@ -172,8 +172,8 @@ def testsdkext_main(d):
         component = "%s %s" % (pn, OESDKExtTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
 
-        tc.logSummary(result, component, context_msg)
-        tc.logDetails()
+        result.logSummary(component, context_msg)
+        result.logDetails()
 
         if not result.wasSuccessful():
             fail = True
diff --git a/meta/lib/oeqa/core/context.py b/meta/lib/oeqa/core/context.py
index 2628651..54958ad 100644
--- a/meta/lib/oeqa/core/context.py
+++ b/meta/lib/oeqa/core/context.py
@@ -7,10 +7,9 @@ import json
 import time
 import logging
 import collections
-import re
 
 from oeqa.core.loader import OETestLoader
-from oeqa.core.runner import OETestRunner, xmlEnabled
+from oeqa.core.runner import OETestRunner
 
 class OETestContext(object):
     loaderClass = OETestLoader
@@ -59,92 +58,6 @@ class OETestContext(object):
 
         return result
 
-    def logSummary(self, result, component, context_msg=''):
-        self.logger.info("SUMMARY:")
-        self.logger.info("%s (%s) - Ran %d test%s in %.3fs" % (component,
-            context_msg, result.testsRun, result.testsRun != 1 and "s" or "",
-            (self._run_end_time - self._run_start_time)))
-
-        if result.wasSuccessful():
-            msg = "%s - OK - All required tests passed" % component
-        else:
-            msg = "%s - FAIL - Required tests failed" % component
-        skipped = len(self._results['skipped'])
-        if skipped: 
-            msg += " (skipped=%d)" % skipped
-        self.logger.info(msg)
-
-    def _getDetailsNotPassed(self, case, type, desc):
-        found = False
-
-        for (scase, msg) in self._results[type]:
-            # XXX: When XML reporting is enabled scase is
-            # xmlrunner.result._TestInfo instance instead of
-            # string.
-            if xmlEnabled:
-                if case.id() == scase.test_id:
-                    found = True
-                    break
-                scase_str = scase.test_id
-            else:
-                if case == scase:
-                    found = True
-                    break
-                scase_str = str(scase)
-
-            # When fails at module or class level the class name is passed as string
-            # so figure out to see if match
-            m = re.search("^setUpModule \((?P<module_name>.*)\)$", scase_str)
-            if m:
-                if case.__class__.__module__ == m.group('module_name'):
-                    found = True
-                    break
-
-            m = re.search("^setUpClass \((?P<class_name>.*)\)$", scase_str)
-            if m:
-                class_name = "%s.%s" % (case.__class__.__module__,
-                        case.__class__.__name__)
-
-                if class_name == m.group('class_name'):
-                    found = True
-                    break
-
-        if found:
-            return (found, msg)
-
-        return (found, None)
-
-    def logDetails(self):
-        self.logger.info("RESULTS:")
-        for case_name in self._registry['cases']:
-            case = self._registry['cases'][case_name]
-
-            result_types = ['failures', 'errors', 'skipped', 'expectedFailures']
-            result_desc = ['FAILED', 'ERROR', 'SKIPPED', 'EXPECTEDFAIL']
-
-            fail = False
-            desc = None
-            for idx, name in enumerate(result_types):
-                (fail, msg) = self._getDetailsNotPassed(case, result_types[idx],
-                        result_desc[idx])
-                if fail:
-                    desc = result_desc[idx]
-                    break
-
-            oeid = -1
-            for d in case.decorators:
-                if hasattr(d, 'oeid'):
-                    oeid = d.oeid
-            
-            if fail:
-                self.logger.info("RESULTS - %s - Testcase %s: %s" % (case.id(),
-                    oeid, desc))
-                if msg:
-                    self.logger.info(msg)
-            else:
-                self.logger.info("RESULTS - %s - Testcase %s: %s" % (case.id(),
-                    oeid, 'PASSED'))
-
 class OETestContextExecutor(object):
     _context_class = OETestContext
 
@@ -227,8 +140,8 @@ class OETestContextExecutor(object):
         self.tc = self._context_class(**self.tc_kwargs['init'])
         self.tc.loadTests(self.module_paths, **self.tc_kwargs['load'])
         rc = self.tc.runTests(**self.tc_kwargs['run'])
-        self.tc.logSummary(rc, self.name)
-        self.tc.logDetails()
+        rc.logSummary(self.name)
+        rc.logDetails()
 
         output_link = os.path.join(os.path.dirname(args.output_log),
                 "%s-results.log" % self.name)
diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
index 0b04e8d..7505838 100644
--- a/meta/lib/oeqa/core/runner.py
+++ b/meta/lib/oeqa/core/runner.py
@@ -5,6 +5,7 @@ import os
 import time
 import unittest
 import logging
+import re
 
 xmlEnabled = False
 try:
@@ -44,6 +45,93 @@ class OETestResult(_TestResult):
         self.tc._results['skipped'] = self.skipped
         self.tc._results['expectedFailures'] = self.expectedFailures
 
+    def logSummary(self, component, context_msg=''):
+        elapsed_time = self.tc._run_end_time - self.tc._run_start_time
+        self.tc.logger.info("SUMMARY:")
+        self.tc.logger.info("%s (%s) - Ran %d test%s in %.3fs" % (component,
+            context_msg, self.testsRun, self.testsRun != 1 and "s" or "",
+            elapsed_time))
+
+        if self.wasSuccessful():
+            msg = "%s - OK - All required tests passed" % component
+        else:
+            msg = "%s - FAIL - Required tests failed" % component
+        skipped = len(self.tc._results['skipped'])
+        if skipped: 
+            msg += " (skipped=%d)" % skipped
+        self.tc.logger.info(msg)
+
+    def _getDetailsNotPassed(self, case, type, desc):
+        found = False
+
+        for (scase, msg) in self.tc._results[type]:
+            # XXX: When XML reporting is enabled scase is
+            # xmlrunner.result._TestInfo instance instead of
+            # string.
+            if xmlEnabled:
+                if case.id() == scase.test_id:
+                    found = True
+                    break
+                scase_str = scase.test_id
+            else:
+                if case == scase:
+                    found = True
+                    break
+                scase_str = str(scase)
+
+            # When fails at module or class level the class name is passed as string
+            # so figure out to see if match
+            m = re.search("^setUpModule \((?P<module_name>.*)\)$", scase_str)
+            if m:
+                if case.__class__.__module__ == m.group('module_name'):
+                    found = True
+                    break
+
+            m = re.search("^setUpClass \((?P<class_name>.*)\)$", scase_str)
+            if m:
+                class_name = "%s.%s" % (case.__class__.__module__,
+                        case.__class__.__name__)
+
+                if class_name == m.group('class_name'):
+                    found = True
+                    break
+
+        if found:
+            return (found, msg)
+
+        return (found, None)
+
+    def logDetails(self):
+        self.tc.logger.info("RESULTS:")
+        for case_name in self.tc._registry['cases']:
+            case = self.tc._registry['cases'][case_name]
+
+            result_types = ['failures', 'errors', 'skipped', 'expectedFailures']
+            result_desc = ['FAILED', 'ERROR', 'SKIPPED', 'EXPECTEDFAIL']
+
+            fail = False
+            desc = None
+            for idx, name in enumerate(result_types):
+                (fail, msg) = self._getDetailsNotPassed(case, result_types[idx],
+                        result_desc[idx])
+                if fail:
+                    desc = result_desc[idx]
+                    break
+
+            oeid = -1
+            for d in case.decorators:
+                if hasattr(d, 'oeid'):
+                    oeid = d.oeid
+
+            if fail:
+                self.tc.logger.info("RESULTS - %s - Testcase %s: %s" % (case.id(),
+                    oeid, desc))
+                if msg:
+                    self.tc.logger.info(msg)
+            else:
+                self.tc.logger.info("RESULTS - %s - Testcase %s: %s" % (case.id(),
+                    oeid, 'PASSED'))
+
 class OETestRunner(_TestRunner):
     streamLoggerClass = OEStreamLogger
 
-- 
2.1.4

[PATCH 07/17] oeqa/core/threaded: Add new module with OETestLoaderThreaded

[Aníbal Limón]

The threded module will implement Thread support into the OEQA
framework.

The OETestLoaderThreaded overrides discover to return a list of
suites every suite will be run by a separate Thread.

Some design considerations are:

- All the tests of a certain module needs to be run at one thread
  because unittest framework needs to keep track of setUp{Module,
Class,} and tearDown{Module, Class,}.

- Tests that depends on other needs to be run at the same thread
because OEQA framework look at the status of dependant test to
decide if skip or not, this constraint can be change in the future
but a sync mechanishm is needed to implement between the Threads.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/threaded.py | 91 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
 create mode 100644 meta/lib/oeqa/core/threaded.py

diff --git a/meta/lib/oeqa/core/threaded.py b/meta/lib/oeqa/core/threaded.py
new file mode 100644
index 0000000..e6f2140
--- /dev/null
+++ b/meta/lib/oeqa/core/threaded.py
@@ -0,0 +1,91 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+import multiprocessing
+
+from unittest.suite import TestSuite
+from oeqa.core.loader import OETestLoader
+
+class OETestLoaderThreaded(OETestLoader):
+    def __init__(self, tc, module_paths, modules, tests, modules_required,
+            filters, process_num=0, *args, **kwargs):
+        super(OETestLoaderThreaded, self).__init__(tc, module_paths, modules,
+                tests, modules_required, filters, *args, **kwargs)
+
+        self.process_num = process_num
+
+    def discover(self):
+        suite = super(OETestLoaderThreaded, self).discover()
+
+        if self.process_num <= 0:
+            self.process_num = min(multiprocessing.cpu_count(),
+                    len(suite._tests))
+
+        suites = []
+        for _ in range(self.process_num):
+            suites.append(self.suiteClass())
+
+        def _search_for_module_idx(suites, case):
+            """
+                Cases in the same module needs to be run
+                in the same thread because PyUnit keeps track
+                of setUp{Module, Class,} and tearDown{Module, Class,}.
+            """
+
+            for idx in range(self.process_num):
+                suite = suites[idx]
+                for c in suite._tests:
+                    if case.__module__ == c.__module__:
+                        return idx
+
+            return -1
+
+        def _search_for_depend_idx(suites, depends):
+            """
+                Dependency cases needs to be run in the same
+                thread, because OEQA framework look at the state
+                of dependant test to figure out if skip or not.
+            """
+
+            for idx in range(self.process_num):
+                suite = suites[idx]
+
+                for case in suite._tests:
+                    if case.id() in depends:
+                        return idx
+            return -1
+
+        def _get_best_idx(suites):
+            sizes = [len(suite._tests) for suite in suites]
+            return sizes.index(min(sizes))
+
+        def _fill_suites(suite):
+            idx = -1
+            for case in suite:
+                if isinstance(case, TestSuite):
+                    _fill_suites(case)
+                else:
+                    idx = _search_for_module_idx(suites, case)
+
+                    depends = {}
+                    if 'depends' in self.tc._registry:
+                        depends = self.tc._registry['depends']
+
+                    if idx == -1 and case.id() in depends:
+                        case_depends = depends[case.id()] 
+                        idx = _search_for_depend_idx(suites, case_depends)
+
+                    if idx == -1:
+                        idx = _get_best_idx(suites)
+
+                    suites[idx].addTest(case)
+        _fill_suites(suite)
+
+        suites_tmp = suites
+        suites = []
+        for suite in suites_tmp:
+            if len(suite._tests) > 0:
+                suites.append(suite)
+
+        return suites
+
-- 
2.1.4

[PATCH 08/17] oeqa/core/threaded: Add OEStreamLoggerThreaded class

[Aníbal Limón]

The OEStreamLoggerThreaded overrides OEStreamLogger to redirect
the PyUnit output to a logger.

Instead of log every line when comes the OEStreamLoggerThreaded
will buffer the PyUnit output and write everything at end of every
suite execution to don't have mixed suite outputs.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/threaded.py | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/meta/lib/oeqa/core/threaded.py b/meta/lib/oeqa/core/threaded.py
index e6f2140..73b7f2f 100644
--- a/meta/lib/oeqa/core/threaded.py
+++ b/meta/lib/oeqa/core/threaded.py
@@ -1,10 +1,13 @@
 # Copyright (C) 2017 Intel Corporation
 # Released under the MIT license (see COPYING.MIT)
 
+import threading
 import multiprocessing
 
 from unittest.suite import TestSuite
+
 from oeqa.core.loader import OETestLoader
+from oeqa.core.runner import OEStreamLogger
 
 class OETestLoaderThreaded(OETestLoader):
     def __init__(self, tc, module_paths, modules, tests, modules_required,
@@ -89,3 +92,25 @@ class OETestLoaderThreaded(OETestLoader):
 
         return suites
 
+class OEStreamLoggerThreaded(OEStreamLogger):
+    _lock = threading.Lock()
+    buffers = {}
+
+    def write(self, msg):
+        tid = threading.get_ident()
+
+        if not tid in self.buffers:
+            self.buffers[tid] = ""
+
+        if msg:
+            self.buffers[tid] += msg
+
+    def finish(self):
+        tid = threading.get_ident()
+        
+        self._lock.acquire()
+        self.logger.info('THREAD: %d' % tid)
+        self.logger.info('-' * 70)
+        for line in self.buffers[tid].split('\n'):
+            self.logger.info(line)
+        self._lock.release()
-- 
2.1.4

[PATCH 09/17] oeqa/core/runner: OETestResult add internal _tc_map_results

[Aníbal Limón]

This method is to assign results into the TestContext, create
an internal one to support change implementation in Thread version.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/runner.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
index 7505838..3ebffc7 100644
--- a/meta/lib/oeqa/core/runner.py
+++ b/meta/lib/oeqa/core/runner.py
@@ -39,7 +39,9 @@ class OETestResult(_TestResult):
         super(OETestResult, self).__init__(*args, **kwargs)
 
         self.tc = tc
+        self._tc_map_results()
 
+    def _tc_map_results(self):
         self.tc._results['failures'] = self.failures
         self.tc._results['errors'] = self.errors
         self.tc._results['skipped'] = self.skipped
-- 
2.1.4

[PATCH 10/17] oeqa/core/threaded: Add OETestResultThreaded{, Internal} classes

[Aníbal Limón]

The OETestResultThreadedInternal extends OETestResult to stores
results by Thread.

The OETestResultThreaded is a simple class that provides the
implementation of interfaces needed by outside like wasSuccesful,
stop, logSummary, logDetails.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/threaded.py | 73 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/core/threaded.py b/meta/lib/oeqa/core/threaded.py
index 73b7f2f..f216685 100644
--- a/meta/lib/oeqa/core/threaded.py
+++ b/meta/lib/oeqa/core/threaded.py
@@ -7,7 +7,7 @@ import multiprocessing
 from unittest.suite import TestSuite
 
 from oeqa.core.loader import OETestLoader
-from oeqa.core.runner import OEStreamLogger
+from oeqa.core.runner import OEStreamLogger, OETestResult
 
 class OETestLoaderThreaded(OETestLoader):
     def __init__(self, tc, module_paths, modules, tests, modules_required,
@@ -114,3 +114,74 @@ class OEStreamLoggerThreaded(OEStreamLogger):
         for line in self.buffers[tid].split('\n'):
             self.logger.info(line)
         self._lock.release()
+
+class OETestResultThreadedInternal(OETestResult):
+    def _tc_map_results(self):
+        tid = threading.get_ident()
+        
+        # PyUnit generates a result for every test module run, test
+        # if the thread already has an entry to avoid lose the previous
+        # test module results.
+        if not tid in self.tc._results:
+            self.tc._results[tid] = {}
+            self.tc._results[tid]['failures'] = self.failures
+            self.tc._results[tid]['errors'] = self.errors
+            self.tc._results[tid]['skipped'] = self.skipped
+            self.tc._results[tid]['expectedFailures'] = self.expectedFailures
+
+class OETestResultThreaded(object):
+    _results = {}
+    _lock = threading.Lock()
+
+    def __init__(self, tc):
+        self.tc = tc
+
+    def _fill_tc_results(self):
+        tids = list(self.tc._results.keys())
+        fields = ['failures', 'errors', 'skipped', 'expectedFailures']
+
+        for tid in tids:
+            result = self.tc._results[tid]
+            for field in fields:
+                if not field in self.tc._results:
+                    self.tc._results[field] = []
+                self.tc._results[field].extend(result[field])
+
+    def addResult(self, result, run_start_time, run_end_time):
+        tid = threading.get_ident()
+
+        self._lock.acquire()
+        self._results[tid] = {}
+        self._results[tid]['result'] = result
+        self._results[tid]['run_start_time'] = run_start_time 
+        self._results[tid]['run_end_time'] = run_end_time 
+        self._results[tid]['result'] = result
+        self._lock.release()
+
+    def wasSuccessful(self):
+        wasSuccessful = True
+        for tid in self._results.keys():
+            wasSuccessful = wasSuccessful and \
+                    self._results[tid]['result'].wasSuccessful()
+        return wasSuccessful
+
+    def stop(self):
+        for tid in self._results.keys():
+            self._results[tid]['result'].stop()
+
+    def logSummary(self, component, context_msg=''):
+        elapsed_time = (self.tc._run_end_time - self.tc._run_start_time)
+
+        self.tc.logger.info("SUMMARY:")
+        self.tc.logger.info("%s (%s) - Ran %d tests in %.3fs" % (component,
+            context_msg, len(self.tc._registry['cases']), elapsed_time))
+        if self.wasSuccessful():
+            msg = "%s - OK - All required tests passed" % component
+        else:
+            msg = "%s - FAIL - Required tests failed" % component
+        self.tc.logger.info(msg)
+
+    def logDetails(self):
+        tid = list(self._results)[0]
+        result = self._results[tid]['result']
+        result.logDetails()
-- 
2.1.4

[PATCH 11/17] oeqa/core/threaded: Add support of OETestRunnerThreaded

[Aníbal Limón]

The OETestRunnerThreaded overrides the run method of OETestRunner
it recieves a list of suites to be executed by a ThreadPool.

The new run method handles the ThreadPool creation and the
OETestResultThreaded fill.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/threaded.py | 75 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 74 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/core/threaded.py b/meta/lib/oeqa/core/threaded.py
index f216685..81df340 100644
--- a/meta/lib/oeqa/core/threaded.py
+++ b/meta/lib/oeqa/core/threaded.py
@@ -3,11 +3,13 @@
 
 import threading
 import multiprocessing
+import queue
+import time
 
 from unittest.suite import TestSuite
 
 from oeqa.core.loader import OETestLoader
-from oeqa.core.runner import OEStreamLogger, OETestResult
+from oeqa.core.runner import OEStreamLogger, OETestResult, OETestRunner
 
 class OETestLoaderThreaded(OETestLoader):
     def __init__(self, tc, module_paths, modules, tests, modules_required,
@@ -185,3 +187,74 @@ class OETestResultThreaded(object):
         tid = list(self._results)[0]
         result = self._results[tid]['result']
         result.logDetails()
+
+class _Worker(threading.Thread):
+    """Thread executing tasks from a given tasks queue"""
+    def __init__(self, tasks, result, stream):
+        threading.Thread.__init__(self)
+        self.tasks = tasks
+
+        self.result = result
+        self.stream = stream
+
+    def run(self):
+        while True:
+            try:
+                func, args, kargs = self.tasks.get(block=False)
+            except queue.Empty:
+                break
+
+            try:
+                run_start_time = time.time()
+                rc = func(*args, **kargs)
+                run_end_time = time.time()
+                self.result.addResult(rc, run_start_time, run_end_time)
+                self.stream.finish()
+            except Exception as e:
+                print(e)
+            finally:
+                self.tasks.task_done()
+
+class _ThreadedPool:
+    """Pool of threads consuming tasks from a queue"""
+    def __init__(self, num_workers, num_tasks, stream=None, result=None):
+        self.tasks = queue.Queue(num_tasks)
+        self.workers = []
+
+        for _ in range(num_workers):
+            worker = _Worker(self.tasks, result, stream)
+            self.workers.append(worker)
+
+    def start(self):
+        for worker in self.workers:
+            worker.start()
+
+    def add_task(self, func, *args, **kargs):
+        """Add a task to the queue"""
+        self.tasks.put((func, args, kargs))
+
+    def wait_completion(self):
+        """Wait for completion of all the tasks in the queue"""
+        self.tasks.join()
+        for worker in self.workers:
+            worker.join()
+
+class OETestRunnerThreaded(OETestRunner):
+    streamLoggerClass = OEStreamLoggerThreaded
+
+    def __init__(self, tc, *args, **kwargs):
+        super(OETestRunnerThreaded, self).__init__(tc, *args, **kwargs)
+        self.resultclass = OETestResultThreadedInternal # XXX: XML reporting overrides at __init__
+
+    def run(self, suites):
+        result = OETestResultThreaded(self.tc)
+
+        pool = _ThreadedPool(len(suites), len(suites), stream=self.stream,
+                result=result)
+        for s in suites:
+            pool.add_task(super(OETestRunnerThreaded, self).run, s)
+        pool.start()
+        pool.wait_completion()
+        result._fill_tc_results()
+
+        return result
-- 
2.1.4

[PATCH 12/17] oeqa/core/threaded: Add OETestContextThreaded class

[Aníbal Limón]

This class sets the {loader,runner}Class to the threaded versions and
overrides loadTests method to be able specify the process_num.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/threaded.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/lib/oeqa/core/threaded.py b/meta/lib/oeqa/core/threaded.py
index 81df340..c6058a3 100644
--- a/meta/lib/oeqa/core/threaded.py
+++ b/meta/lib/oeqa/core/threaded.py
@@ -10,6 +10,7 @@ from unittest.suite import TestSuite
 
 from oeqa.core.loader import OETestLoader
 from oeqa.core.runner import OEStreamLogger, OETestResult, OETestRunner
+from oeqa.core.context import OETestContext
 
 class OETestLoaderThreaded(OETestLoader):
     def __init__(self, tc, module_paths, modules, tests, modules_required,
@@ -258,3 +259,16 @@ class OETestRunnerThreaded(OETestRunner):
         result._fill_tc_results()
 
         return result
+
+class OETestContextThreaded(OETestContext):
+    loaderClass = OETestLoaderThreaded
+    runnerClass = OETestRunnerThreaded
+
+    def loadTests(self, module_paths, modules=[], tests=[],
+            modules_manifest="", modules_required=[], filters={}, process_num=0):
+        if modules_manifest:
+            modules = self._read_modules_from_manifest(modules_manifest)
+
+        self.loader = self.loaderClass(self, module_paths, modules, tests,
+                modules_required, filters, process_num)
+        self.suites = self.loader.discover()
-- 
2.1.4

[PATCH 13/17] oeqa/core/decorator/depends: Add support for threading mode

[Aníbal Limón]

The _skipTestDependency needs to know if the thread mode is
enabled because the _results are by thread.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/decorator/depends.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/core/decorator/depends.py b/meta/lib/oeqa/core/decorator/depends.py
index 195711c..baa0434 100644
--- a/meta/lib/oeqa/core/decorator/depends.py
+++ b/meta/lib/oeqa/core/decorator/depends.py
@@ -3,6 +3,7 @@
 
 from unittest import SkipTest
 
+from oeqa.core.threaded import OETestRunnerThreaded
 from oeqa.core.exception import OEQADependency
 
 from . import OETestDiscover, registerDecorator
@@ -63,7 +64,12 @@ def _order_test_case_by_depends(cases, depends):
     return [cases[case_id] for case_id in cases_ordered]
 
 def _skipTestDependency(case, depends):
-    results = case.tc._results
+    if isinstance(case.tc.runner, OETestRunnerThreaded):
+        import threading
+        results = case.tc._results[threading.get_ident()]
+    else:
+        results = case.tc._results
+
     skipReasons = ['errors', 'failures', 'skipped']
 
     for reason in skipReasons:
-- 
2.1.4

[PATCH 14/17] oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode

[Aníbal Limón]

In python signals are only allowed to delivery into the main thread,
to support the threading mode test if the runner is threaded and
use threading.Timer instead.

There are some considerations like SIGALRM interrupts the execution
after N seconds but the Timer only starts a Thread to notice the
timeout and the exception will be raised when the test run ends.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/core/decorator/oetimeout.py | 40 +++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 10 deletions(-)

diff --git a/meta/lib/oeqa/core/decorator/oetimeout.py b/meta/lib/oeqa/core/decorator/oetimeout.py
index a247583..f85e7d9 100644
--- a/meta/lib/oeqa/core/decorator/oetimeout.py
+++ b/meta/lib/oeqa/core/decorator/oetimeout.py
@@ -1,8 +1,12 @@
 # Copyright (C) 2016 Intel Corporation
 # Released under the MIT license (see COPYING.MIT)
 
-import signal
 from . import OETestDecorator, registerDecorator
+
+import signal
+from threading import Timer
+
+from oeqa.core.threaded import OETestRunnerThreaded
 from oeqa.core.exception import OEQATimeoutError
 
 @registerDecorator
@@ -10,16 +14,32 @@ class OETimeout(OETestDecorator):
     attrs = ('oetimeout',)
 
     def setUpDecorator(self):
-        timeout = self.oetimeout
-        def _timeoutHandler(signum, frame):
-            raise OEQATimeoutError("Timed out after %s "
+        self.logger.debug("Setting up a %d second(s) timeout" % self.oetimeout)
+
+        if isinstance(self.case.tc.runner, OETestRunnerThreaded):
+            self.timeouted = False
+            def _timeoutHandler():
+                self.timeouted = True
+
+            self.timer = Timer(self.oetimeout, _timeoutHandler)
+            self.timer.start()
+        else:
+            timeout = self.oetimeout
+            def _timeoutHandler(signum, frame):
+                raise OEQATimeoutError("Timed out after %s "
                     "seconds of execution" % timeout)
 
-        self.logger.debug("Setting up a %d second(s) timeout" % self.oetimeout)
-        self.alarmSignal = signal.signal(signal.SIGALRM, _timeoutHandler)
-        signal.alarm(self.oetimeout)
+            self.alarmSignal = signal.signal(signal.SIGALRM, _timeoutHandler)
+            signal.alarm(self.oetimeout)
 
     def tearDownDecorator(self):
-        signal.alarm(0)
-        signal.signal(signal.SIGALRM, self.alarmSignal)
-        self.logger.debug("Removed SIGALRM handler")
+        if isinstance(self.case.tc.runner, OETestRunnerThreaded):
+            self.timer.cancel()
+            self.logger.debug("Removed Timer handler")
+            if self.timeouted:
+                raise OEQATimeoutError("Timed out after %s "
+                    "seconds of execution" % self.oetimeout)
+        else:
+            signal.alarm(0)
+            signal.signal(signal.SIGALRM, self.alarmSignal)
+            self.logger.debug("Removed SIGALRM handler")
-- 
2.1.4

[PATCH 15/17] oeqa/core/tests: Add tests of OEQA Threaded mode

[Aníbal Limón]

Add needed tests to validate the OEQA Threaded mode, the remaining
parts are tested by the OEQA without Threaded mode.

- test_loader.py: Add a test to validate rules when creating the
list of test suites.
- test_decorators.py: Add oetimeout test because the threaded mode
uses Timer instead of signal.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../core/tests/cases/loader/threaded/threaded.py   | 12 +++++++++
 .../tests/cases/loader/threaded/threaded_alone.py  |  8 ++++++
 .../cases/loader/threaded/threaded_depends.py      | 10 ++++++++
 .../tests/cases/loader/threaded/threaded_module.py | 12 +++++++++
 meta/lib/oeqa/core/tests/common.py                 | 10 ++++++++
 meta/lib/oeqa/core/tests/test_decorators.py        | 12 +++++++++
 meta/lib/oeqa/core/tests/test_loader.py            | 30 +++++++++++++++++++++-
 7 files changed, 93 insertions(+), 1 deletion(-)
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded.py
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_alone.py
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_depends.py
 create mode 100644 meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_module.py

diff --git a/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded.py b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded.py
new file mode 100644
index 0000000..0fe4cb3
--- /dev/null
+++ b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded.py
@@ -0,0 +1,12 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+from oeqa.core.case import OETestCase
+
+class ThreadedTest(OETestCase):
+    def test_threaded_no_depends(self):
+        self.assertTrue(True, msg='How is this possible?')
+
+class ThreadedTest2(OETestCase):
+    def test_threaded_same_module(self):
+        self.assertTrue(True, msg='How is this possible?')
diff --git a/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_alone.py b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_alone.py
new file mode 100644
index 0000000..905f397
--- /dev/null
+++ b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_alone.py
@@ -0,0 +1,8 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+from oeqa.core.case import OETestCase
+
+class ThreadedTestAlone(OETestCase):
+    def test_threaded_alone(self):
+        self.assertTrue(True, msg='How is this possible?')
diff --git a/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_depends.py b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_depends.py
new file mode 100644
index 0000000..0c158d3
--- /dev/null
+++ b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_depends.py
@@ -0,0 +1,10 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+from oeqa.core.case import OETestCase
+from oeqa.core.decorator.depends import OETestDepends
+
+class ThreadedTest3(OETestCase):
+    @OETestDepends(['threaded.ThreadedTest.test_threaded_no_depends'])
+    def test_threaded_depends(self):
+        self.assertTrue(True, msg='How is this possible?')
diff --git a/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_module.py b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_module.py
new file mode 100644
index 0000000..63d17e0
--- /dev/null
+++ b/meta/lib/oeqa/core/tests/cases/loader/threaded/threaded_module.py
@@ -0,0 +1,12 @@
+# Copyright (C) 2017 Intel Corporation
+# Released under the MIT license (see COPYING.MIT)
+
+from oeqa.core.case import OETestCase
+
+class ThreadedTestModule(OETestCase):
+    def test_threaded_module(self):
+        self.assertTrue(True, msg='How is this possible?')
+
+class ThreadedTestModule2(OETestCase):
+    def test_threaded_module2(self):
+        self.assertTrue(True, msg='How is this possible?')
diff --git a/meta/lib/oeqa/core/tests/common.py b/meta/lib/oeqa/core/tests/common.py
index 52b18a1..1932323 100644
--- a/meta/lib/oeqa/core/tests/common.py
+++ b/meta/lib/oeqa/core/tests/common.py
@@ -33,3 +33,13 @@ class TestBase(unittest.TestCase):
         tc.loadTests(self.cases_path, modules=modules, tests=tests,
                      filters=filters)
         return tc
+
+    def _testLoaderThreaded(self, d={}, modules=[],
+            tests=[], filters={}):
+        from oeqa.core.threaded import OETestContextThreaded
+
+        tc = OETestContextThreaded(d, self.logger)
+        tc.loadTests(self.cases_path, modules=modules, tests=tests,
+                     filters=filters)
+
+        return tc
diff --git a/meta/lib/oeqa/core/tests/test_decorators.py b/meta/lib/oeqa/core/tests/test_decorators.py
index f7d11e8..cf99e0d 100755
--- a/meta/lib/oeqa/core/tests/test_decorators.py
+++ b/meta/lib/oeqa/core/tests/test_decorators.py
@@ -131,5 +131,17 @@ class TestTimeoutDecorator(TestBase):
         msg = "OETestTimeout didn't restore SIGALRM"
         self.assertIs(alarm_signal, signal.getsignal(signal.SIGALRM), msg=msg)
 
+    def test_timeout_thread(self):
+        tests = ['timeout.TimeoutTest.testTimeoutPass']
+        msg = 'Failed to run test using OETestTimeout'
+        tc = self._testLoaderThreaded(modules=self.modules, tests=tests)
+        self.assertTrue(tc.runTests().wasSuccessful(), msg=msg)
+
+    def test_timeout_threaded_fail(self):
+        tests = ['timeout.TimeoutTest.testTimeoutFail']
+        msg = "OETestTimeout test didn't timeout as expected"
+        tc = self._testLoaderThreaded(modules=self.modules, tests=tests)
+        self.assertFalse(tc.runTests().wasSuccessful(), msg=msg)
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/meta/lib/oeqa/core/tests/test_loader.py b/meta/lib/oeqa/core/tests/test_loader.py
index b79b8ba..e0d917d 100755
--- a/meta/lib/oeqa/core/tests/test_loader.py
+++ b/meta/lib/oeqa/core/tests/test_loader.py
@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 
-# Copyright (C) 2016 Intel Corporation
+# Copyright (C) 2016-2017 Intel Corporation
 # Released under the MIT license (see COPYING.MIT)
 
 import os
@@ -82,5 +82,33 @@ class TestLoader(TestBase):
         msg = 'Expected modules from two different paths'
         self.assertEqual(modules, expected_modules, msg=msg)
 
+    def test_loader_threaded(self):
+        cases_path = self.cases_path
+
+        self.cases_path = [os.path.join(self.cases_path, 'loader', 'threaded')]
+
+        tc = self._testLoaderThreaded()
+        self.assertEqual(len(tc.suites), 3, "Expected to be 3 suites")
+
+        case_ids = ['threaded.ThreadedTest.test_threaded_no_depends',
+                'threaded.ThreadedTest2.test_threaded_same_module',
+                'threaded_depends.ThreadedTest3.test_threaded_depends']
+        for case in tc.suites[0]._tests:
+            self.assertEqual(case.id(),
+                    case_ids[tc.suites[0]._tests.index(case)])
+
+        case_ids = ['threaded_alone.ThreadedTestAlone.test_threaded_alone']
+        for case in tc.suites[1]._tests:
+            self.assertEqual(case.id(),
+                    case_ids[tc.suites[1]._tests.index(case)])
+
+        case_ids = ['threaded_module.ThreadedTestModule.test_threaded_module',
+                'threaded_module.ThreadedTestModule2.test_threaded_module2']
+        for case in tc.suites[2]._tests:
+            self.assertEqual(case.id(),
+                    case_ids[tc.suites[2]._tests.index(case)])
+
+        self.cases_path = cases_path
+
 if __name__ == '__main__':
     unittest.main()
-- 
2.1.4

[PATCH 16/17] oeqa/sdkext/cases: Move sdk_update test into devtool module

[Aníbal Limón]

With the new OEQA thread support there are problems running
devtool twice at the same time because only one instance of
bitbake/devtool is allowed.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/lib/oeqa/sdkext/cases/devtool.py    | 32 ++++++++++++++++++++++++++
 meta/lib/oeqa/sdkext/cases/sdk_update.py | 39 --------------------------------
 2 files changed, 32 insertions(+), 39 deletions(-)
 delete mode 100644 meta/lib/oeqa/sdkext/cases/sdk_update.py

diff --git a/meta/lib/oeqa/sdkext/cases/devtool.py b/meta/lib/oeqa/sdkext/cases/devtool.py
index a01bc0b..ea90517 100644
--- a/meta/lib/oeqa/sdkext/cases/devtool.py
+++ b/meta/lib/oeqa/sdkext/cases/devtool.py
@@ -1,12 +1,14 @@
 # Copyright (C) 2016 Intel Corporation
 # Released under the MIT license (see COPYING.MIT)
 
+import os
 import shutil
 import subprocess
 
 from oeqa.sdkext.case import OESDKExtTestCase
 from oeqa.core.decorator.depends import OETestDepends
 from oeqa.core.decorator.oeid import OETestID
+from oeqa.utils.httpserver import HTTPService
 
 class DevtoolTest(OESDKExtTestCase):
     @classmethod
@@ -95,3 +97,33 @@ class DevtoolTest(OESDKExtTestCase):
             self._run('devtool build %s ' % package_nodejs)
         finally:
             self._run('devtool reset %s '% package_nodejs)
+
+class SdkUpdateTest(OESDKExtTestCase):
+    @classmethod
+    def setUpClass(self):
+        self.publish_dir = os.path.join(self.tc.sdk_dir, 'esdk_publish')
+        if os.path.exists(self.publish_dir):
+            shutil.rmtree(self.publish_dir)
+        os.mkdir(self.publish_dir)
+
+        base_tcname = "%s/%s" % (self.td.get("SDK_DEPLOY", ''),
+            self.td.get("TOOLCHAINEXT_OUTPUTNAME", ''))
+        tcname_new = "%s-new.sh" % base_tcname
+        if not os.path.exists(tcname_new):
+            tcname_new = "%s.sh" % base_tcname
+
+        cmd = 'oe-publish-sdk %s %s' % (tcname_new, self.publish_dir)
+        subprocess.check_output(cmd, shell=True)
+
+        self.http_service = HTTPService(self.publish_dir)
+        self.http_service.start()
+
+        self.http_url = "http://127.0.0.1:%d" % self.http_service.port
+
+    def test_sdk_update_http(self):
+        output = self._run("devtool sdk-update \"%s\"" % self.http_url)
+
+    @classmethod
+    def tearDownClass(self):
+        self.http_service.stop()
+        shutil.rmtree(self.publish_dir)
diff --git a/meta/lib/oeqa/sdkext/cases/sdk_update.py b/meta/lib/oeqa/sdkext/cases/sdk_update.py
deleted file mode 100644
index 2f8598b..0000000
--- a/meta/lib/oeqa/sdkext/cases/sdk_update.py
+++ /dev/null
@@ -1,39 +0,0 @@
-# Copyright (C) 2016 Intel Corporation
-# Released under the MIT license (see COPYING.MIT)
-
-import os
-import shutil
-import subprocess
-
-from oeqa.sdkext.case import OESDKExtTestCase
-from oeqa.utils.httpserver import HTTPService
-
-class SdkUpdateTest(OESDKExtTestCase):
-    @classmethod
-    def setUpClass(self):
-        self.publish_dir = os.path.join(self.tc.sdk_dir, 'esdk_publish')
-        if os.path.exists(self.publish_dir):
-            shutil.rmtree(self.publish_dir)
-        os.mkdir(self.publish_dir)
-
-        base_tcname = "%s/%s" % (self.td.get("SDK_DEPLOY", ''),
-            self.td.get("TOOLCHAINEXT_OUTPUTNAME", ''))
-        tcname_new = "%s-new.sh" % base_tcname
-        if not os.path.exists(tcname_new):
-            tcname_new = "%s.sh" % base_tcname
-
-        cmd = 'oe-publish-sdk %s %s' % (tcname_new, self.publish_dir)
-        subprocess.check_output(cmd, shell=True)
-
-        self.http_service = HTTPService(self.publish_dir)
-        self.http_service.start()
-
-        self.http_url = "http://127.0.0.1:%d" % self.http_service.port
-
-    def test_sdk_update_http(self):
-        output = self._run("devtool sdk-update \"%s\"" % self.http_url)
-
-    @classmethod
-    def tearDownClass(self):
-        self.http_service.stop()
-        shutil.rmtree(self.publish_dir)
-- 
2.1.4

[PATCH 17/17] oeqa/sdk: Enable usage of OEQA thread mode

[Aníbal Limón]

oeqa/sdk/context.py - Use OETestContextThreaded.
classes/testsdk.bbclass - Enable bb event thread mode to avoid
corrupt the PIPE when multiple threads writes.

[YOCTO #11450]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta/classes/testsdk.bbclass | 4 ++++
 meta/lib/oeqa/sdk/context.py | 5 +++--
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index edea89c..8a9e680 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -24,6 +24,8 @@ def testsdk_main(d):
     from oeqa.sdk.context import OESDKTestContext, OESDKTestContextExecutor
     from oeqa.utils import make_logger_bitbake_compatible
 
+    bb.event.enable_threadlock()
+
     pn = d.getVar("PN")
     logger = make_logger_bitbake_compatible(logging.getLogger("BitBake"))
 
@@ -97,6 +99,8 @@ def testsdkext_main(d):
     from oeqa.utils import avoid_paths_in_environ, make_logger_bitbake_compatible, subprocesstweak
     from oeqa.sdkext.context import OESDKExtTestContext, OESDKExtTestContextExecutor
 
+    bb.event.enable_threadlock()
+
     pn = d.getVar("PN")
     logger = make_logger_bitbake_compatible(logging.getLogger("BitBake"))
 
diff --git a/meta/lib/oeqa/sdk/context.py b/meta/lib/oeqa/sdk/context.py
index 82e4c19..b3d7c75 100644
--- a/meta/lib/oeqa/sdk/context.py
+++ b/meta/lib/oeqa/sdk/context.py
@@ -6,9 +6,10 @@ import sys
 import glob
 import re
 
-from oeqa.core.context import OETestContext, OETestContextExecutor
+from oeqa.core.context import OETestContextExecutor
+from oeqa.core.threaded import OETestContextThreaded
 
-class OESDKTestContext(OETestContext):
+class OESDKTestContext(OETestContextThreaded):
     sdk_files_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "files")
 
     def __init__(self, td=None, logger=None, sdk_dir=None, sdk_env=None,
-- 
2.1.4