[PATCH 4/4] testsdk.bbclass: write testresult to json files

[PATCH 4/4] testsdk.bbclass: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA sdk and sdkext need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

By default, sdk and sdkext will write json testresult, to disable
this, specify OEQA_SKIP_OUTPUT_JSON="1" as configuration.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/classes/testsdk.bbclass | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index d3f475d..26c4789 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -83,7 +83,17 @@ def testsdk_main(d):
 
         result.logDetails()
         result.logSummary(component, context_msg)
-
+        if (d.getVar('OEQA_SKIP_OUTPUT_JSON')) == '1':
+            bb.debug(2, 'Skip the OEQA output json testresult as OEQA_SKIP_OUTPUT_JSON=1')
+        else:
+            workdir = d.getVar("WORKDIR")
+            image_basename = d.getVar("IMAGE_BASENAME")
+            json_result_dir = os.path.join(workdir,
+                                           'temp',
+                                           'json_testresults-%s' % os.getpid(),
+                                           'sdk',
+                                           image_basename)
+            result.logDetailsInJson(json_result_dir)
         if not result.wasSuccessful():
             fail = True
 
@@ -187,7 +197,17 @@ def testsdkext_main(d):
 
         result.logDetails()
         result.logSummary(component, context_msg)
-
+        if (d.getVar('OEQA_SKIP_OUTPUT_JSON')) == '1':
+            bb.debug(2, 'Skip the OEQA output json testresult as OEQA_SKIP_OUTPUT_JSON=1')
+        else:
+            workdir = d.getVar("WORKDIR")
+            image_basename = d.getVar("IMAGE_BASENAME")
+            json_result_dir = os.path.join(workdir,
+                                           'temp',
+                                           'json_testresults-%s' % os.getpid(),
+                                           'sdkext',
+                                           image_basename)
+            result.logDetailsInJson(json_result_dir)
         if not result.wasSuccessful():
             fail = True
 
-- 
2.7.4

[PATCH 4/4] testsdk.bbclass: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA sdk and sdkext need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

By default, sdk and sdkext will write json testresult, to disable
this, specify OEQA_SKIP_OUTPUT_JSON="1" as configuration.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/classes/testsdk.bbclass | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index d3f475d..6f81fe0 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -83,6 +83,17 @@ def testsdk_main(d):
 
         result.logDetails()
         result.logSummary(component, context_msg)
+        if (d.getVar('OEQA_SKIP_OUTPUT_JSON')) == '1':
+            bb.debug(2, 'Skip the OEQA output json testresult as OEQA_SKIP_OUTPUT_JSON=1')
+        else:
+            workdir = d.getVar("WORKDIR")
+            image_basename = d.getVar("IMAGE_BASENAME")
+            json_result_dir = os.path.join(workdir,
+                                           'temp',
+                                           'json_testresults-%s' % os.getpid(),
+                                           'sdk',
+                                           image_basename)
+            result.logDetailsInJson(json_result_dir)
 
         if not result.wasSuccessful():
             fail = True
@@ -187,6 +198,17 @@ def testsdkext_main(d):
 
         result.logDetails()
         result.logSummary(component, context_msg)
+        if (d.getVar('OEQA_SKIP_OUTPUT_JSON')) == '1':
+            bb.debug(2, 'Skip the OEQA output json testresult as OEQA_SKIP_OUTPUT_JSON=1')
+        else:
+            workdir = d.getVar("WORKDIR")
+            image_basename = d.getVar("IMAGE_BASENAME")
+            json_result_dir = os.path.join(workdir,
+                                           'temp',
+                                           'json_testresults-%s' % os.getpid(),
+                                           'sdkext',
+                                           image_basename)
+            result.logDetailsInJson(json_result_dir)
 
         if not result.wasSuccessful():
             fail = True
-- 
2.7.4

[PATCH 4/4] testsdk.bbclass: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA sdk and sdkext need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

To configure multiple instances of bitbake to write json testresult
to a single testresult file, user will define the variable
"OEQA_JSON_RESULT_COMMON_DIR" with the common directory for writing
json testresult.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/classes/testsdk.bbclass | 40 ++++++++++++++++++++++++++++++++++++----
 1 file changed, 36 insertions(+), 4 deletions(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index d3f475d..c4a0f37 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -2,6 +2,7 @@
 #
 # Released under the MIT license (see COPYING.MIT)
 
+inherit metadata_scm
 # testsdk.bbclass enables testing for SDK and Extensible SDK
 #
 # To run SDK tests, run the commands:
@@ -14,6 +15,33 @@
 #
 # where "<image-name>" is an image like core-image-sato.
 
+def _get_configuration(d, test_type, pid):
+    import platform
+    bb_core_dir = d.getVar("BBFILE_PATTERN_core=")
+    return {'TEST_TYPE': test_type,
+            'BRANCH': base_get_metadata_git_branch(bb_core_dir, None).strip(),
+            'COMMIT': base_get_metadata_git_revision(bb_core_dir, None),
+            'PROCESS_ID': pid,
+            'SDK_MACHINE': d.getVar("SDKMACHINE"),
+            'IMAGE_BASENAME': d.getVar("IMAGE_BASENAME"),
+            'IMAGE_PKGTYPE': d.getVar("IMAGE_PKGTYPE"),
+            'HOST_DISTRO': platform.linux_distribution()}
+
+def _get_json_result_dir(d, configuration):
+    json_result_dir = os.path.join(d.getVar("WORKDIR"),
+                                   'temp',
+                                   'json_testresults-%s' % configuration['PROCESS_ID'],
+                                   configuration['TEST_TYPE'],
+                                   configuration['SDK_MACHINE'],
+                                   configuration['IMAGE_BASENAME'])
+    oeqa_json_result_common_dir = d.getVar("OEQA_JSON_RESULT_COMMON_DIR")
+    if oeqa_json_result_common_dir:
+        json_result_dir = oeqa_json_result_common_dir
+    return json_result_dir
+
+def _get_result_id(configuration):
+    return '%s-%s-%s' % (configuration['TEST_TYPE'], configuration['IMAGE_BASENAME'], configuration['SDK_MACHINE'])
+
 def testsdk_main(d):
     import os
     import subprocess
@@ -80,8 +108,10 @@ def testsdk_main(d):
 
         component = "%s %s" % (pn, OESDKTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
-
-        result.logDetails()
+        configuration = _get_configuration(d, 'sdk', os.getpid())
+        result.logDetails(_get_json_result_dir(d, configuration),
+                           configuration,
+                           _get_result_id(configuration))
         result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
@@ -184,8 +214,10 @@ def testsdkext_main(d):
 
         component = "%s %s" % (pn, OESDKExtTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
-
-        result.logDetails()
+        configuration = _get_configuration(d, 'sdkext', os.getpid())
+        result.logDetails(_get_json_result_dir(d, configuration),
+                           configuration,
+                           _get_result_id(configuration))
         result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
-- 
2.7.4

[PATCH 1/4] oeqa/core/runner: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA need to output testresult into single json file, where json
testresult file will be stored in git repository by the future
test-case-management tools.

The json testresult file will store more than one set of results,
where each set of results was uniquely identified by the result_id.
The result_id would be like "runtime-qemux86-core-image-sato", where
it was a runtime test with target machine equal to qemux86 and running
on core-image-sato image. The json testresult file will only store
the latest test content for a given result_id. The json testresult
file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
result (eg. PASSED, FAILED, ERROR), test log, and result_id.

Based on the destination json testresult file directory provided,
it could have multiple instances of bitbake trying to write json
testresult to a single testresult file, using locking a lockfile
alongside the results file directory to prevent races.

Also the library class inside this patch will be reused by the future
test-case-management tools to write json testresult for manual test
case executed.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/lib/oeqa/core/runner.py | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
index f1dd080..2243a10 100644
--- a/meta/lib/oeqa/core/runner.py
+++ b/meta/lib/oeqa/core/runner.py
@@ -6,6 +6,7 @@ import time
 import unittest
 import logging
 import re
+import json
 
 from unittest import TextTestResult as _TestResult
 from unittest import TextTestRunner as _TestRunner
@@ -119,8 +120,9 @@ class OETestResult(_TestResult):
         self.successes.append((test, None))
         super(OETestResult, self).addSuccess(test)
 
-    def logDetails(self):
+    def logDetails(self, json_file_dir=None, configuration=None, result_id=None):
         self.tc.logger.info("RESULTS:")
+        result = {}
         for case_name in self.tc._registry['cases']:
             case = self.tc._registry['cases'][case_name]
 
@@ -137,6 +139,11 @@ class OETestResult(_TestResult):
                 t = " (" + "{0:.2f}".format(self.endtime[case.id()] - self.starttime[case.id()]) + "s)"
 
             self.tc.logger.info("RESULTS - %s - Testcase %s: %s%s" % (case.id(), oeid, status, t))
+            result[case.id()] = {'status': status, 'log': log}
+
+        if json_file_dir:
+            tresultjsonhelper = OETestResultJSONHelper()
+            tresultjsonhelper.dump_testresult_file(result_id, result, configuration, json_file_dir)
 
 class OEListTestsResult(object):
     def wasSuccessful(self):
@@ -249,3 +256,33 @@ class OETestRunner(_TestRunner):
             self._list_tests_module(suite)
 
         return OEListTestsResult()
+
+class OETestResultJSONHelper(object):
+
+    testresult_filename = 'testresults.json'
+
+    def _get_existing_testresults_if_available(self, write_dir):
+        testresults = {}
+        file = os.path.join(write_dir, self.testresult_filename)
+        if os.path.exists(file):
+            with open(file, "r") as f:
+                testresults = json.load(f)
+        return testresults
+
+    def _create_json_testresults_string(self, test_results, result_id, test_result, configuration):
+        test_results[result_id] = {'configuration': configuration, 'result': test_result}
+        return json.dumps(test_results, sort_keys=True, indent=4)
+
+    def _write_file(self, write_dir, file_name, file_content):
+        file_path = os.path.join(write_dir, file_name)
+        with open(file_path, 'w') as the_file:
+            the_file.write(file_content)
+
+    def dump_testresult_file(self, result_id, test_result, configuration, write_dir):
+        bb.utils.mkdirhier(write_dir)
+        lf = bb.utils.lockfile(os.path.join(write_dir, 'jsontestresult.lock'))
+        test_results = self._get_existing_testresults_if_available(write_dir)
+        test_results[result_id] = {'configuration': configuration, 'result': test_result}
+        json_testresults = json.dumps(test_results, sort_keys=True, indent=4)
+        self._write_file(write_dir, self.testresult_filename, json_testresults)
+        bb.utils.unlockfile(lf)
-- 
2.7.4

[PATCH 2/4] oeqa/selftest/context: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA selftest need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom direcotry, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for writing
json testresult.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/lib/oeqa/selftest/context.py | 36 +++++++++++++++++++++++++++++++++---
 1 file changed, 33 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index c78947e..59d4b59 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -99,8 +99,8 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
         return cases_paths
 
     def _process_args(self, logger, args):
-        args.output_log = '%s-results-%s.log' % (self.name,
-                time.strftime("%Y%m%d%H%M%S"))
+        args.test_start_time = time.strftime("%Y%m%d%H%M%S")
+        args.output_log = '%s-results-%s.log' % (self.name, args.test_start_time)
         args.test_data_file = None
         args.CASES_PATHS = None
 
@@ -204,6 +204,33 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
         self.tc.logger.info("Running bitbake -e to test the configuration is valid/parsable")
         runCmd("bitbake -e")
 
+    def _get_json_result_dir(self, args):
+        json_result_dir = os.path.join(os.path.dirname(os.path.abspath(args.output_log)), 'oeqa')
+        if "OEQA_JSON_RESULT_DIR" in self.tc.td:
+            json_result_dir = self.tc.td["OEQA_JSON_RESULT_DIR"]
+
+        return json_result_dir
+
+    def _get_configuration(self, args):
+        import platform
+        from oeqa.utils.metadata import metadata_from_bb
+        metadata = metadata_from_bb()
+        configuration = {'TEST_TYPE': 'oeselftest',
+                        'START_TIME': args.test_start_time,
+                        'MACHINE': self.tc.td["MACHINE"],
+                        'HOST_DISTRO': platform.linux_distribution(),
+                        'HOST_NAME': metadata['hostname']}
+        layers = metadata['layers']
+        for l in layers:
+            configuration['%s_BRANCH_REV' % os.path.basename(l)] = '%s:%s' % (
+                                                                    metadata['layers'][l]['branch'],
+                                                                    metadata['layers'][l]['commit'])
+        return configuration
+
+    def _get_result_id(self, configuration):
+        distro = '_'.join(configuration['HOST_DISTRO'])
+        return '%s-%s-%s' % (configuration['TEST_TYPE'], distro, configuration['MACHINE'])
+
     def _internal_run(self, logger, args):
         self.module_paths = self._get_cases_paths(
                 self.tc_kwargs['init']['td']['BBPATH'].split(':'))
@@ -220,7 +247,10 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
         else:
             self._pre_run()
             rc = self.tc.runTests(**self.tc_kwargs['run'])
-            rc.logDetails()
+            configuration = self._get_configuration(args)
+            rc.logDetails(self._get_json_result_dir(args),
+                          configuration,
+                          self._get_result_id(configuration))
             rc.logSummary(self.name)
 
         return rc
-- 
2.7.4

[PATCH 3/4] testimage.bbclass: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA testimage need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom direcotry, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for writing
json testresult.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/classes/testimage.bbclass | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 2642a72..df91d90 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -2,7 +2,7 @@
 #
 # Released under the MIT license (see COPYING.MIT)
 
-
+inherit metadata_scm
 # testimage.bbclass enables testing of qemu images using python unittests.
 # Most of the tests are commands run on target image over ssh.
 # To use it add testimage to global inherit and call your target image with -c testimage
@@ -141,6 +141,30 @@ def testimage_sanity(d):
         bb.fatal('When TEST_TARGET is set to "simpleremote" '
                  'TEST_TARGET_IP and TEST_SERVER_IP are needed too.')
 
+def _get_testimage_configuration(d, test_type, pid, machine):
+    import platform
+    configuration = {'TEST_TYPE': test_type,
+                    'PROCESS_ID': pid,
+                    'MACHINE': machine,
+                    'IMAGE_BASENAME': d.getVar("IMAGE_BASENAME"),
+                    'IMAGE_PKGTYPE': d.getVar("IMAGE_PKGTYPE"),
+                    'HOST_DISTRO': platform.linux_distribution()}
+    layers = (d.getVar("BBLAYERS") or "").split()
+    for l in layers:
+        configuration['%s_BRANCH_REV' % os.path.basename(l)] = '%s:%s' % (base_get_metadata_git_branch(l, None).strip(),
+                                                                          base_get_metadata_git_revision(l, None))
+    return configuration
+
+def _get_testimage_json_result_dir(d, configuration):
+    json_result_dir = os.path.join(d.getVar("WORKDIR"), 'oeqa')
+    oeqa_json_result_common_dir = d.getVar("OEQA_JSON_RESULT_DIR")
+    if oeqa_json_result_common_dir:
+        json_result_dir = oeqa_json_result_common_dir
+    return json_result_dir
+
+def _get_testimage_result_id(configuration):
+    return '%s-%s-%s' % (configuration['TEST_TYPE'], configuration['IMAGE_BASENAME'], configuration['MACHINE'])
+
 def testimage_main(d):
     import os
     import json
@@ -308,7 +332,10 @@ def testimage_main(d):
     # Show results (if we have them)
     if not results:
         bb.fatal('%s - FAILED - tests were interrupted during execution' % pn, forcelog=True)
-    results.logDetails()
+    configuration = _get_testimage_configuration(d, 'runtime', os.getpid(), machine)
+    results.logDetails(_get_testimage_json_result_dir(d, configuration),
+                       configuration,
+                       _get_testimage_result_id(configuration))
     results.logSummary(pn)
     if not results.wasSuccessful():
         bb.fatal('%s - FAILED - check the task log and the ssh log' % pn, forcelog=True)
-- 
2.7.4

[PATCH 4/4] testsdk.bbclass: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA sdk and sdkext need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom direcotry, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for writing
json testresult.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/classes/testsdk.bbclass | 36 ++++++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index d3f475d..2e5f672 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -14,6 +14,30 @@
 #
 # where "<image-name>" is an image like core-image-sato.
 
+def _get_sdk_configuration(d, test_type, pid):
+    import platform
+    configuration = {'TEST_TYPE': test_type,
+                    'PROCESS_ID': pid,
+                    'SDK_MACHINE': d.getVar("SDKMACHINE"),
+                    'IMAGE_BASENAME': d.getVar("IMAGE_BASENAME"),
+                    'IMAGE_PKGTYPE': d.getVar("IMAGE_PKGTYPE"),
+                    'HOST_DISTRO': platform.linux_distribution()}
+    layers = (d.getVar("BBLAYERS") or "").split()
+    for l in layers:
+        configuration['%s_BRANCH_REV' % os.path.basename(l)] = '%s:%s' % (base_get_metadata_git_branch(l, None).strip(),
+                                                                          base_get_metadata_git_revision(l, None))
+    return configuration
+
+def _get_sdk_json_result_dir(d, configuration):
+    json_result_dir = os.path.join(d.getVar("WORKDIR"), 'oeqa')
+    oeqa_json_result_common_dir = d.getVar("OEQA_JSON_RESULT_DIR")
+    if oeqa_json_result_common_dir:
+        json_result_dir = oeqa_json_result_common_dir
+    return json_result_dir
+
+def _get_sdk_result_id(configuration):
+    return '%s-%s-%s' % (configuration['TEST_TYPE'], configuration['IMAGE_BASENAME'], configuration['SDK_MACHINE'])
+
 def testsdk_main(d):
     import os
     import subprocess
@@ -80,8 +104,10 @@ def testsdk_main(d):
 
         component = "%s %s" % (pn, OESDKTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
-
-        result.logDetails()
+        configuration = _get_sdk_configuration(d, 'sdk', os.getpid())
+        result.logDetails(_get_sdk_json_result_dir(d, configuration),
+                           configuration,
+                           _get_sdk_result_id(configuration))
         result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
@@ -184,8 +210,10 @@ def testsdkext_main(d):
 
         component = "%s %s" % (pn, OESDKExtTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
-
-        result.logDetails()
+        configuration = _get_sdk_configuration(d, 'sdkext', os.getpid())
+        result.logDetails(_get_sdk_json_result_dir(d, configuration),
+                           configuration,
+                           _get_sdk_result_id(configuration))
         result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
-- 
2.7.4

Re: [PATCH 1/4] oeqa/core/runner: write testresult to json files

[Richard Purdie]

On Mon, 2018-10-22 at 18:34 +0800, Yeoh Ee Peng wrote:
> As part of the solution to replace Testopia to store testresult,
> OEQA need to output testresult into single json file, where json
> testresult file will be stored in git repository by the future
> test-case-management tools.
> 
> The json testresult file will store more than one set of results,
> where each set of results was uniquely identified by the result_id.
> The result_id would be like "runtime-qemux86-core-image-sato", where
> it was a runtime test with target machine equal to qemux86 and running
> on core-image-sato image. The json testresult file will only store
> the latest test content for a given result_id. The json testresult
> file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
> result (eg. PASSED, FAILED, ERROR), test log, and result_id.
> 
> Based on the destination json testresult file directory provided,
> it could have multiple instances of bitbake trying to write json
> testresult to a single testresult file, using locking a lockfile
> alongside the results file directory to prevent races.
> 
> Also the library class inside this patch will be reused by the future
> test-case-management tools to write json testresult for manual test
> case executed.
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
> ---
>  meta/lib/oeqa/core/runner.py | 39 ++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 38 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/core/runner.py b/meta/lib/oeqa/core/runner.py
> index f1dd080..2243a10 100644
> --- a/meta/lib/oeqa/core/runner.py
> +++ b/meta/lib/oeqa/core/runner.py
> @@ -6,6 +6,7 @@ import time
>  import unittest
>  import logging
>  import re
> +import json
>  
>  from unittest import TextTestResult as _TestResult
>  from unittest import TextTestRunner as _TestRunner
> @@ -119,8 +120,9 @@ class OETestResult(_TestResult):
>          self.successes.append((test, None))
>          super(OETestResult, self).addSuccess(test)
>  
> -    def logDetails(self):
> +    def logDetails(self, json_file_dir=None, configuration=None, result_id=None):
>          self.tc.logger.info("RESULTS:")
> +        result = {}
>          for case_name in self.tc._registry['cases']:
>              case = self.tc._registry['cases'][case_name]
>  
> @@ -137,6 +139,11 @@ class OETestResult(_TestResult):
>                  t = " (" + "{0:.2f}".format(self.endtime[case.id()] - self.starttime[case.id()]) + "s)"
>  
>              self.tc.logger.info("RESULTS - %s - Testcase %s: %s%s" % (case.id(), oeid, status, t))
> +            result[case.id()] = {'status': status, 'log': log}
> +
> +        if json_file_dir:
> +            tresultjsonhelper = OETestResultJSONHelper()
> +            tresultjsonhelper.dump_testresult_file(result_id, result, configuration, json_file_dir)
>  
>  class OEListTestsResult(object):
>      def wasSuccessful(self):
> @@ -249,3 +256,33 @@ class OETestRunner(_TestRunner):
>              self._list_tests_module(suite)
>  
>          return OEListTestsResult()
> +
> +class OETestResultJSONHelper(object):
> +
> +    testresult_filename = 'testresults.json'
> +
> +    def _get_existing_testresults_if_available(self, write_dir):
> +        testresults = {}
> +        file = os.path.join(write_dir, self.testresult_filename)
> +        if os.path.exists(file):
> +            with open(file, "r") as f:
> +                testresults = json.load(f)
> +        return testresults
> +
> +    def _create_json_testresults_string(self, test_results, result_id, test_result, configuration):
> +        test_results[result_id] = {'configuration': configuration, 'result': test_result}
> +        return json.dumps(test_results, sort_keys=True, indent=4)

I think the above function can be removed as its no longer used
anywhere.

I've queued these patches for testing on the main autobuilder as I
think we're close and wanted to check it doesn't show issues. I still
wonder if we couldn't write out the files to a specific directory more
reliably but I think I'd need to look at the code for longer to come up
with any proposal. Right now I guess we can at least configure the
autobuilder to put the files in a common location (so it can collect
them up).

Cheers,

Richard

> +    def _write_file(self, write_dir, file_name, file_content):
> +        file_path = os.path.join(write_dir, file_name)
> +        with open(file_path, 'w') as the_file:
> +            the_file.write(file_content)
> +
> +    def dump_testresult_file(self, result_id, test_result, configuration, write_dir):
> +        bb.utils.mkdirhier(write_dir)
> +        lf = bb.utils.lockfile(os.path.join(write_dir, 'jsontestresult.lock'))
> +        test_results = self._get_existing_testresults_if_available(write_dir)
> +        test_results[result_id] = {'configuration': configuration, 'result': test_result}
> +        json_testresults = json.dumps(test_results, sort_keys=True, indent=4)
> +        self._write_file(write_dir, self.testresult_filename, json_testresults)
> +        bb.utils.unlockfile(lf)
> -- 
> 2.7.4
> 

[PATCH 4/4] testsdk.bbclass: write testresult to json files

[Yeoh Ee Peng]

As part of the solution to replace Testopia to store testresult,
OEQA sdk and sdkext need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.

By default, json testresult file will be written to "oeqa"
directory under the "WORKDIR" directory.

To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.

Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
---
 meta/classes/testsdk.bbclass | 36 ++++++++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

diff --git a/meta/classes/testsdk.bbclass b/meta/classes/testsdk.bbclass
index d3f475d..a6c3627 100644
--- a/meta/classes/testsdk.bbclass
+++ b/meta/classes/testsdk.bbclass
@@ -14,6 +14,30 @@
 #
 # where "<image-name>" is an image like core-image-sato.
 
+def _get_sdk_configuration(d, test_type, pid):
+    import platform
+    configuration = {'TEST_TYPE': test_type,
+                    'PROCESS_ID': pid,
+                    'SDK_MACHINE': d.getVar("SDKMACHINE"),
+                    'IMAGE_BASENAME': d.getVar("IMAGE_BASENAME"),
+                    'IMAGE_PKGTYPE': d.getVar("IMAGE_PKGTYPE"),
+                    'HOST_DISTRO': ('-'.join(platform.linux_distribution())).replace(' ', '-')}
+    layers = (d.getVar("BBLAYERS") or "").split()
+    for l in layers:
+        configuration['%s_BRANCH_REV' % os.path.basename(l)] = '%s:%s' % (base_get_metadata_git_branch(l, None).strip(),
+                                                                          base_get_metadata_git_revision(l, None))
+    return configuration
+
+def _get_sdk_json_result_dir(d):
+    json_result_dir = os.path.join(d.getVar("WORKDIR"), 'oeqa')
+    custom_json_result_dir = d.getVar("OEQA_JSON_RESULT_DIR")
+    if custom_json_result_dir:
+        json_result_dir = custom_json_result_dir
+    return json_result_dir
+
+def _get_sdk_result_id(configuration):
+    return '%s_%s_%s' % (configuration['TEST_TYPE'], configuration['IMAGE_BASENAME'], configuration['SDK_MACHINE'])
+
 def testsdk_main(d):
     import os
     import subprocess
@@ -80,8 +104,10 @@ def testsdk_main(d):
 
         component = "%s %s" % (pn, OESDKTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
-
-        result.logDetails()
+        configuration = _get_sdk_configuration(d, 'sdk', os.getpid())
+        result.logDetails(_get_sdk_json_result_dir(d),
+                          configuration,
+                          _get_sdk_result_id(configuration))
         result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
@@ -184,8 +210,10 @@ def testsdkext_main(d):
 
         component = "%s %s" % (pn, OESDKExtTestContextExecutor.name)
         context_msg = "%s:%s" % (os.path.basename(tcname), os.path.basename(sdk_env))
-
-        result.logDetails()
+        configuration = _get_sdk_configuration(d, 'sdkext', os.getpid())
+        result.logDetails(_get_sdk_json_result_dir(d),
+                          configuration,
+                          _get_sdk_result_id(configuration))
         result.logSummary(component, context_msg)
 
         if not result.wasSuccessful():
-- 
2.7.4

Re: [PATCH 1/4] oeqa/core/runner: write testresult to json files

[Yeoh, Ee Peng]

Hi Richard,

I submitted the revised patches below. Sorry for the missing "version #" in the patch title. After this, I will add the "version #" into the patch title. 

Please let me know if any question or inputs. Thank you very much for your attention & sharing! 

Best regards,
Yeoh Ee Peng 

A) oeqa/core/runner:
	- removed no longer used function
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156990.html

B) oeqa/selftest/context:
	- concatenate list of element from DISTRO to single element, replace empty character
	Eg. 'HOST_DISTRO': ('-'.join(platform.linux_distribution())).replace(' ', '-')
	- change result_id to separator character from "-" to "_"  
	Eg. '%s_%s_%s' % (configuration['TEST_TYPE'], configuration['HOST_DISTRO'], configuration['MACHINE'])
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156991.html

C) testimage.bbclass
	- removed no longer used "configuration" argument inside function
	Eg. def _get_testimage_json_result_dir(d):
	- concatenate list of element from DISTRO to single element, replace empty character
	- change result_id to separator character from "-" to "_"  
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156992.html

D) testsdk.bbclass
	- removed no longer used "configuration" argument inside function
	Eg. def _get_testimage_json_result_dir(d):
	- concatenate list of element from DISTRO to single element, replace empty character
	- change result_id to separator character from "-" to "_"  
http://lists.openembedded.org/pipermail/openembedded-core/2018-October/156993.html



-----Original Message-----
From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org] 
Sent: Tuesday, October 23, 2018 6:54 AM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 1/4] oeqa/core/runner: write testresult to json files

On Mon, 2018-10-22 at 18:34 +0800, Yeoh Ee Peng wrote:
> As part of the solution to replace Testopia to store testresult, OEQA 
> need to output testresult into single json file, where json testresult 
> file will be stored in git repository by the future 
> test-case-management tools.
> 
> The json testresult file will store more than one set of results, 
> where each set of results was uniquely identified by the result_id.
> The result_id would be like "runtime-qemux86-core-image-sato", where 
> it was a runtime test with target machine equal to qemux86 and running 
> on core-image-sato image. The json testresult file will only store the 
> latest test content for a given result_id. The json testresult file 
> contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE), 
> result (eg. PASSED, FAILED, ERROR), test log, and result_id.
> 
> Based on the destination json testresult file directory provided, it 
> could have multiple instances of bitbake trying to write json 
> testresult to a single testresult file, using locking a lockfile 
> alongside the results file directory to prevent races.
> 
> Also the library class inside this patch will be reused by the future 
> test-case-management tools to write json testresult for manual test 
> case executed.
> 
> Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
> ---
>  meta/lib/oeqa/core/runner.py | 39 
> ++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 38 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oeqa/core/runner.py 
> b/meta/lib/oeqa/core/runner.py index f1dd080..2243a10 100644
> --- a/meta/lib/oeqa/core/runner.py
> +++ b/meta/lib/oeqa/core/runner.py
> @@ -6,6 +6,7 @@ import time
>  import unittest
>  import logging
>  import re
> +import json
>  
>  from unittest import TextTestResult as _TestResult  from unittest 
> import TextTestRunner as _TestRunner @@ -119,8 +120,9 @@ class 
> OETestResult(_TestResult):
>          self.successes.append((test, None))
>          super(OETestResult, self).addSuccess(test)
>  
> -    def logDetails(self):
> +    def logDetails(self, json_file_dir=None, configuration=None, result_id=None):
>          self.tc.logger.info("RESULTS:")
> +        result = {}
>          for case_name in self.tc._registry['cases']:
>              case = self.tc._registry['cases'][case_name]
>  
> @@ -137,6 +139,11 @@ class OETestResult(_TestResult):
>                  t = " (" + "{0:.2f}".format(self.endtime[case.id()] - self.starttime[case.id()]) + "s)"
>  
>              self.tc.logger.info("RESULTS - %s - Testcase %s: %s%s" % 
> (case.id(), oeid, status, t))
> +            result[case.id()] = {'status': status, 'log': log}
> +
> +        if json_file_dir:
> +            tresultjsonhelper = OETestResultJSONHelper()
> +            tresultjsonhelper.dump_testresult_file(result_id, result, 
> + configuration, json_file_dir)
>  
>  class OEListTestsResult(object):
>      def wasSuccessful(self):
> @@ -249,3 +256,33 @@ class OETestRunner(_TestRunner):
>              self._list_tests_module(suite)
>  
>          return OEListTestsResult()
> +
> +class OETestResultJSONHelper(object):
> +
> +    testresult_filename = 'testresults.json'
> +
> +    def _get_existing_testresults_if_available(self, write_dir):
> +        testresults = {}
> +        file = os.path.join(write_dir, self.testresult_filename)
> +        if os.path.exists(file):
> +            with open(file, "r") as f:
> +                testresults = json.load(f)
> +        return testresults
> +
> +    def _create_json_testresults_string(self, test_results, result_id, test_result, configuration):
> +        test_results[result_id] = {'configuration': configuration, 'result': test_result}
> +        return json.dumps(test_results, sort_keys=True, indent=4)

I think the above function can be removed as its no longer used anywhere.

I've queued these patches for testing on the main autobuilder as I think we're close and wanted to check it doesn't show issues. I still wonder if we couldn't write out the files to a specific directory more reliably but I think I'd need to look at the code for longer to come up with any proposal. Right now I guess we can at least configure the autobuilder to put the files in a common location (so it can collect them up).

Cheers,

Richard

> +    def _write_file(self, write_dir, file_name, file_content):
> +        file_path = os.path.join(write_dir, file_name)
> +        with open(file_path, 'w') as the_file:
> +            the_file.write(file_content)
> +
> +    def dump_testresult_file(self, result_id, test_result, configuration, write_dir):
> +        bb.utils.mkdirhier(write_dir)
> +        lf = bb.utils.lockfile(os.path.join(write_dir, 'jsontestresult.lock'))
> +        test_results = self._get_existing_testresults_if_available(write_dir)
> +        test_results[result_id] = {'configuration': configuration, 'result': test_result}
> +        json_testresults = json.dumps(test_results, sort_keys=True, indent=4)
> +        self._write_file(write_dir, self.testresult_filename, json_testresults)
> +        bb.utils.unlockfile(lf)
> --
> 2.7.4
> 

Re: [PATCH 1/4] oeqa/core/runner: write testresult to json files

[richard.purdie]

On Tue, 2018-10-23 at 06:39 +0000, Yeoh, Ee Peng wrote:
> I submitted the revised patches below. Sorry for the missing "version
> #" in the patch title. After this, I will add the "version #" into
> the patch title. 
> 
> Please let me know if any question or inputs. Thank you very much for
> your attention & sharing! 

Thanks for the changes. I was at a conference last week and then became
unwell. I wanted to have a further look and test the patches before
merging and I wanted a clearer head to do that.

Unfortunately I found another problem. On my build machine, it shows:

Traceback (most recent call last):
  File "/media/build1/poky/scripts/oe-selftest", line 70, in <module>
    ret = main()
  File "/media/build1/poky/scripts/oe-selftest", line 57, in main
    results = args.func(logger, args)
  File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line 289, in run
    rc = self._internal_run(logger, args)
  File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line 248, in _internal_run
    configuration = self._get_configuration(args)
  File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line 217, in _get_configuration
    metadata = metadata_from_bb()
  File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 42, in metadata_from_bb
    info_dict['layers'] = get_layers(data_dict['BBLAYERS'])
  File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 81, in get_layers
    layer_dict[layer_name] = git_rev_info(layer)
  File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 61, in git_rev_info
    from git import Repo, InvalidGitRepositoryError, NoSuchPathError
ModuleNotFoundError: No module named 'git'

That is obviously easily fixed by installing the git module but it does
raise some questions, in particular, why we have two code paths which
do the same thing (one in metadata_scm.bbclass and one in
lib/oeqa/utils/metadata.py).

It also means we've just added new module dependencies to oe-selftest
and the other test utilities which we don't test for anywhere or have
documented. Doing this last thing in M4 is bad.

So this is going to need a little more thought...

Cheers,

Richard

Re: [PATCH 1/4] oeqa/core/runner: write testresult to json files

[Richard Purdie]

On Mon, 2018-10-29 at 10:44 +0000, richard.purdie@linuxfoundation.org
wrote:
> On Tue, 2018-10-23 at 06:39 +0000, Yeoh, Ee Peng wrote:
> > I submitted the revised patches below. Sorry for the missing
> > "version
> > #" in the patch title. After this, I will add the "version #" into
> > the patch title. 
> > 
> > Please let me know if any question or inputs. Thank you very much
> > for
> > your attention & sharing! 
> 
> Thanks for the changes. I was at a conference last week and then
> became
> unwell. I wanted to have a further look and test the patches before
> merging and I wanted a clearer head to do that.
> 
> Unfortunately I found another problem. On my build machine, it shows:
> 
> Traceback (most recent call last):
>   File "/media/build1/poky/scripts/oe-selftest", line 70, in <module>
>     ret = main()
>   File "/media/build1/poky/scripts/oe-selftest", line 57, in main
>     results = args.func(logger, args)
>   File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line
> 289, in run
>     rc = self._internal_run(logger, args)
>   File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line
> 248, in _internal_run
>     configuration = self._get_configuration(args)
>   File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line
> 217, in _get_configuration
>     metadata = metadata_from_bb()
>   File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 42,
> in metadata_from_bb
>     info_dict['layers'] = get_layers(data_dict['BBLAYERS'])
>   File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 81,
> in get_layers
>     layer_dict[layer_name] = git_rev_info(layer)
>   File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 61,
> in git_rev_info
>     from git import Repo, InvalidGitRepositoryError, NoSuchPathError
> ModuleNotFoundError: No module named 'git'
> 
> That is obviously easily fixed by installing the git module but it
> does
> raise some questions, in particular, why we have two code paths which
> do the same thing (one in metadata_scm.bbclass and one in
> lib/oeqa/utils/metadata.py).
> 
> It also means we've just added new module dependencies to oe-selftest
> and the other test utilities which we don't test for anywhere or have
> documented. Doing this last thing in M4 is bad.
> 
> So this is going to need a little more thought...

I've played with the patches today. The above can be addressed by using
the same code as we use in metadata_scm in the oeqa metadata file as a
quick fix. This all needs to be reworked in 2.7 after we sort out 2.6.
I've a patch queued. I also noticed that:

* We have pointless empty log entries in the json files
* SDKs don't record which MACHINE built them (in the unique identifier 
  or the configuration section)
* the identifiers for the configuration sections in the json files are 
  not unique and results from multiple runs were being overwritten 
  locally
* the patches call SDKMACHINE SDK_MACHINE which just confuses things
* The layer metadata config was being squashed into a single entry 
  with multiple contents, there is no good reason to do that in json, 
  just leave the fields separate
* The layer metadata was being obtained from different functions, 
  potentially leading to different processing
* The output data was not being placed in LOG_DIR
* The functions still had weird "_" prefixes

Since time is short, to fix these issues I'm going to include a set of
tweaks for the patches.

Cheers,

Richard

Re: [PATCH 1/4] oeqa/core/runner: write testresult to json files

[Yeoh, Ee Peng]

Hi Richard,

Thanks for sharing with us your inputs for this patch. 

For the weird "_" prefixes, the 3 new methods were initially developed in “OESelftestTestContextExecutor” class to gather information needed for selftest to write json test result file, while the “OESelftestTestContextExecutor” class have the public interface such as "run" and "register_commands", the decision at that point was to use "_" for the 3 new methods to highlight that these new methods for was internal use and avoid confusion with the existing public interface.  While porting these new methods to bbclass, I made the mistake and missed the fact that "_" was a weird practice and make no sense in bbclass. Thank you for sharing this with me, I will pay special attention and avoid this in future.    

For the identifiers for the configuration sections in the json files are not unique and results from multiple runs were being overwritten locally. This was done intentionally as the initial design of json test results file was for the QA usage. From QA usage, there could be multiple set of tests being executed for the same configuration, some of these test result sets may be invalid due to environments used, where QA only want the latest set of test that was valid.  Thus this result in the decision to have the code that generate the unique key without timestamp, so that the json test results file will only store the latest valid result. Hope this explain the reason. Now that you shared with us the important of timestamp, we understand the responsibility of this json test results file was more than QA, where it was important to keep record for all executed tests. 

Hope this explain some reasons for the patch submitted. Thank you again for sharing your inputs with us!

Best regards,
Yeoh Ee Peng 

-----Original Message-----
From: richard.purdie@linuxfoundation.org [mailto:richard.purdie@linuxfoundation.org] 
Sent: Monday, October 29, 2018 9:59 PM
To: Yeoh, Ee Peng <ee.peng.yeoh@intel.com>; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 1/4] oeqa/core/runner: write testresult to json files

On Mon, 2018-10-29 at 10:44 +0000, richard.purdie@linuxfoundation.org
wrote:
> On Tue, 2018-10-23 at 06:39 +0000, Yeoh, Ee Peng wrote:
> > I submitted the revised patches below. Sorry for the missing 
> > "version #" in the patch title. After this, I will add the "version 
> > #" into the patch title.
> > 
> > Please let me know if any question or inputs. Thank you very much 
> > for your attention & sharing!
> 
> Thanks for the changes. I was at a conference last week and then 
> became unwell. I wanted to have a further look and test the patches 
> before merging and I wanted a clearer head to do that.
> 
> Unfortunately I found another problem. On my build machine, it shows:
> 
> Traceback (most recent call last):
>   File "/media/build1/poky/scripts/oe-selftest", line 70, in <module>
>     ret = main()
>   File "/media/build1/poky/scripts/oe-selftest", line 57, in main
>     results = args.func(logger, args)
>   File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line 
> 289, in run
>     rc = self._internal_run(logger, args)
>   File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line 
> 248, in _internal_run
>     configuration = self._get_configuration(args)
>   File "/media/build1/poky/meta/lib/oeqa/selftest/context.py", line 
> 217, in _get_configuration
>     metadata = metadata_from_bb()
>   File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 42, 
> in metadata_from_bb
>     info_dict['layers'] = get_layers(data_dict['BBLAYERS'])
>   File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 81, 
> in get_layers
>     layer_dict[layer_name] = git_rev_info(layer)
>   File "/media/build1/poky/meta/lib/oeqa/utils/metadata.py", line 61, 
> in git_rev_info
>     from git import Repo, InvalidGitRepositoryError, NoSuchPathError
> ModuleNotFoundError: No module named 'git'
> 
> That is obviously easily fixed by installing the git module but it 
> does raise some questions, in particular, why we have two code paths 
> which do the same thing (one in metadata_scm.bbclass and one in 
> lib/oeqa/utils/metadata.py).
> 
> It also means we've just added new module dependencies to oe-selftest 
> and the other test utilities which we don't test for anywhere or have 
> documented. Doing this last thing in M4 is bad.
> 
> So this is going to need a little more thought...

I've played with the patches today. The above can be addressed by using the same code as we use in metadata_scm in the oeqa metadata file as a quick fix. This all needs to be reworked in 2.7 after we sort out 2.6.
I've a patch queued. I also noticed that:

* We have pointless empty log entries in the json files
* SDKs don't record which MACHINE built them (in the unique identifier
  or the configuration section)
* the identifiers for the configuration sections in the json files are
  not unique and results from multiple runs were being overwritten
  locally
* the patches call SDKMACHINE SDK_MACHINE which just confuses things
* The layer metadata config was being squashed into a single entry
  with multiple contents, there is no good reason to do that in json,
  just leave the fields separate
* The layer metadata was being obtained from different functions,
  potentially leading to different processing
* The output data was not being placed in LOG_DIR
* The functions still had weird "_" prefixes

Since time is short, to fix these issues I'm going to include a set of tweaks for the patches.

Cheers,

Richard