[v2][PATCH 0/2] qa crashme

[v2][PATCH 0/2] qa crashme

[Armin Kuster]

Instead of reworking the ltp class at this time, 
here is the crashme test from Manual in an automated form.

Armin Kuster (2):
  oeqa/manual/compliance-test: move crashme to runtime
  oeqa/manual/compliance-test: remove crashme tests

 meta/lib/oeqa/manual/compliance-test.json | 48 ---------------
 meta/lib/oeqa/runtime/cases/ltp_stress.py | 97 +++++++++++++++++++++++++++++++
 2 files changed, 97 insertions(+), 48 deletions(-)
 create mode 100644 meta/lib/oeqa/runtime/cases/ltp_stress.py

-- 
2.7.4

[PATCH 1/2] oeqa/manual/compliance-test: move crashme to runtime

[Armin Kuster]

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/runtime/cases/ltp_stress.py | 97 +++++++++++++++++++++++++++++++
 1 file changed, 97 insertions(+)
 create mode 100644 meta/lib/oeqa/runtime/cases/ltp_stress.py

diff --git a/meta/lib/oeqa/runtime/cases/ltp_stress.py b/meta/lib/oeqa/runtime/cases/ltp_stress.py
new file mode 100644
index 0000000..98d65a7
--- /dev/null
+++ b/meta/lib/oeqa/runtime/cases/ltp_stress.py
@@ -0,0 +1,97 @@
+# LTP Stress runtime
+#
+# Copyright (c) 2019 MontaVista Software, LLC
+#
+# SPDX-License-Identifier: MIT
+#
+
+import time
+import datetime
+import pprint
+
+from oeqa.runtime.case import OERuntimeTestCase
+from oeqa.core.decorator.depends import OETestDepends
+from oeqa.runtime.decorator.package import OEHasPackage
+from oeqa.core.decorator.data import skipIfQemu
+from oeqa.utils.logparser import LtpParser
+
+class LtpStressBase(OERuntimeTestCase):
+
+    @classmethod
+    def setUpClass(cls):
+        cls.ltp_startup()
+
+    @classmethod
+    def tearDownClass(cls):
+        cls.ltp_finishup()
+
+    @classmethod
+    def ltp_startup(cls):
+        cls.sections = {}
+        cls.failmsg = ""
+        test_log_dir = os.path.join(cls.td.get('WORKDIR', ''), 'testimage')
+        timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
+
+        cls.ltptest_log_dir_link = os.path.join(test_log_dir, 'ltpstress_log')
+        cls.ltptest_log_dir = '%s.%s' % (cls.ltptest_log_dir_link, timestamp)
+        os.makedirs(cls.ltptest_log_dir)
+
+        cls.tc.target.run("mkdir -p /opt/ltp/results")
+
+        if not hasattr(cls.tc, "extraresults"):
+            cls.tc.extraresults = {}
+        cls.extras = cls.tc.extraresults
+        cls.extras['ltpstressresult.rawlogs'] = {'log': ""}
+
+ 
+    @classmethod
+    def ltp_finishup(cls):
+        cls.extras['ltpstressresult.sections'] =  cls.sections
+
+        # update symlink to ltp_log
+        if os.path.exists(cls.ltptest_log_dir_link):
+            os.remove(cls.ltptest_log_dir_link)
+
+        os.symlink(os.path.basename(cls.ltptest_log_dir), cls.ltptest_log_dir_link)
+
+        if cls.failmsg:
+            cls.fail(cls.failmsg)
+
+class LtpStressTest(LtpStressBase):
+
+    def runltp(self, stress_group):
+            cmd = '/opt/ltp/runltp -f %s -p -q 2>@1 | tee /opt/ltp/results/%s' % (stress_group, stress_group)
+            starttime = time.time()
+            (status, output) = self.target.run(cmd)
+            endtime = time.time()
+            with open(os.path.join(self.ltptest_log_dir, "%s" % stress_group), 'w') as f:
+                f.write(output)
+
+            self.extras['ltpstressresult.rawlogs']['log'] = self.extras['ltpstressresult.rawlogs']['log'] + output
+
+            parser = LtpParser()
+            results, sections  = parser.parse(os.path.join(self.ltptest_log_dir, "%s" % stress_group))
+
+            runtime = int(endtime-starttime)
+            sections['duration'] = runtime
+            self.sections[stress_group] =  sections
+ 
+            failed_tests = {}
+            for test in results:
+                result = results[test]
+                testname = ("ltpstressresult." + stress_group + "." + test)
+                self.extras[testname] = {'status': result}
+                if result == 'FAILED':
+                    failed_tests[stress_group] = test 
+
+            if failed_tests:
+                self.failmsg = self.failmsg + "Failed ptests:\n%s" % pprint.pformat(failed_tests)
+
+    # LTP stress runtime tests
+    #
+    @skipIfQemu('qemuall', 'Test only runs on real hardware')
+
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(["ltp"])
+    def test_ltp_stress(self):
+        self.runltp('crashme')
-- 
2.7.4

[PATCH 2/2] oeqa/manual/compliance-test: remove crashme tests

[Armin Kuster]

Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 meta/lib/oeqa/manual/compliance-test.json | 48 -------------------------------
 1 file changed, 48 deletions(-)

diff --git a/meta/lib/oeqa/manual/compliance-test.json b/meta/lib/oeqa/manual/compliance-test.json
index 367a416..e374c5b 100644
--- a/meta/lib/oeqa/manual/compliance-test.json
+++ b/meta/lib/oeqa/manual/compliance-test.json
@@ -1,52 +1,4 @@
 [
-    {
-        "test": {
-            "@alias": "compliance-test.compliance-test.stress_test_-_Genericx86-64",
-            "author": [
-                {
-                    "email": "corneliux.stoicescu@intel.com",
-                    "name": "corneliux.stoicescu@intel.com"
-                }
-            ],
-            "execution": {
-                "1": {
-                    "action": "Bootup with core-image-sato-sdk image",
-                    "expected_results": ""
-                },
-                "2": {
-                    "action": "Execute the crashme test with below command  \n\n./opt/ltp/runltp f  crashme",
-                    "expected_results": "The stress testing should not make the target crash. Check CPU usage and basic functionality of the system after the tests are over. "
-                }
-            },
-            "summary": "stress_test_-_Genericx86-64"
-        }
-    },
-     {
-    "test": {
-      "@alias": "compliance-test.compliance-test.stress_test_-_- crashme_-_-Beaglebone",
-      "author": [
-        {
-          "email": "corneliux.stoicescu@intel.com",
-          "name": "corneliux.stoicescu@intel.com"
-        }
-      ],
-      "execution": {
-        "1": {
-          "action": " Get crashme from http://people.delphiforums.com/gjc/crashme.html",
-          "expected_results": ""
-        },
-        "2": {
-          "action": "Follow the setup steps on above URL, build crashme in target",
-          "expected_results": ""
-        },
-        "3": {
-          "action": " Run crashme for 24 hours",
-          "expected_results": "Target should not crash with the program."
-        }
-      },
-      "summary": "stress_test_-_crashme_-Beaglebone"
-    }
-  },
   {
     "test": {
       "@alias": "compliance-test.compliance-test.stress_test_-_ltp_-Beaglebone",
-- 
2.7.4

✗ patchtest: failure for qa crashme

[Patchwork]

== Series Details ==

Series: qa crashme
Revision: 1
URL   : https://patchwork.openembedded.org/series/21178/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Errors in your Python code were encountered [test_pylint] 
  Suggested fix    Correct the lines introduced by your patch
  Output           Please, fix the listed issues:
                   meta/lib/oeqa/runtime/cases/ltp_stress.py does not exist



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe