[PATCH 1/3] runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

[PATCH 1/3] runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

[Alexander Kanavin]

Doing so means that all available ptests for packages in the image
will be installed and run; some of them may be broken, never finish,
take a very long time or simply irrelevant to the user who wants to
check ptests of only a few specific packages, and does so by listing
them explicitly via IMAGE_INSTALL_append or similar.

Conversely, do not run the test if there are no ptests available.

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index ec8c038a566..6669e24f141 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase):
 
     @OETestID(1600)
     @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
-    @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     def test_ptestrunner(self):
+        status, output = self.target.run('find /usr -name ptest', 0)
+        if len(output) == 0:
+            self.skipTest("No -ptest packages are installed in the image")
+
         import datetime
 
         test_log_dir = self.td.get('TEST_LOG_DIR', '')
-- 
2.15.1

[PATCH 2/3] testimage.bbclass: add ptest to the list of runtime tests whenever possible

[Alexander Kanavin]

If no ptest packages are installed in the image, the test does nothing;
if ptest packages are installed in the image, then they should be
run without user having to enable that manually.

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/classes/testimage.bbclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 7260ad4517a..3acccbb8fdb 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -44,16 +44,16 @@ DEVTESTSUITE = "gcc kernelmodule ldd"
 DEFAULT_TEST_SUITES = "${MINTESTSUITE} auto"
 DEFAULT_TEST_SUITES_pn-core-image-minimal = "${MINTESTSUITE}"
 DEFAULT_TEST_SUITES_pn-core-image-minimal-dev = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate"
+DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate ptest"
 DEFAULT_TEST_SUITES_pn-core-image-x11 = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE}"
+DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-sato = "${NETTESTSUITE} connman xorg parselogs ${RPMTESTSUITE} \
-    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)}"
+    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg"
-DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE}"
+    connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg ptest"
+DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE}"
+    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-meta-toolchain = "auto"
 
 # aarch64 has no graphics
-- 
2.15.1

[PATCH 3/3] runtime/cases/ptest.py: fail when ptests fail on target

[Alexander Kanavin]

That's the whole point isn't it? Previously this testcase succeeded
even if some of the underlying on-target tests failed; the only way
to find out if anything was wrong was to manually inspect the logs.

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/lib/oeqa/runtime/cases/ptest.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index 6669e24f141..c5b8c9cb8fb 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -83,3 +83,11 @@ class PtestRunnerTest(OERuntimeTestCase):
             # Remove the old link to create a new one
             os.remove(ptest_log_dir_link)
         os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
+
+        failed_tests = {}
+        for section in parse_result.result_dict:
+            failed_testcases = [ test for test, result in parse_result.result_dict[section] if result == 'fail' ]
+            if failed_testcases:
+                failed_tests[section] = failed_testcases
+
+        self.assertFalse(failed_tests, msg = "Failed ptests: %s" %(str(failed_tests)))
-- 
2.15.1

Re: [PATCH 3/3] runtime/cases/ptest.py: fail when ptests fail on target

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1931 bytes --]

So the context was that so many ptests were failing that the test couldn't
be done automatically and expected to succeed. Bugs were filed and the
problem chipped away at, but last I looked it wasn't in a state to be
enabled.

If a core-image-sato passes with ptests enabled then that's great, but I'll
be surprised if it does!

Ross

On 20 December 2017 at 15:45, Alexander Kanavin <
alexander.kanavin@linux.intel.com> wrote:

> That's the whole point isn't it? Previously this testcase succeeded
> even if some of the underlying on-target tests failed; the only way
> to find out if anything was wrong was to manually inspect the logs.
>
> Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
> ---
>  meta/lib/oeqa/runtime/cases/ptest.py | 8 ++++++++
>  1 file changed, 8 insertions(+)
>
> diff --git a/meta/lib/oeqa/runtime/cases/ptest.py
> b/meta/lib/oeqa/runtime/cases/ptest.py
> index 6669e24f141..c5b8c9cb8fb 100644
> --- a/meta/lib/oeqa/runtime/cases/ptest.py
> +++ b/meta/lib/oeqa/runtime/cases/ptest.py
> @@ -83,3 +83,11 @@ class PtestRunnerTest(OERuntimeTestCase):
>              # Remove the old link to create a new one
>              os.remove(ptest_log_dir_link)
>          os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
> +
> +        failed_tests = {}
> +        for section in parse_result.result_dict:
> +            failed_testcases = [ test for test, result in
> parse_result.result_dict[section] if result == 'fail' ]
> +            if failed_testcases:
> +                failed_tests[section] = failed_testcases
> +
> +        self.assertFalse(failed_tests, msg = "Failed ptests: %s"
> %(str(failed_tests)))
> --
> 2.15.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2782 bytes --]

Re: [PATCH 3/3] runtime/cases/ptest.py: fail when ptests fail on target

[Alexander Kanavin]

On 12/20/2017 07:23 PM, Burton, Ross wrote:
> So the context was that so many ptests were failing that the test 
> couldn't be done automatically and expected to succeed. Bugs were filed 
> and the problem chipped away at, but last I looked it wasn't in a state 
> to be enabled.
> 
> If a core-image-sato passes with ptests enabled then that's great, but 
> I'll be surprised if it does!

It doesn't. In fact, it never finishes, because bluez ptest hangs 
forever. That shouldn't be a problem though, because ptests are not 
included by default in any of the standard images, and so the test is 
simply skipped (see patch 1 of 3). The point of this patch is to 
eliminate misleading passing behaviour when people do include ptests - 
either for everything in their image or for specific packages they 
maintain, and want, for example, to check for regressions on version 
upgrades.

Alex