[PATCHv2 1/3] runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

[PATCHv2 1/3] runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

[Alexander Kanavin]

Doing so means that all available ptests for packages in the image
will be installed and run; some of them may be broken, never finish,
take a very long time or simply irrelevant to the user who wants to
check ptests of only a few specific packages, and does so by listing
them explicitly via IMAGE_INSTALL_append or similar.

Conversely, do not run the test if ptest-runner is not installed
(which means there are no ptests available, as they pull it in via a
dependency).

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index ec8c038a566..63e119530dc 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase):
 
     @OETestID(1600)
     @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
-    @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES')
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     def test_ptestrunner(self):
+        status, output = self.target.run('which ptest-runner', 0)
+        if len(output) == 0:
+            self.skipTest("No -ptest packages are installed in the image")
+
         import datetime
 
         test_log_dir = self.td.get('TEST_LOG_DIR', '')
-- 
2.15.1

[PATCHv2 2/3] testimage.bbclass: add ptest to the list of runtime tests whenever possible

[Alexander Kanavin]

If no ptest packages are installed in the image, the test does nothing;
if ptest packages are installed in the image, then they should be
run without user having to enable that manually.

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/classes/testimage.bbclass | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/meta/classes/testimage.bbclass b/meta/classes/testimage.bbclass
index 7260ad4517a..3acccbb8fdb 100644
--- a/meta/classes/testimage.bbclass
+++ b/meta/classes/testimage.bbclass
@@ -44,16 +44,16 @@ DEVTESTSUITE = "gcc kernelmodule ldd"
 DEFAULT_TEST_SUITES = "${MINTESTSUITE} auto"
 DEFAULT_TEST_SUITES_pn-core-image-minimal = "${MINTESTSUITE}"
 DEFAULT_TEST_SUITES_pn-core-image-minimal-dev = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate"
+DEFAULT_TEST_SUITES_pn-core-image-full-cmdline = "${NETTESTSUITE} perl python logrotate ptest"
 DEFAULT_TEST_SUITES_pn-core-image-x11 = "${MINTESTSUITE}"
-DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE}"
+DEFAULT_TEST_SUITES_pn-core-image-lsb = "${NETTESTSUITE} pam parselogs ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-sato = "${NETTESTSUITE} connman xorg parselogs ${RPMTESTSUITE} \
-    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)}"
+    ${@bb.utils.contains('IMAGE_PKGTYPE', 'rpm', 'python', '', d)} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-sato-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg"
-DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE}"
+    connman ${DEVTESTSUITE} logrotate perl parselogs python ${RPMTESTSUITE} xorg ptest"
+DEFAULT_TEST_SUITES_pn-core-image-lsb-dev = "${NETTESTSUITE} pam perl python parselogs ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-core-image-lsb-sdk = "${NETTESTSUITE} buildcpio buildlzip buildgalculator \
-    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE}"
+    connman ${DEVTESTSUITE} logrotate pam parselogs perl python ${RPMTESTSUITE} ptest"
 DEFAULT_TEST_SUITES_pn-meta-toolchain = "auto"
 
 # aarch64 has no graphics
-- 
2.15.1

Re: [PATCHv2 2/3] testimage.bbclass: add ptest to the list of runtime tests whenever possible

[Richard Purdie]

On Thu, 2017-12-21 at 14:44 +0200, Alexander Kanavin wrote:
> If no ptest packages are installed in the image, the test does
> nothing;
> if ptest packages are installed in the image, then they should be
> run without user having to enable that manually.
> 
> Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
> ---
>  meta/classes/testimage.bbclass | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)

https://autobuilder.yocto.io/builders/nightly-deb-non-deb/builds/673/steps/Running%20Sanity%20Tests/logs/stdio

Cheers,

Richard

[PATCHv2 3/3] runtime/cases/ptest.py: fail when ptests fail on target

[Alexander Kanavin]

That's the whole point isn't it? Previously this testcase succeeded
even if some of the underlying on-target tests failed; the only way
to find out if anything was wrong was to manually inspect the logs.

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
---
 meta/lib/oeqa/runtime/cases/ptest.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index 63e119530dc..8f208a5ff21 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -83,3 +83,11 @@ class PtestRunnerTest(OERuntimeTestCase):
             # Remove the old link to create a new one
             os.remove(ptest_log_dir_link)
         os.symlink(os.path.basename(ptest_log_dir), ptest_log_dir_link)
+
+        failed_tests = {}
+        for section in parse_result.result_dict:
+            failed_testcases = [ test for test, result in parse_result.result_dict[section] if result == 'fail' ]
+            if failed_testcases:
+                failed_tests[section] = failed_testcases
+
+        self.assertFalse(failed_tests, msg = "Failed ptests: %s" %(str(failed_tests)))
-- 
2.15.1

Re: [PATCHv2 1/3] runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

[Cal Sullivan]

[-- Attachment #1: Type: text/plain, Size: 2749 bytes --]

Looks like it doesn't skip the test correctly:

| NOTE: test_ptestrunner (ptest.PtestRunnerTest) | DEBUG: Checking if 
ptest is in DISTRO_FEATURES or IMAGE_FEATURES | DEBUG: [Running]$ ssh -l 
root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o 
LogLevel=ERROR 192.168.7.2 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; 
which ptest-runner | DEBUG: Data from SSH call: which: no ptest-runner 
in (/usr/sbin:/sbin:/usr/bin:/bin) | DEBUG: [Command returned '1' after 
0.10 seconds] | DEBUG: Command: which ptest-runner | Output: which: no 
ptest-runner in (/usr/sbin:/sbin:/usr/bin:/bin) | | DEBUG: [Running]$ 
ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no 
-o LogLevel=ERROR 192.168.7.2 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; 
ptest-runner | DEBUG: Data from SSH call: sh: ptest-runner: command not 
found | DEBUG: [Command returned '127' after 0.10 seconds] | DEBUG: 
Command: ptest-runner | Output: sh: ptest-runner: command not found | | 
NOTE: ... FAIL 
https://autobuilder.yocto.io/builders/nightly-deb-non-deb/builds/673/steps/Running%20Sanity%20Tests/logs/stdio 
Thanks, Cal

On 12/21/2017 04:44 AM, Alexander Kanavin wrote:
> Doing so means that all available ptests for packages in the image
> will be installed and run; some of them may be broken, never finish,
> take a very long time or simply irrelevant to the user who wants to
> check ptests of only a few specific packages, and does so by listing
> them explicitly via IMAGE_INSTALL_append or similar.
>
> Conversely, do not run the test if ptest-runner is not installed
> (which means there are no ptests available, as they pull it in via a
> dependency).
>
> Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
> ---
>   meta/lib/oeqa/runtime/cases/ptest.py | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
> index ec8c038a566..63e119530dc 100644
> --- a/meta/lib/oeqa/runtime/cases/ptest.py
> +++ b/meta/lib/oeqa/runtime/cases/ptest.py
> @@ -48,9 +48,12 @@ class PtestRunnerTest(OERuntimeTestCase):
>   
>       @OETestID(1600)
>       @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
> -    @skipIfNotFeature('ptest-pkgs', 'Test requires ptest-pkgs to be in IMAGE_FEATURES')
>       @OETestDepends(['ssh.SSHTest.test_ssh'])
>       def test_ptestrunner(self):
> +        status, output = self.target.run('which ptest-runner', 0)
> +        if len(output) == 0:
> +            self.skipTest("No -ptest packages are installed in the image")
> +
>           import datetime
>   
>           test_log_dir = self.td.get('TEST_LOG_DIR', '')


[-- Attachment #2: Type: text/html, Size: 3356 bytes --]

Re: [PATCHv2 1/3] runtime/cases/ptest.py: do not require ptest-pkgs in IMAGE_FEATURES

[Alexander Kanavin]

On 01/03/2018 10:42 PM, Cal Sullivan wrote:
> Looks like it doesn't skip the test correctly:
>> +        status, output = self.target.run('which ptest-runner', 0)
>> +        if len(output) == 0:
>> +            self.skipTest("No -ptest packages are installed in the image")
>> +

Looks like there are no less than three versions of 'which' (busybox, 
debianutils, gnu), and one of them is noisy when the argument is not 
found. I'll fix and resend the patchset.

Alex