[PATCH 0/2] ptest-runner

[PATCH 0/2] ptest-runner

[Aníbal Limón]

These two commits adds a new ptest-runner written in python the sh one is conserved
to use in tiny systems.

What's new in ptest runner:

	- Monitor/timeout stdout, stderr of the test suite to avoid block indefinetly.
	- Add option for change ptest root directory.
	- Add option for list available tests.
	- Add option for only run certain tests.

The following changes since commit d9bc1f4b9a8aff859c59dbf9bbdd6b16e99381b2:

  classes/sstate.bbclass: checkhashes modify to take into account siginfo. (2015-12-02 10:55:25 -0600)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib alimon/ptest-runner
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner

Aníbal Limón (2):
  ptest-runner: Add version 2.0 re-implementation in python.
  ptest-runner: Add a recipe for install ptest-runner 2.0.

 .../ptest-runner/files/ptest-runner_2.0.py         | 144 +++++++++++++++++++++
 .../ptest-runner/ptest-runner_2.0.bb               |  28 ++++
 2 files changed, 172 insertions(+)
 create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb

-- 
2.1.4

[PATCH 1/2] ptest-runner: Add version 2.0 re-implementation in python.

[Aníbal Limón]

The new ptest-runner supports timeout of upstream tests executed,
it looks for stdout of process and if no information is available
in certain time (defaults to 5m) the process is treaty as blocked
and ptest-runner kills it, this handles problems of ptest-runner
being blocked indefinitly for upstream test suites.

Also an options was added for list test available, specify directory
of ptests and run only certain tests.

[YOCTO #8021]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../ptest-runner/files/ptest-runner_2.0.py         | 144 +++++++++++++++++++++
 1 file changed, 144 insertions(+)
 create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py

diff --git a/meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py b/meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
new file mode 100755
index 0000000..fc0447c
--- /dev/null
+++ b/meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+
+# Ptest-runner
+#
+# Copyright (C) 2014-2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import fcntl
+import time
+
+import argparse
+import logging 
+from datetime import datetime
+
+import subprocess
+
+DEFAULT_LIBDIR='/usr/lib'
+DEFAULT_TIMEOUT_SECS = 500
+
+logging.basicConfig(format="%(message)s")
+logger = logging.getLogger()
+logger.setLevel(logging.INFO)
+
+def get_available_ptests(libdir):
+    ptests = {}
+
+    if os.path.exists(libdir) and os.path.isdir(libdir):
+        for d in os.listdir(libdir):
+            full_d = os.path.join(libdir, d)
+            if os.path.isdir(full_d) and not os.path.islink(full_d):
+                run_ptest = os.path.join(full_d, 'ptest', 'run-ptest')
+                if os.path.exists(run_ptest):
+                    ptests[d] = run_ptest
+
+    return ptests
+
+def print_ptests(ptests):
+    if ptests.keys():
+        logger.info("Available ptests:\n")
+        for p in ptests.keys():
+            logger.info("%s\t%s" % (p, ptests[p]))
+        return 0
+    else:
+        logger.error("No ptests found.\n")
+        return 1
+
+def get_ptests_to_run(available_ptests, requested_ptests):
+    if requested_ptests:
+        for rp in requested_ptests:
+            if not rp in available_ptests.keys():
+                logger.error("%s ptest isn't available." % rp)
+                return 1
+
+        for ap in available_ptests.keys():
+            if not ap in requested_ptests:
+                del available_ptests[ap]
+    return 0
+
+def run_ptests(ptests, timeout):
+    def _set_nonblocking(fp):
+        fd = process.stdout.fileno()
+        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
+        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+
+    exit_code = 0
+
+    print "START: %s" % sys.argv[0]
+    for p in ptests.keys():
+        run_ptest = ptests[p]
+
+        print datetime.strftime(datetime.now(), "%Y-%m-%dT%H:%M")
+        print "BEGIN: %s" % run_ptest
+
+        os.chdir(os.path.dirname(run_ptest))
+
+        process = subprocess.Popen([run_ptest], stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE)
+        _set_nonblocking(process.stdout)
+        _set_nonblocking(process.stderr)
+
+        sentinel = datetime.now()
+        while process.poll() is None:
+            try:
+                data = process.stdout.read()
+                sys.stdout.write(data)
+                sentinel = datetime.now()
+            except:
+                if ((datetime.now() - sentinel).total_seconds() > timeout):
+                    print "TIMEOUT: %s" % run_ptest
+                    process.kill()
+                    break
+            time.sleep(0.5)
+
+        if process.returncode:
+            exit_code = process.returncode
+
+        print "END: %s" % run_ptest
+        print datetime.strftime(datetime.now(), "%Y-%m-%dT%H:%M")
+    print "STOP: %s" % sys.argv[0]
+
+    return exit_code
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description="ptest-runner runs ptests from upstream packages",
+            add_help=False,
+            epilog="Use %(prog)s [ptest1 ptest2 ...] --help to get help on a specific command")
+    parser.add_argument('ptests', metavar='ptests', help='ptests to run',
+            nargs='*')
+    parser.add_argument('-d', '--directory', help='Directory for search ptests',
+            action='store', default=DEFAULT_LIBDIR)
+    parser.add_argument('-l', '--list', help='List available ptests',
+            action='store_true')
+    parser.add_argument('-t', '--timeout',
+            help='Timeout (secs) for use when run ptests', action='store',
+            default=DEFAULT_TIMEOUT_SECS)
+    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
+                        help='show this help message and exit')
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    ptests = get_available_ptests(global_args.directory)
+    if global_args.list:
+        exit_code = print_ptests(ptests)
+        sys.exit(exit_code)
+
+    exit_code = get_ptests_to_run(ptests, global_args.ptests)
+    if exit_code:
+        sys.exit(exit_code)
+
+    sys.exit(run_ptests(ptests, global_args.timeout))
-- 
2.1.4

Re: [PATCH 1/2] ptest-runner: Add version 2.0 re-implementation in python.

[Nathan Lynch]

On 12/07/2015 03:26 PM, Aníbal Limón wrote:
> The new ptest-runner supports timeout of upstream tests executed,
> it looks for stdout of process and if no information is available
> in certain time (defaults to 5m) the process is treaty as blocked
> and ptest-runner kills it, this handles problems of ptest-runner
> being blocked indefinitly for upstream test suites.

Having the option to specify a timeout is obviously useful, but please
add the ability to wait indefinitely, and consider making this the
default behavior.  The appropriate value for a timeout, if any, will
always be a function of the particular circumstances of the test run.
An indefinite wait at least gives one the opportunity to investigate a
misbehaving test without racing the test harness.

> +DEFAULT_TIMEOUT_SECS = 500

The change description says the default is five minutes, but this is not
five minutes :-)

Re: [PATCH 1/2] ptest-runner: Add version 2.0 re-implementation in python.

[Aníbal Limón]

On 12/11/2015 05:23 PM, Nathan Lynch wrote:
> On 12/07/2015 03:26 PM, Aníbal Limón wrote:
>> The new ptest-runner supports timeout of upstream tests executed,
>> it looks for stdout of process and if no information is available
>> in certain time (defaults to 5m) the process is treaty as blocked
>> and ptest-runner kills it, this handles problems of ptest-runner
>> being blocked indefinitly for upstream test suites.
> 
> Having the option to specify a timeout is obviously useful, but please
> add the ability to wait indefinitely, and consider making this the
> default behavior.  The appropriate value for a timeout, if any, will
> always be a function of the particular circumstances of the test run.
> An indefinite wait at least gives one the opportunity to investigate a
> misbehaving test without racing the test harness.

I'll add the ability to wait indefinitely adding a -1 to timeout option
but for default i think is better to wait certain time by default
because when ptest-runner is used in automatically manner
(i.e.testimage) this cause the block indefinitely.

> 
>> +DEFAULT_TIMEOUT_SECS = 500
> 
> The change description says the default is five minutes, but this is not
> five minutes :-)

Thanks for notice it, it was a mistake :).

> 
> 

[PATCH 2/2] ptest-runner: Add a recipe for install ptest-runner 2.0.

[Aníbal Limón]

[YOCTO #8021]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../ptest-runner/ptest-runner_2.0.bb               | 28 ++++++++++++++++++++++
 1 file changed, 28 insertions(+)
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb

diff --git a/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb b/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
new file mode 100644
index 0000000..64fd0df
--- /dev/null
+++ b/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
@@ -0,0 +1,28 @@
+SUMMARY = "A python script to run all installed ptests"
+
+DESCRIPTION = "The ptest-runner package installs a ptest-runner \
+python script which loops through all installed ptest test suites and \
+runs them in sequence."
+
+HOMEPAGE = "https://wiki.yoctoproject.org/wiki/Ptest"
+SRC_URI += "file://ptest-runner_2.0.py"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \
+                    file://${COREBASE}/meta/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+INHIBIT_DEFAULT_DEPS = "1"
+RDEPENDS_${PN} = "python python-fcntl python-argparse python-logging python-datetime \
+                  python-subprocess"
+
+S = "${WORKDIR}"
+
+do_install () {
+    mkdir -p ${D}${bindir}
+    install -m 0755 ${WORKDIR}/ptest-runner_2.0.py ${D}${bindir}/ptest-runner
+}
+
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_build[noexec] = "1"
-- 
2.1.4

Re: [PATCH 0/2] ptest-runner

[Aníbal Limón]

ping

On 12/07/2015 03:26 PM, Aníbal Limón wrote:
> These two commits adds a new ptest-runner written in python the sh one is conserved
> to use in tiny systems.
> 
> What's new in ptest runner:
> 
> 	- Monitor/timeout stdout, stderr of the test suite to avoid block indefinetly.
> 	- Add option for change ptest root directory.
> 	- Add option for list available tests.
> 	- Add option for only run certain tests.
> 
> The following changes since commit d9bc1f4b9a8aff859c59dbf9bbdd6b16e99381b2:
> 
>   classes/sstate.bbclass: checkhashes modify to take into account siginfo. (2015-12-02 10:55:25 -0600)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib alimon/ptest-runner
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner
> 
> Aníbal Limón (2):
>   ptest-runner: Add version 2.0 re-implementation in python.
>   ptest-runner: Add a recipe for install ptest-runner 2.0.
> 
>  .../ptest-runner/files/ptest-runner_2.0.py         | 144 +++++++++++++++++++++
>  .../ptest-runner/ptest-runner_2.0.bb               |  28 ++++
>  2 files changed, 172 insertions(+)
>  create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
>  create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
>