[PATCH v3 0/3] ptest-runner 2.0

[PATCH v3 0/3] ptest-runner 2.0

[Aníbal Limón]

The whole set is sent to two ML for reviewing purposes the first two patches are for
oe-core and the last one for poky.

The first two commits (oe-core) adds a new ptest-runner written in python the sh one is conserved
to use in tiny systems.

What's new in ptest runner:

	- Monitor/timeout stdout, stderr of the test suite to avoid block indefinetly.
	- Add option for change ptest root directory.
	- Add option for list available tests.
	- Add option for only run certain tests.

The last commit (meta-yocto) sets ptest-runner to 1.0 in poky-tiny systems due to python dependency.

The changes are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib alimon/ptest-runner
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner

Aníbal Limón (3):
  ptest-runner: Add version 2.0 re-implementation in python.
  ptest-runner: Add a recipe for install ptest-runner 2.0.
  conf/distro/poky-tiny.conf: Add default install of ptest-runner 1.0

 meta-yocto/conf/distro/poky-tiny.conf              |   4 +
 .../ptest-runner/files/ptest-runner_2.0.py         | 162 +++++++++++++++++++++
 .../ptest-runner/ptest-runner_2.0.bb               |  27 ++++
 3 files changed, 193 insertions(+)
 create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb

-- 
2.1.4

[PATCH v3 1/3] ptest-runner: Add version 2.0 re-implementation in python.

[Aníbal Limón]

The new ptest-runner supports timeout of upstream tests executed,
it looks for stdout of process and if no information is available
in certain time (defaults to 5m) the process is treaty as blocked
and ptest-runner kills it, this handles problems of ptest-runner
being blocked indefinitly for upstream test suites.

Also an options was added for list test available, specify directory
of ptests and run only certain tests.

[YOCTO #8021]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../ptest-runner/files/ptest-runner_2.0.py         | 162 +++++++++++++++++++++
 1 file changed, 162 insertions(+)
 create mode 100755 meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py

diff --git a/meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py b/meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
new file mode 100755
index 0000000..2b71970
--- /dev/null
+++ b/meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
@@ -0,0 +1,162 @@
+#!/usr/bin/env python
+
+# Ptest-runner
+#
+# Copyright (C) 2014-2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+import os
+import sys
+import fcntl
+import time
+
+import argparse
+import logging 
+from datetime import datetime
+
+import subprocess
+
+DEFAULT_LIBDIR='/usr/lib'
+DEFAULT_TIMEOUT_SECS = 300
+
+logging.basicConfig(format="%(message)s")
+logger = logging.getLogger()
+logger.setLevel(logging.INFO)
+
+def get_available_ptests(libdir):
+    ptests = {}
+
+    if os.path.exists(libdir) and os.path.isdir(libdir):
+        for d in os.listdir(libdir):
+            full_d = os.path.join(libdir, d)
+            if os.path.isdir(full_d) and not os.path.islink(full_d):
+                run_ptest = os.path.join(full_d, 'ptest', 'run-ptest')
+                if os.path.exists(run_ptest):
+                    ptests[d] = run_ptest
+
+    return ptests
+
+def print_ptests(ptests):
+    if ptests.keys():
+        logger.info("Available ptests:\n")
+        for p in ptests.keys():
+            logger.info("%s\t%s" % (p, ptests[p]))
+        return 0
+    else:
+        logger.error("No ptests found.\n")
+        return 1
+
+def get_ptests_to_run(available_ptests, requested_ptests):
+    if requested_ptests:
+        for rp in requested_ptests:
+            if not rp in available_ptests.keys():
+                logger.error("%s ptest isn't available." % rp)
+                return 1
+
+        for ap in available_ptests.keys():
+            if not ap in requested_ptests:
+                del available_ptests[ap]
+    return 0
+
+def run_ptests(ptests, timeout):
+    def _set_nonblocking(fp):
+        fd = process.stdout.fileno()
+        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
+        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
+
+    def _write_output(stdout):
+        import select
+        data_written = False
+
+        rlist = [stdout.fileno()]
+        (rrlist, _, _) = select.select(rlist, [], [], 0)
+
+        for fd in rrlist:
+            try:
+                sys.stdout.write(stdout.read())
+                data_written = True
+            except:
+                pass
+
+        return data_written
+
+    exit_code = 0
+
+    print "START: %s" % sys.argv[0]
+    for p in ptests.keys():
+        run_ptest = ptests[p]
+
+        print datetime.strftime(datetime.now(), "%Y-%m-%dT%H:%M")
+        print "BEGIN: %s" % run_ptest
+
+        os.chdir(os.path.dirname(run_ptest))
+
+        process = subprocess.Popen([run_ptest], stdout=subprocess.PIPE,
+                stderr=subprocess.STDOUT, shell=True, close_fds=True,
+                bufsize=-1)
+        _set_nonblocking(process.stdout)
+
+        sentinel = datetime.now()
+        while True:
+            if _write_output(process.stdout):
+                sentinel = datetime.now()
+            elif timeout >= 0 and \
+                ((datetime.now() - sentinel).total_seconds() > timeout):
+                    print "\nTIMEOUT: %s" % run_ptest
+                    process.kill()
+                    break
+
+            if process.poll() is not None:
+                break
+
+            time.sleep(0.5)
+
+        if process.returncode:
+            exit_code = process.returncode
+
+        print "END: %s" % run_ptest
+        print datetime.strftime(datetime.now(), "%Y-%m-%dT%H:%M")
+    print "STOP: %s" % sys.argv[0]
+
+    return exit_code
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description="ptest-runner runs ptests from upstream packages",
+            add_help=False,
+            epilog="Use %(prog)s [ptest1 ptest2 ...] --help to get help on a specific command")
+    parser.add_argument('ptests', metavar='ptests', help='ptests to run',
+            nargs='*')
+    parser.add_argument('-d', '--directory', help='Directory for search ptests',
+            action='store', default=DEFAULT_LIBDIR)
+    parser.add_argument('-l', '--list', help='List available ptests',
+            action='store_true')
+    parser.add_argument('-t', '--timeout',
+            help='Timeout (secs) for use when run ptests, -1 waits indefinitely', action='store', type=int,
+            default=DEFAULT_TIMEOUT_SECS)
+    parser.add_argument('-h', '--help', action='help', default=argparse.SUPPRESS,
+                        help='show this help message and exit')
+
+    global_args, unparsed_args = parser.parse_known_args()
+
+    ptests = get_available_ptests(global_args.directory)
+    if global_args.list:
+        exit_code = print_ptests(ptests)
+        sys.exit(exit_code)
+
+    exit_code = get_ptests_to_run(ptests, global_args.ptests)
+    if exit_code:
+        sys.exit(exit_code)
+
+    sys.exit(run_ptests(ptests, global_args.timeout))
-- 
2.1.4

[PATCH v3 2/3] ptest-runner: Add a recipe for install ptest-runner 2.0.

[Aníbal Limón]

[YOCTO #8021]

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 .../ptest-runner/ptest-runner_2.0.bb               | 27 ++++++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb

diff --git a/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb b/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
new file mode 100644
index 0000000..0d0037e
--- /dev/null
+++ b/meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
@@ -0,0 +1,27 @@
+SUMMARY = "A python script to run all installed ptests"
+
+DESCRIPTION = "The ptest-runner package installs a ptest-runner \
+python script which loops through all installed ptest test suites and \
+runs them in sequence."
+
+HOMEPAGE = "https://wiki.yoctoproject.org/wiki/Ptest"
+SRC_URI += "file://ptest-runner_2.0.py"
+
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=4d92cd373abda3937c2bc47fbc49d690 \
+                    file://${COREBASE}/meta/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+INHIBIT_DEFAULT_DEPS = "1"
+RDEPENDS_${PN} = "python python-fcntl python-argparse python-logging python-datetime \
+                  python-subprocess"
+
+S = "${WORKDIR}"
+
+do_install () {
+    install -D -m 0755 ${WORKDIR}/ptest-runner_2.0.py ${D}${bindir}/ptest-runner
+}
+
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_build[noexec] = "1"
-- 
2.1.4

[PATCH v3 3/3] conf/distro/poky-tiny.conf: Add default install of ptest-runner 1.0

[Aníbal Limón]

ptest-runner 2.0 depends on python that is too big for poky tiny distro,
so install the ptest-runner 1.0 that only depends on shell.

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
---
 meta-yocto/conf/distro/poky-tiny.conf | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta-yocto/conf/distro/poky-tiny.conf b/meta-yocto/conf/distro/poky-tiny.conf
index b0227de..6bc4e15 100644
--- a/meta-yocto/conf/distro/poky-tiny.conf
+++ b/meta-yocto/conf/distro/poky-tiny.conf
@@ -39,6 +39,10 @@ DISTRO = "poky-tiny"
 PREFERRED_PROVIDER_virtual/kernel = "linux-yocto-tiny"
 PREFERRED_VERSION_linux-yocto-tiny ?= "3.19%"
 
+# ptest-runner 2.0 depends on python that is too big for poky tiny distro, so install
+# the ptest-runner 1.0 that only depends on shell
+PREFERRED_VERSION_ptest-runner ?= "1.0"
+
 # We can use packagegroup-core-boot, but in the future we may need a new packagegroup-core-tiny
 #POKY_DEFAULT_EXTRA_RDEPENDS += "packagegroup-core-boot"
 # Drop kernel-module-af-packet from RRECOMMENDS
-- 
2.1.4

Re: [PATCH v3 0/3] ptest-runner 2.0

[Tudor Florea]

Hi Anibal,
Please see my comments inline.
Regards,
  Tudor.

On 04/01/2016 20:29, Aníbal Limón wrote:
> The whole set is sent to two ML for reviewing purposes the first
> two patches are for oe-core and the last one for poky.
> 
> The first two commits (oe-core) adds a new ptest-runner written in
> python the sh one is conserved to use in tiny systems.
> 
> What's new in ptest runner:
> 
> - Monitor/timeout stdout, stderr of the test suite to avoid block
> indefinetly.
This is definitely something useful.
> - Add option for change ptest root directory. - Add option for list
> available tests. - Add option for only run certain tests.
> 
> The last commit (meta-yocto) sets ptest-runner to 1.0 in poky-tiny
> systems due to python dependency.
I think adding python dependency is a shift in a wrong direction for
for ptest-runner. The assumption that most embedded devices (tiny or
not) have python might not be correct.
The alternative of using the old version of ptest-runner only
complicate things.
I do think this kind of work is really useful but this should be done
into a testing framework that runs outside of the DUT.
> 
> The changes are available in the git repository at:
> 
> git://git.yoctoproject.org/poky-contrib alimon/ptest-runner 
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner
>
>  Aníbal Limón (3): ptest-runner: Add version 2.0 re-implementation
> in python. ptest-runner: Add a recipe for install ptest-runner
> 2.0. conf/distro/poky-tiny.conf: Add default install of
> ptest-runner 1.0
> 
> meta-yocto/conf/distro/poky-tiny.conf              |   4 + 
> .../ptest-runner/files/ptest-runner_2.0.py         | 162
> +++++++++++++++++++++ .../ptest-runner/ptest-runner_2.0.bb
> |  27 ++++ 3 files changed, 193 insertions(+) create mode 100755
> meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py create
> mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
> 

Re: [PATCH v3 0/3] ptest-runner 2.0

[Aníbal Limón]

Hi Todor,

Comments below,
	alimon


On 01/05/2016 06:13 PM, Tudor Florea wrote:
> Hi Anibal,
> Please see my comments inline.
> Regards,
>   Tudor.
> 
> On 04/01/2016 20:29, Aníbal Limón wrote:
>> The whole set is sent to two ML for reviewing purposes the first
>> two patches are for oe-core and the last one for poky.
>>
>> The first two commits (oe-core) adds a new ptest-runner written in
>> python the sh one is conserved to use in tiny systems.
>>
>> What's new in ptest runner:
>>
>> - Monitor/timeout stdout, stderr of the test suite to avoid block
>> indefinetly.
> This is definitely something useful.
>> - Add option for change ptest root directory. - Add option for list
>> available tests. - Add option for only run certain tests.
>>
>> The last commit (meta-yocto) sets ptest-runner to 1.0 in poky-tiny
>> systems due to python dependency.
> I think adding python dependency is a shift in a wrong direction for
> for ptest-runner. The assumption that most embedded devices (tiny or
> not) have python might not be correct.
> The alternative of using the old version of ptest-runner only
> complicate things.
> I do think this kind of work is really useful but this should be done
> into a testing framework that runs outside of the DUT.

The original problem is the current sh ptest-runner blocks indefinitely
when a package ptest blocks (we don't control this), now python-ptest is
blocking the ptest-runner, see [1]. The way to solve this is adding
timeout in some place.

I know that bash support timeout option but this option works over the
whole process making us to define timeouts too big or per machine/device
also tiny systems mainly don't use bash. That's the reason for i decide
to write python version of ptest-runner that applies a timeout over the
output of the program give us better control/granularity over the
running ptest.


If you now other way to solve the problem please tell us.


[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=8021


>>
>> The changes are available in the git repository at:
>>
>> git://git.yoctoproject.org/poky-contrib alimon/ptest-runner 
>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner
>>
>>  Aníbal Limón (3): ptest-runner: Add version 2.0 re-implementation
>> in python. ptest-runner: Add a recipe for install ptest-runner
>> 2.0. conf/distro/poky-tiny.conf: Add default install of
>> ptest-runner 1.0
>>
>> meta-yocto/conf/distro/poky-tiny.conf              |   4 + 
>> .../ptest-runner/files/ptest-runner_2.0.py         | 162
>> +++++++++++++++++++++ .../ptest-runner/ptest-runner_2.0.bb
>> |  27 ++++ 3 files changed, 193 insertions(+) create mode 100755
>> meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py create
>> mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
>>

Re: [PATCH v3 0/3] ptest-runner 2.0

[Aníbal Limón]

Hi again Tudor,

Comments below,

	alimon



On 01/06/2016 09:46 AM, Aníbal Limón wrote:
> Hi Todor,
> 
> Comments below,
> 	alimon
> 
> 
> On 01/05/2016 06:13 PM, Tudor Florea wrote:
>> Hi Anibal,
>> Please see my comments inline.
>> Regards,
>>   Tudor.
>>
>> On 04/01/2016 20:29, Aníbal Limón wrote:
>>> The whole set is sent to two ML for reviewing purposes the first
>>> two patches are for oe-core and the last one for poky.
>>>
>>> The first two commits (oe-core) adds a new ptest-runner written in
>>> python the sh one is conserved to use in tiny systems.
>>>
>>> What's new in ptest runner:
>>>
>>> - Monitor/timeout stdout, stderr of the test suite to avoid block
>>> indefinetly.
>> This is definitely something useful.
>>> - Add option for change ptest root directory. - Add option for list
>>> available tests. - Add option for only run certain tests.
>>>
>>> The last commit (meta-yocto) sets ptest-runner to 1.0 in poky-tiny
>>> systems due to python dependency.
>> I think adding python dependency is a shift in a wrong direction for
>> for ptest-runner. The assumption that most embedded devices (tiny or
>> not) have python might not be correct.
>> The alternative of using the old version of ptest-runner only
>> complicate things.
>> I do think this kind of work is really useful but this should be done
>> into a testing framework that runs outside of the DUT.
> 
> The original problem is the current sh ptest-runner blocks indefinitely
> when a package ptest blocks (we don't control this), now python-ptest is
> blocking the ptest-runner, see [1]. The way to solve this is adding
> timeout in some place.
> 
> I know that bash support timeout option but this option works over the
> whole process making us to define timeouts too big or per machine/device
> also tiny systems mainly don't use bash. That's the reason for i decide
> to write python version of ptest-runner that applies a timeout over the
> output of the program give us better control/granularity over the
> running ptest.
> 
> 
> If you now other way to solve the problem please tell us.

I was thinking and one possible solution for this will be implement in C
the ptest-runner with the same features of python version, any comment?


> 
> 
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=8021
> 
> 
>>>
>>> The changes are available in the git repository at:
>>>
>>> git://git.yoctoproject.org/poky-contrib alimon/ptest-runner 
>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner
>>>
>>>  Aníbal Limón (3): ptest-runner: Add version 2.0 re-implementation
>>> in python. ptest-runner: Add a recipe for install ptest-runner
>>> 2.0. conf/distro/poky-tiny.conf: Add default install of
>>> ptest-runner 1.0
>>>
>>> meta-yocto/conf/distro/poky-tiny.conf              |   4 + 
>>> .../ptest-runner/files/ptest-runner_2.0.py         | 162
>>> +++++++++++++++++++++ .../ptest-runner/ptest-runner_2.0.bb
>>> |  27 ++++ 3 files changed, 193 insertions(+) create mode 100755
>>> meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py create
>>> mode 100644 meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
>>>

Re: [PATCH v3 0/3] ptest-runner 2.0

[Tudor Florea]

Hi Alimon
See inline.
Tudor.

On 06/01/2016 18:07, Aníbal Limón wrote:
> Hi again Tudor,
> 
> Comments below,
> 
> alimon
> 
> 
> 
> On 01/06/2016 09:46 AM, Aníbal Limón wrote:
>> Hi Todor,
>> 
>> Comments below, alimon
>> 
>> 
>> On 01/05/2016 06:13 PM, Tudor Florea wrote:
>>> Hi Anibal, Please see my comments inline. Regards, Tudor.
>>> 
>>> On 04/01/2016 20:29, Aníbal Limón wrote:
>>>> The whole set is sent to two ML for reviewing purposes the
>>>> first two patches are for oe-core and the last one for poky.
>>>> 
>>>> The first two commits (oe-core) adds a new ptest-runner
>>>> written in python the sh one is conserved to use in tiny
>>>> systems.
>>>> 
>>>> What's new in ptest runner:
>>>> 
>>>> - Monitor/timeout stdout, stderr of the test suite to avoid
>>>> block indefinetly.
>>> This is definitely something useful.
>>>> - Add option for change ptest root directory. - Add option
>>>> for list available tests. - Add option for only run certain
>>>> tests.
>>>> 
>>>> The last commit (meta-yocto) sets ptest-runner to 1.0 in
>>>> poky-tiny systems due to python dependency.
>>> I think adding python dependency is a shift in a wrong
>>> direction for for ptest-runner. The assumption that most
>>> embedded devices (tiny or not) have python might not be
>>> correct. The alternative of using the old version of
>>> ptest-runner only complicate things. I do think this kind of
>>> work is really useful but this should be done into a testing
>>> framework that runs outside of the DUT.
>> 
>> The original problem is the current sh ptest-runner blocks
>> indefinitely when a package ptest blocks (we don't control this),
>> now python-ptest is blocking the ptest-runner, see [1]. The way
>> to solve this is adding timeout in some place.
>> 
>> I know that bash support timeout option but this option works
>> over the whole process making us to define timeouts too big or
>> per machine/device also tiny systems mainly don't use bash.
>> That's the reason for i decide to write python version of
>> ptest-runner that applies a timeout over the output of the
>> program give us better control/granularity over the running
>> ptest.
>> 
>> 
>> If you now other way to solve the problem please tell us.
We may have to use our own version of timeout as we cannot rely on
bash either. A good starting point is here:
http://www.pixelbeat.org/scripts/timeout

I'm thinking this is a good opportunity to improve ptest-runner to run
the package tests in parallel. That is, we could spawn a shell per
each run-ptest (wrapper) that have the output redirected to its own
file  (e.g. /var/log/ptest/{package}/output) and then we only timeout
monitoring the output file.
In future we may eventually mimic the logic of automake parallel test
harness
(https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html)

> 
> I was thinking and one possible solution for this will be implement
> in C the ptest-runner with the same features of python version, any
> comment?
That would be great!
> 
> 
>> 
>> 
>> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=8021
>> 
>> 
>>>> 
>>>> The changes are available in the git repository at:
>>>> 
>>>> git://git.yoctoproject.org/poky-contrib alimon/ptest-runner 
>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner
>>>>
>>>>
>>>> 
Aníbal Limón (3): ptest-runner: Add version 2.0 re-implementation
>>>> in python. ptest-runner: Add a recipe for install
>>>> ptest-runner 2.0. conf/distro/poky-tiny.conf: Add default
>>>> install of ptest-runner 1.0
>>>> 
>>>> meta-yocto/conf/distro/poky-tiny.conf              |   4 + 
>>>> .../ptest-runner/files/ptest-runner_2.0.py         | 162 
>>>> +++++++++++++++++++++ .../ptest-runner/ptest-runner_2.0.bb |
>>>> 27 ++++ 3 files changed, 193 insertions(+) create mode
>>>> 100755 
>>>> meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
>>>> create mode 100644
>>>> meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
>>>>