[Qemu-devel] [PATCH V2 0/2] runner: Control test duration

[Qemu-devel] [PATCH V2 0/2] runner: Control test duration

[Maria Kustova]

The first patch adds the '--duration SECONDS' argument. After the specified
duration the runner allows to end the current test and then exits.

The second patch adds forced termination of a program under test, if the test
execution takes more than 10 minutes to indicate program freezes.

If a program under test hangs, then the specified test duration can be overrun
up to 10 minutes.

The patch series is based on https://github.com/stefanha/qemu/commits/block,
commit 07a45925fa88376f8583a333e74f7eeb0f455685

v1 -> v2:
 * Trivial fixes based on the review of Fam Zheng
 * Increased time-out (in some cases 5 minutes interval returned false
   negatives)

Maria Kustova (2):
  runner: Add an argument for test duration
  runner: Kill a program under test by time-out

 tests/image-fuzzer/runner.py | 50 +++++++++++++++++++++++++++++++++++++-------
 1 file changed, 42 insertions(+), 8 deletions(-)

-- 
1.9.3

[Qemu-devel] [PATCH V2 1/2] runner: Add an argument for test duration

[Maria Kustova]

After the specified duration the runner stops executing new tests, but it
doesn't interrupt running ones.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/runner.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index 3fa7fca..ea9916b 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -25,6 +25,7 @@ import subprocess
 import random
 import shutil
 from itertools import count
+import time
 import getopt
 import StringIO
 import resource
@@ -269,6 +270,7 @@ if __name__ == '__main__':
 
         Optional arguments:
           -h, --help                    display this help and exit
+          -d, --duration=NUMBER         finish tests after NUMBER of seconds
           -c, --command=JSON            run tests for all commands specified in
                                         the JSON array
           -s, --seed=STRING             seed for a test image generation,
@@ -325,10 +327,15 @@ if __name__ == '__main__':
         finally:
             test.finish()
 
+    def should_continue(duration, start_time):
+        """Return True if a new test can be started and False otherwise."""
+        current_time = int(time.time())
+        return (duration is None) or (current_time - start_time < duration)
+
     try:
-        opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hs:kv',
+        opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hs:kvd:',
                                        ['command=', 'help', 'seed=', 'config=',
-                                        'keep_passed', 'verbose'])
+                                        'keep_passed', 'verbose', 'duration='])
     except getopt.error, e:
         print >>sys.stderr, \
             "Error: %s\n\nTry 'runner.py --help' for more information" % e
@@ -339,6 +346,8 @@ if __name__ == '__main__':
     log_all = False
     seed = None
     config = None
+    duration = None
+
     for opt, arg in opts:
         if opt in ('-h', '--help'):
             usage()
@@ -357,6 +366,8 @@ if __name__ == '__main__':
             log_all = True
         elif opt in ('-s', '--seed'):
             seed = arg
+        elif opt in ('-d', '--duration'):
+            duration = int(arg)
         elif opt == '--config':
             try:
                 config = json.loads(arg)
@@ -394,9 +405,11 @@ if __name__ == '__main__':
     resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
     # If a seed is specified, only one test will be executed.
     # Otherwise runner will terminate after a keyboard interruption
-    for test_id in count(1):
+    start_time = int(time.time())
+    test_id = count(1)
+    while should_continue(duration, start_time):
         try:
-            run_test(str(test_id), seed, work_dir, run_log, cleanup,
+            run_test(str(test_id.next()), seed, work_dir, run_log, cleanup,
                      log_all, command, config)
         except (KeyboardInterrupt, SystemExit):
             sys.exit(1)
-- 
1.9.3

[Qemu-devel] [PATCH V2 2/2] runner: Kill a program under test by time-out

[Maria Kustova]

If a program under test get frozen, the test should finish and report about its
failure.
In such cases the runner waits for 10 minutes until the program ends its
execution. After this time-out the program will be terminated and the test will
be marked as failed.

For current limitation of test image size to 10 MB as a maximum an execution of
each command takes about several seconds in general, so 10 minutes is enough to
discriminate freeze, but not drastically increase an overall test duration.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/runner.py | 29 +++++++++++++++++++++++++----
 1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/tests/image-fuzzer/runner.py b/tests/image-fuzzer/runner.py
index ea9916b..2e1bd51 100755
--- a/tests/image-fuzzer/runner.py
+++ b/tests/image-fuzzer/runner.py
@@ -65,14 +65,35 @@ def run_app(fd, q_args):
     """Start an application with specified arguments and return its exit code
     or kill signal depending on the result of execution.
     """
+
+    class Alarm(Exception):
+        """Exception for signal.alarm events."""
+        pass
+
+    def handler(*arg):
+        """Notify that an alarm event occurred."""
+        raise Alarm
+
+    signal.signal(signal.SIGALRM, handler)
+    signal.alarm(600)
+    term_signal = signal.SIGKILL
     devnull = open('/dev/null', 'r+')
     process = subprocess.Popen(q_args, stdin=devnull,
                                stdout=subprocess.PIPE,
                                stderr=subprocess.PIPE)
-    out, err = process.communicate()
-    fd.write(out)
-    fd.write(err)
-    return process.returncode
+    try:
+        out, err = process.communicate()
+        signal.alarm(0)
+        fd.write(out)
+        fd.write(err)
+        fd.flush()
+        return process.returncode
+
+    except Alarm:
+        os.kill(process.pid, term_signal)
+        fd.write('The command was terminated by timeout.\n')
+        fd.flush()
+        return -term_signal
 
 
 class TestException(Exception):
-- 
1.9.3

Re: [Qemu-devel] [PATCH V2 0/2] runner: Control test duration

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1088 bytes --]

On Tue, Aug 19, 2014 at 12:02:33AM +0400, Maria Kustova wrote:
> The first patch adds the '--duration SECONDS' argument. After the specified
> duration the runner allows to end the current test and then exits.
> 
> The second patch adds forced termination of a program under test, if the test
> execution takes more than 10 minutes to indicate program freezes.
> 
> If a program under test hangs, then the specified test duration can be overrun
> up to 10 minutes.
> 
> The patch series is based on https://github.com/stefanha/qemu/commits/block,
> commit 07a45925fa88376f8583a333e74f7eeb0f455685
> 
> v1 -> v2:
>  * Trivial fixes based on the review of Fam Zheng
>  * Increased time-out (in some cases 5 minutes interval returned false
>    negatives)
> 
> Maria Kustova (2):
>   runner: Add an argument for test duration
>   runner: Kill a program under test by time-out
> 
>  tests/image-fuzzer/runner.py | 50 +++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 42 insertions(+), 8 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [PATCH V2 0/2] runner: Control test duration

[Kevin Wolf]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

Am 19.08.2014 um 11:18 hat Stefan Hajnoczi geschrieben:
> On Tue, Aug 19, 2014 at 12:02:33AM +0400, Maria Kustova wrote:
> > The first patch adds the '--duration SECONDS' argument. After the specified
> > duration the runner allows to end the current test and then exits.
> > 
> > The second patch adds forced termination of a program under test, if the test
> > execution takes more than 10 minutes to indicate program freezes.
> > 
> > If a program under test hangs, then the specified test duration can be overrun
> > up to 10 minutes.
> > 
> > The patch series is based on https://github.com/stefanha/qemu/commits/block,
> > commit 07a45925fa88376f8583a333e74f7eeb0f455685
> > 
> > v1 -> v2:
> >  * Trivial fixes based on the review of Fam Zheng
> >  * Increased time-out (in some cases 5 minutes interval returned false
> >    negatives)
> > 
> > Maria Kustova (2):
> >   runner: Add an argument for test duration
> >   runner: Kill a program under test by time-out
> > 
> >  tests/image-fuzzer/runner.py | 50 +++++++++++++++++++++++++++++++++++++-------
> >  1 file changed, 42 insertions(+), 8 deletions(-)
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

Thanks, applied all to the block branch.

Kevin

[-- Attachment #2: Type: application/pgp-signature, Size: 836 bytes --]