Re: [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution

[Fam Zheng <famz@redhat.com>]

On Fri, 08/08 12:58, M.Kustova wrote:
> On Fri, Aug 8, 2014 at 12:50 PM, Fam Zheng <famz@redhat.com> wrote:
> > On Wed, 08/06 17:12, Maria Kustova wrote:
> >> The purpose of the test runner is to prepare the test environment (e.g. create
> >> a work directory, a test image, etc), execute a program under test with
> >> parameters, indicate a test failure if the program was killed during the test
> >> execution and collect core dumps, logs and other test artifacts.
> >>
> >> The test runner doesn't depend on an image format or a program will be tested,
> >> so it can be used with any external image generator and program under test.
> >>
> >> Signed-off-by: Maria Kustova <maria.k@catit.be>
> >> ---
> >>  tests/image-fuzzer/runner/runner.py | 405 ++++++++++++++++++++++++++++++++++++
> >>  1 file changed, 405 insertions(+)
> >>  create mode 100755 tests/image-fuzzer/runner/runner.py
> >>
> >> diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
> >> new file mode 100755
> >> index 0000000..3fa7fca
> >> --- /dev/null
> >> +++ b/tests/image-fuzzer/runner/runner.py
> >> @@ -0,0 +1,405 @@
> >> +#!/usr/bin/env python
> >> +
> >> +# Tool for running fuzz tests
> >> +#
> >> +# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
> >> +#
> >> +# This program is free software: you can redistribute it and/or modify
> >> +# it under the terms of the GNU General Public License as published by
> >> +# the Free Software Foundation, either version 2 of the License, or
> >> +# (at your option) any later version.
> >> +#
> >> +# This program is distributed in the hope that it will be useful,
> >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >> +# GNU General Public License for more details.
> >> +#
> >> +# You should have received a copy of the GNU General Public License
> >> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> >> +#
> >> +
> >> +import sys
> >> +import os
> >> +import signal
> >> +import subprocess
> >> +import random
> >> +import shutil
> >> +from itertools import count
> >> +import getopt
> >> +import StringIO
> >> +import resource
> >> +
> >> +try:
> >> +    import json
> >> +except ImportError:
> >> +    try:
> >> +        import simplejson as json
> >> +    except ImportError:
> >> +        print >>sys.stderr, \
> >> +            "Warning: Module for JSON processing is not found.\n" \
> >> +            "'--config' and '--command' options are not supported."
> >> +
> >> +# Backing file sizes in MB
> >> +MAX_BACKING_FILE_SIZE = 10
> >> +MIN_BACKING_FILE_SIZE = 1
> >> +
> >> +
> >> +def multilog(msg, *output):
> >> +    """ Write an object to all of specified file descriptors."""
> >> +    for fd in output:
> >> +        fd.write(msg)
> >> +        fd.flush()
> >> +
> >> +
> >> +def str_signal(sig):
> >> +    """ Convert a numeric value of a system signal to the string one
> >> +    defined by the current operational system.
> >> +    """
> >> +    for k, v in signal.__dict__.items():
> >> +        if v == sig:
> >> +            return k
> >> +
> >> +
> >> +def run_app(fd, q_args):
> >> +    """Start an application with specified arguments and return its exit code
> >> +    or kill signal depending on the result of execution.
> >> +    """
> >> +    devnull = open('/dev/null', 'r+')
> >> +    process = subprocess.Popen(q_args, stdin=devnull,
> >> +                               stdout=subprocess.PIPE,
> >> +                               stderr=subprocess.PIPE)
> >> +    out, err = process.communicate()
> >> +    fd.write(out)
> >> +    fd.write(err)
> >> +    return process.returncode
> >> +
> >> +
> >> +class TestException(Exception):
> >> +    """Exception for errors risen by TestEnv objects."""
> >> +    pass
> >> +
> >> +
> >> +class TestEnv(object):
> >> +
> >> +    """Test object.
> >> +
> >> +    The class sets up test environment, generates backing and test images
> >> +    and executes application under tests with specified arguments and a test
> >> +    image provided.
> >> +
> >> +    All logs are collected.
> >> +
> >> +    The summary log will contain short descriptions and statuses of tests in
> >> +    a run.
> >> +
> >> +    The test log will include application (e.g. 'qemu-img') logs besides info
> >> +    sent to the summary log.
> >> +    """
> >> +
> >> +    def __init__(self, test_id, seed, work_dir, run_log,
> >> +                 cleanup=True, log_all=False):
> >> +        """Set test environment in a specified work directory.
> >> +
> >> +        Path to qemu-img and qemu-io will be retrieved from 'QEMU_IMG' and
> >> +        'QEMU_IO' environment variables.
> >> +        """
> >> +        if seed is not None:
> >> +            self.seed = seed
> >> +        else:
> >> +            self.seed = str(random.randint(0, sys.maxint))
> >> +        random.seed(self.seed)
> >> +
> >> +        self.init_path = os.getcwd()
> >> +        self.work_dir = work_dir
> >> +        self.current_dir = os.path.join(work_dir, 'test-' + test_id)
> >> +        self.qemu_img = os.environ.get('QEMU_IMG', 'qemu-img')\
> >> +                                  .strip().split(' ')
> >
> > Nitpicking. I think split(' ') doesn't make sense, this could instead be:
> >
> >            self.qemu_img = [os.environ.get('QEMU_IMG', 'qemu-img').strip()]
> >
> > Otherwise user won't be able to pass in a QEMU_IMG path with space.
> >
> > Corner case, though. Otherwise looks good,
> 
> This functionality was inherited from qemu-iotests/iotests.py.
> Rationale for this splitting was that environment variables can
> contain call arguments as well.
> Ruining paths with white spaces was a reasonable trade-off.

OK, thanks for explanation.

Fam