* [Qemu-devel] [PATCH V5 1/5] docs: Specification for the image fuzzer
2014-08-06 13:12 [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Maria Kustova
@ 2014-08-06 13:12 ` Maria Kustova
2014-08-08 6:17 ` Stefan Hajnoczi
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution Maria Kustova
` (4 subsequent siblings)
5 siblings, 1 reply; 16+ messages in thread
From: Maria Kustova @ 2014-08-06 13:12 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, famz, Maria Kustova, stefanha
'Overall fuzzer requirements' chapter contains the current product vision and
features done and to be done. This chapter is still in progress.
Signed-off-by: Maria Kustova <maria.k@catit.be>
---
tests/image-fuzzer/docs/image-fuzzer.txt | 239 +++++++++++++++++++++++++++++++
1 file changed, 239 insertions(+)
create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt
diff --git a/tests/image-fuzzer/docs/image-fuzzer.txt b/tests/image-fuzzer/docs/image-fuzzer.txt
new file mode 100644
index 0000000..efe0ed4
--- /dev/null
+++ b/tests/image-fuzzer/docs/image-fuzzer.txt
@@ -0,0 +1,239 @@
+# Specification for the fuzz testing tool
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+Image fuzzer
+============
+
+Description
+-----------
+
+The goal of the image fuzzer is to catch crashes of qemu-io/qemu-img
+by providing to them randomly corrupted images.
+Test images are generated from scratch and have valid inner structure with some
+elements, e.g. L1/L2 tables, having random invalid values.
+
+
+Test runner
+-----------
+
+The test runner generates test images, executes tests utilizing generated
+images, indicates their results and collects all test related artifacts (logs,
+core dumps, test images).
+The test means execution of all available commands under test with the same
+generated test image.
+By default, the test runner generates new tests and executes them until
+keyboard interruption. But if a test seed is specified via the '--seed' runner
+parameter, then only one test with this seed will be executed, after its finish
+the runner will exit.
+
+The runner uses an external image fuzzer to generate test images. An image
+generator should be specified as a mandatory parameter of the test runner.
+Details about interactions between the runner and fuzzers see "Module
+interfaces".
+
+The runner activates generation of core dumps during test executions, but it
+assumes that core dumps will be generated in the current working directory.
+For comprehensive test results, please, set up your test environment
+properly.
+
+Paths to binaries under test (SUTs) qemu-img and qemu-io are retrieved from
+environment variables. If the environment check fails the runner will
+use SUTs installed in system paths.
+qemu-img is required for creation of backing files, so it's mandatory to set
+the related environment variable if it's not installed in the system path.
+For details about environment variables see qemu-iotests/check.
+
+The runner accepts via the '--config' argument a JSON array of fields expected
+to be fuzzed, e.g.
+
+ '[["feature_name_table"], ["header", "l1_table_offset"]]'
+
+Each sublist can be have one or two strings defining image structure elements.
+In the latter case a parent element should be placed on the first position,
+and a field name on the second one.
+
+The runner accepts a list of commands under test as a JSON array via
+the '--command' argument. Each command is a list containing a SUT and all its
+arguments, e.g.
+
+ runner.py -c '[["qemu-io", "$test_img", "-c", "write $off $len"]]'
+ /tmp/test ../qcow2
+
+For variable arguments next aliases can be used:
+ - $test_img for a fuzzed img
+ - $off for an offset in the fuzzed image
+ - $len for a data size
+
+Values for last two aliases will be generated based on the size of the virtal
+disk in the fuzzed image.
+In case when no commands are specified the runner will execute commands from
+the default list:
+ - qemu-img check
+ - qemu-img info
+ - qemu-img convert
+ - qemu-io -c read
+ - qemu-io -c write
+ - qemu-io -c aio_read
+ - qemu-io -c aio_write
+ - qemu-io -c flush
+ - qemu-io -c discard
+ - qemu-io -c truncate
+
+
+Qcow2 image generator
+---------------------
+
+The 'qcow2' generator is a Python package providing 'create_image' method as
+a single public API. See details in 'Test runner/image fuzzer' in 'Module
+interfaces'.
+
+Qcow2 contains two submodules: fuzz.py and layout.py.
+
+'fuzz.py' contains all fuzzing functions, one per image field. It's assumed
+that after code analysis every field will have own constraints for its value.
+For now only universal potentially dangerous values are used, e.g. type limits
+for integers or unsafe symbols as '%s' for strings. For bitmasks random amount
+of bits are set to ones. All fuzzed values are checked on non-equality to the
+current valid value of the field. In case of equality the value will be
+regenerated.
+
+'layout.py' creates a random valid image, fuzzes a random subset of the image
+fields by 'fuzz.py' module and writes a fuzzed image to the file specified.
+If a fuzzer configuration is specified, then it has the next interpretation:
+
+ 1. If a list contains a parent image element only, then some random portion
+ of fields of this element will be fuzzed every test.
+ The same behavior is applied for the entire image if no configuration is
+ used. This case is useful for the test specialization.
+
+ 2. If a list contains a parent element and a field name, then a field
+ will be always fuzzed for every test. This case is useful for regression
+ testing.
+
+For now only header fields and header extensions are generated.
+
+
+Module interfaces
+-----------------
+
+* Test runner/image fuzzer
+
+The runner calls an image generator specifying the path to a test image file,
+path to a backing file and its format and a fuzzer configuration.
+An image generator is expected to provide a
+
+ 'create_image(test_img_path, backing_file_path=None,
+ backing_file_format=None, fuzz_config=None)'
+
+method that creates a test image, writes it to the specified file and returns
+the size of the virtual disk.
+The file should be created if it doesn't exist or overwritten otherwise.
+fuzz_config has a form of a list of lists. Every sublist can have one
+or two elements: first element is a name of a parent image element, second one
+if exists is a name of a field in this element.
+Example,
+ [['header', 'l1_table_offset'],
+ ['header', 'nb_snapshots'],
+ ['feature_name_table']]
+
+Random seed is set by the runner at every test execution for the regression
+purpose, so an image generator is not recommended to modify it internally.
+
+
+Overall fuzzer requirements
+===========================
+
+Input data:
+----------
+
+ - image template (generator)
+ - work directory
+ - action vector (optional)
+ - seed (optional)
+ - SUT and its arguments (optional)
+
+
+Fuzzer requirements:
+-------------------
+
+1. Should be able to inject random data
+2. Should be able to select a random value from the manually pregenerated
+ vector (boundary values, e.g. max/min cluster size)
+3. Image template should describe a general structure invariant for all
+ test images (image format description)
+4. Image template should be autonomous and other fuzzer parts should not
+ rely on it
+5. Image template should contain reference rules (not only block+size
+ description)
+6. Should generate the test image with the correct structure based on an image
+ template
+7. Should accept a seed as an argument (for regression purpose)
+8. Should generate a seed if it is not specified as an input parameter.
+9. The same seed should generate the same image for the same action vector,
+ specified or generated.
+10. Should accept a vector of actions as an argument (for test reproducing and
+ for test case specification, e.g. group of tests for header structure,
+ group of test for snapshots, etc)
+11. Action vector should be randomly generated from the pool of available
+ actions, if it is not specified as an input parameter
+12. Pool of actions should be defined automatically based on an image template
+13. Should accept a SUT and its call parameters as an argument or select them
+ randomly otherwise. As far as it's expected to be rarely changed, the list
+ of all possible test commands can be available in the test runner
+ internally.
+14. Should support an external cancellation of a test run
+15. Seed should be logged (for regression purpose)
+16. All files related to a test result should be collected: a test image,
+ SUT logs, fuzzer logs and crash dumps
+17. Should be compatible with python version 2.4-2.7
+18. Usage of external libraries should be limited as much as possible.
+
+
+Image formats:
+-------------
+
+Main target image format is qcow2, but support of image templates should
+provide an ability to add any other image format.
+
+
+Effectiveness:
+-------------
+
+The fuzzer can be controlled via template, seed and action vector;
+it makes the fuzzer itself invariant to an image format and test logic.
+It should be able to perform rather complex and precise tests, that can be
+specified via an action vector. Otherwise, knowledge about an image structure
+allows the fuzzer to generate the pool of all available areas can be fuzzed
+and randomly select some of them and so compose its own action vector.
+Also complexity of a template defines complexity of the fuzzer, so its
+functionality can be varied from simple model-independent fuzzing to smart
+model-based one.
+
+
+Glossary:
+--------
+
+Action vector is a sequence of structure elements retrieved from an image
+format, each of them will be fuzzed for the test image. It's a subset of
+elements of the action pool. Example: header, refcount table, etc.
+Action pool is all available elements of an image structure that generated
+automatically from an image template.
+Image template is a formal description of an image structure and relations
+between image blocks.
+Test image is an output image of the fuzzer defined by the current seed and
+action vector.
--
1.9.3
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 1/5] docs: Specification for the image fuzzer
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 1/5] docs: Specification for the image fuzzer Maria Kustova
@ 2014-08-08 6:17 ` Stefan Hajnoczi
0 siblings, 0 replies; 16+ messages in thread
From: Stefan Hajnoczi @ 2014-08-08 6:17 UTC (permalink / raw)
To: Maria Kustova; +Cc: kwolf, famz, qemu-devel, stefanha, Maria Kustova
[-- Attachment #1: Type: text/plain, Size: 748 bytes --]
On Wed, Aug 06, 2014 at 05:12:46PM +0400, Maria Kustova wrote:
> 'Overall fuzzer requirements' chapter contains the current product vision and
> features done and to be done. This chapter is still in progress.
>
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
> tests/image-fuzzer/docs/image-fuzzer.txt | 239 +++++++++++++++++++++++++++++++
> 1 file changed, 239 insertions(+)
> create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt
>
> diff --git a/tests/image-fuzzer/docs/image-fuzzer.txt b/tests/image-fuzzer/docs/image-fuzzer.txt
> new file mode 100644
> index 0000000..efe0ed4
> --- /dev/null
> +++ b/tests/image-fuzzer/docs/image-fuzzer.txt
Please put documentation into QEMU's top-level docs/ directory.
[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution
2014-08-06 13:12 [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Maria Kustova
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 1/5] docs: Specification for the image fuzzer Maria Kustova
@ 2014-08-06 13:12 ` Maria Kustova
2014-08-08 6:52 ` Stefan Hajnoczi
2014-08-08 8:50 ` Fam Zheng
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 3/5] fuzz: Fuzzing functions for qcow2 images Maria Kustova
` (3 subsequent siblings)
5 siblings, 2 replies; 16+ messages in thread
From: Maria Kustova @ 2014-08-06 13:12 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, famz, Maria Kustova, stefanha
The purpose of the test runner is to prepare the test environment (e.g. create
a work directory, a test image, etc), execute a program under test with
parameters, indicate a test failure if the program was killed during the test
execution and collect core dumps, logs and other test artifacts.
The test runner doesn't depend on an image format or a program will be tested,
so it can be used with any external image generator and program under test.
Signed-off-by: Maria Kustova <maria.k@catit.be>
---
tests/image-fuzzer/runner/runner.py | 405 ++++++++++++++++++++++++++++++++++++
1 file changed, 405 insertions(+)
create mode 100755 tests/image-fuzzer/runner/runner.py
diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
new file mode 100755
index 0000000..3fa7fca
--- /dev/null
+++ b/tests/image-fuzzer/runner/runner.py
@@ -0,0 +1,405 @@
+#!/usr/bin/env python
+
+# Tool for running fuzz tests
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import sys
+import os
+import signal
+import subprocess
+import random
+import shutil
+from itertools import count
+import getopt
+import StringIO
+import resource
+
+try:
+ import json
+except ImportError:
+ try:
+ import simplejson as json
+ except ImportError:
+ print >>sys.stderr, \
+ "Warning: Module for JSON processing is not found.\n" \
+ "'--config' and '--command' options are not supported."
+
+# Backing file sizes in MB
+MAX_BACKING_FILE_SIZE = 10
+MIN_BACKING_FILE_SIZE = 1
+
+
+def multilog(msg, *output):
+ """ Write an object to all of specified file descriptors."""
+ for fd in output:
+ fd.write(msg)
+ fd.flush()
+
+
+def str_signal(sig):
+ """ Convert a numeric value of a system signal to the string one
+ defined by the current operational system.
+ """
+ for k, v in signal.__dict__.items():
+ if v == sig:
+ return k
+
+
+def run_app(fd, q_args):
+ """Start an application with specified arguments and return its exit code
+ or kill signal depending on the result of execution.
+ """
+ devnull = open('/dev/null', 'r+')
+ process = subprocess.Popen(q_args, stdin=devnull,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE)
+ out, err = process.communicate()
+ fd.write(out)
+ fd.write(err)
+ return process.returncode
+
+
+class TestException(Exception):
+ """Exception for errors risen by TestEnv objects."""
+ pass
+
+
+class TestEnv(object):
+
+ """Test object.
+
+ The class sets up test environment, generates backing and test images
+ and executes application under tests with specified arguments and a test
+ image provided.
+
+ All logs are collected.
+
+ The summary log will contain short descriptions and statuses of tests in
+ a run.
+
+ The test log will include application (e.g. 'qemu-img') logs besides info
+ sent to the summary log.
+ """
+
+ def __init__(self, test_id, seed, work_dir, run_log,
+ cleanup=True, log_all=False):
+ """Set test environment in a specified work directory.
+
+ Path to qemu-img and qemu-io will be retrieved from 'QEMU_IMG' and
+ 'QEMU_IO' environment variables.
+ """
+ if seed is not None:
+ self.seed = seed
+ else:
+ self.seed = str(random.randint(0, sys.maxint))
+ random.seed(self.seed)
+
+ self.init_path = os.getcwd()
+ self.work_dir = work_dir
+ self.current_dir = os.path.join(work_dir, 'test-' + test_id)
+ self.qemu_img = os.environ.get('QEMU_IMG', 'qemu-img')\
+ .strip().split(' ')
+ self.qemu_io = os.environ.get('QEMU_IO', 'qemu-io').strip().split(' ')
+ self.commands = [['qemu-img', 'check', '-f', 'qcow2', '$test_img'],
+ ['qemu-img', 'info', '-f', 'qcow2', '$test_img'],
+ ['qemu-io', '$test_img', '-c', 'read $off $len'],
+ ['qemu-io', '$test_img', '-c', 'write $off $len'],
+ ['qemu-io', '$test_img', '-c',
+ 'aio_read $off $len'],
+ ['qemu-io', '$test_img', '-c',
+ 'aio_write $off $len'],
+ ['qemu-io', '$test_img', '-c', 'flush'],
+ ['qemu-io', '$test_img', '-c',
+ 'discard $off $len'],
+ ['qemu-io', '$test_img', '-c',
+ 'truncate $off']]
+ for fmt in ['raw', 'vmdk', 'vdi', 'cow', 'qcow2', 'file',
+ 'qed', 'vpc']:
+ self.commands.append(
+ ['qemu-img', 'convert', '-f', 'qcow2', '-O', fmt,
+ '$test_img', 'converted_image.' + fmt])
+
+ try:
+ os.makedirs(self.current_dir)
+ except OSError, e:
+ print >>sys.stderr, \
+ "Error: The working directory '%s' cannot be used. Reason: %s"\
+ % (self.work_dir, e[1])
+ raise TestException
+ self.log = open(os.path.join(self.current_dir, "test.log"), "w")
+ self.parent_log = open(run_log, "a")
+ self.failed = False
+ self.cleanup = cleanup
+ self.log_all = log_all
+
+ def _create_backing_file(self):
+ """Create a backing file in the current directory.
+
+ Return a tuple of a backing file name and format.
+
+ Format of a backing file is randomly chosen from all formats supported
+ by 'qemu-img create'.
+ """
+ # All formats supported by the 'qemu-img create' command.
+ backing_file_fmt = random.choice(['raw', 'vmdk', 'vdi', 'cow', 'qcow2',
+ 'file', 'qed', 'vpc'])
+ backing_file_name = 'backing_img.' + backing_file_fmt
+ backing_file_size = random.randint(MIN_BACKING_FILE_SIZE,
+ MAX_BACKING_FILE_SIZE) * (1 << 20)
+ cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
+ backing_file_name, str(backing_file_size)]
+ temp_log = StringIO.StringIO()
+ retcode = run_app(temp_log, cmd)
+ if retcode == 0:
+ temp_log.close()
+ return (backing_file_name, backing_file_fmt)
+ else:
+ multilog("Warning: The %s backing file was not created.\n\n"
+ % backing_file_fmt, sys.stderr, self.log, self.parent_log)
+ self.log.write("Log for the failure:\n" + temp_log.getvalue() +
+ '\n\n')
+ temp_log.close()
+ return (None, None)
+
+ def execute(self, input_commands=None, fuzz_config=None):
+ """ Execute a test.
+
+ The method creates backing and test images, runs test app and analyzes
+ its exit status. If the application was killed by a signal, the test
+ is marked as failed.
+ """
+ if input_commands is None:
+ commands = self.commands
+ else:
+ commands = input_commands
+
+ os.chdir(self.current_dir)
+ backing_file_name, backing_file_fmt = self._create_backing_file()
+ img_size = image_generator.create_image('test.img',
+ backing_file_name,
+ backing_file_fmt,
+ fuzz_config)
+ for item in commands:
+ shutil.copy('test.img', 'copy.img')
+ # 'off' and 'len' are multiple of the sector size
+ sector_size = 512
+ start = random.randrange(0, img_size + 1, sector_size)
+ end = random.randrange(start, img_size + 1, sector_size)
+
+ if item[0] == 'qemu-img':
+ current_cmd = list(self.qemu_img)
+ elif item[0] == 'qemu-io':
+ current_cmd = list(self.qemu_io)
+ else:
+ multilog("Warning: test command '%s' is not defined.\n" \
+ % item[0], sys.stderr, self.log, self.parent_log)
+ continue
+ # Replace all placeholders with their real values
+ for v in item[1:]:
+ c = (v
+ .replace('$test_img', 'copy.img')
+ .replace('$off', str(start))
+ .replace('$len', str(end - start)))
+ current_cmd.append(c)
+
+ # Log string with the test header
+ test_summary = "Seed: %s\nCommand: %s\nTest directory: %s\n" \
+ "Backing file: %s\n" \
+ % (self.seed, " ".join(current_cmd),
+ self.current_dir, backing_file_name)
+
+ temp_log = StringIO.StringIO()
+ try:
+ retcode = run_app(temp_log, current_cmd)
+ except OSError, e:
+ multilog(test_summary + "Error: Start of '%s' failed. " \
+ "Reason: %s\n\n" % (os.path.basename(
+ current_cmd[0]), e[1]),
+ sys.stderr, self.log, self.parent_log)
+ raise TestException
+
+ if retcode < 0:
+ self.log.write(temp_log.getvalue())
+ multilog(test_summary + "FAIL: Test terminated by signal " +
+ "%s\n\n" % str_signal(-retcode), sys.stderr, self.log,
+ self.parent_log)
+ self.failed = True
+ else:
+ if self.log_all:
+ self.log.write(temp_log.getvalue())
+ multilog(test_summary + "PASS: Application exited with" +
+ " the code '%d'\n\n" % retcode, sys.stdout,
+ self.log, self.parent_log)
+ temp_log.close()
+ os.remove('copy.img')
+
+ def finish(self):
+ """Restore the test environment after a test execution."""
+ self.log.close()
+ self.parent_log.close()
+ os.chdir(self.init_path)
+ if self.cleanup and not self.failed:
+ shutil.rmtree(self.current_dir)
+
+if __name__ == '__main__':
+
+ def usage():
+ print """
+ Usage: runner.py [OPTION...] TEST_DIR IMG_GENERATOR
+
+ Set up test environment in TEST_DIR and run a test in it. A module for
+ test image generation should be specified via IMG_GENERATOR.
+ Example:
+ runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test ../qcow2
+
+ Optional arguments:
+ -h, --help display this help and exit
+ -c, --command=JSON run tests for all commands specified in
+ the JSON array
+ -s, --seed=STRING seed for a test image generation,
+ by default will be generated randomly
+ --config=JSON take fuzzer configuration from the JSON
+ array
+ -k, --keep_passed don't remove folders of passed tests
+ -v, --verbose log information about passed tests
+
+ JSON:
+
+ '--command' accepts a JSON array of commands. Each command presents
+ an application under test with all its paramaters as a list of strings,
+ e.g.
+ ["qemu-io", "$test_img", "-c", "write $off $len"]
+
+ Supported application aliases: 'qemu-img' and 'qemu-io'.
+ Supported argument aliases: $test_img for the fuzzed image, $off
+ for an offset, $len for length.
+
+ Values for $off and $len will be generated based on the virtual disk
+ size of the fuzzed image
+ Paths to 'qemu-img' and 'qemu-io' are retrevied from 'QEMU_IMG' and
+ 'QEMU_IO' environment variables
+
+ '--config' accepts a JSON array of fields to be fuzzed, e.g.
+ '[["header"], ["header", "version"]]'
+ Each of the list elements can consist of a complex image element only
+ as ["header"] or ["feature_name_table"] or an exact field as
+ ["header", "version"]. In the first case random portion of the element
+ fields will be fuzzed, in the second one the specified field will be
+ fuzzed always.
+
+ If '--config' argument is specified, fields not listed in
+ the configuration array will not be fuzzed.
+ """
+
+ def run_test(test_id, seed, work_dir, run_log, cleanup, log_all,
+ command, fuzz_config):
+ """Setup environment for one test and execute this test."""
+ try:
+ test = TestEnv(test_id, seed, work_dir, run_log, cleanup,
+ log_all)
+ except TestException:
+ sys.exit(1)
+
+ # Python 2.4 doesn't support 'finally' and 'except' in the same 'try'
+ # block
+ try:
+ try:
+ test.execute(command, fuzz_config)
+ except TestException:
+ sys.exit(1)
+ finally:
+ test.finish()
+
+ try:
+ opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hs:kv',
+ ['command=', 'help', 'seed=', 'config=',
+ 'keep_passed', 'verbose'])
+ except getopt.error, e:
+ print >>sys.stderr, \
+ "Error: %s\n\nTry 'runner.py --help' for more information" % e
+ sys.exit(1)
+
+ command = None
+ cleanup = True
+ log_all = False
+ seed = None
+ config = None
+ for opt, arg in opts:
+ if opt in ('-h', '--help'):
+ usage()
+ sys.exit()
+ elif opt in ('-c', '--command'):
+ try:
+ command = json.loads(arg)
+ except (TypeError, ValueError, NameError), e:
+ print >>sys.stderr, \
+ "Error: JSON array of test commands cannot be loaded.\n" \
+ "Reason: %s" % e
+ sys.exit(1)
+ elif opt in ('-k', '--keep_passed'):
+ cleanup = False
+ elif opt in ('-v', '--verbose'):
+ log_all = True
+ elif opt in ('-s', '--seed'):
+ seed = arg
+ elif opt == '--config':
+ try:
+ config = json.loads(arg)
+ except (TypeError, ValueError, NameError), e:
+ print >>sys.stderr, \
+ "Error: JSON array with the fuzzer configuration cannot" \
+ " be loaded\nReason: %s" % e
+ sys.exit(1)
+
+ if not len(args) == 2:
+ print >>sys.stderr, \
+ "Expected two parameters\nTry 'runner.py --help'" \
+ " for more information."
+ sys.exit(1)
+
+ work_dir = os.path.realpath(args[0])
+ # run_log is created in 'main', because multiple tests are expected to
+ # log in it
+ run_log = os.path.join(work_dir, 'run.log')
+
+ # Add the path to the image generator module to sys.path
+ sys.path.append(os.path.realpath(os.path.dirname(args[1])))
+ # Remove a script extension from image generator module if any
+ generator_name = os.path.splitext(os.path.basename(args[1]))[0]
+
+ try:
+ image_generator = __import__(generator_name)
+ except ImportError, e:
+ print >>sys.stderr, \
+ "Error: The image generator '%s' cannot be imported.\n" \
+ "Reason: %s" % (generator_name, e)
+ sys.exit(1)
+
+ # Enable core dumps
+ resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
+ # If a seed is specified, only one test will be executed.
+ # Otherwise runner will terminate after a keyboard interruption
+ for test_id in count(1):
+ try:
+ run_test(str(test_id), seed, work_dir, run_log, cleanup,
+ log_all, command, config)
+ except (KeyboardInterrupt, SystemExit):
+ sys.exit(1)
+
+ if seed is not None:
+ break
--
1.9.3
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution Maria Kustova
@ 2014-08-08 6:52 ` Stefan Hajnoczi
2014-08-08 8:50 ` Fam Zheng
1 sibling, 0 replies; 16+ messages in thread
From: Stefan Hajnoczi @ 2014-08-08 6:52 UTC (permalink / raw)
To: Maria Kustova; +Cc: kwolf, famz, qemu-devel, stefanha, Maria Kustova
[-- Attachment #1: Type: text/plain, Size: 831 bytes --]
On Wed, Aug 06, 2014 at 05:12:47PM +0400, Maria Kustova wrote:
> The purpose of the test runner is to prepare the test environment (e.g. create
> a work directory, a test image, etc), execute a program under test with
> parameters, indicate a test failure if the program was killed during the test
> execution and collect core dumps, logs and other test artifacts.
>
> The test runner doesn't depend on an image format or a program will be tested,
> so it can be used with any external image generator and program under test.
>
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
> tests/image-fuzzer/runner/runner.py | 405 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 405 insertions(+)
> create mode 100755 tests/image-fuzzer/runner/runner.py
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution Maria Kustova
2014-08-08 6:52 ` Stefan Hajnoczi
@ 2014-08-08 8:50 ` Fam Zheng
2014-08-08 8:58 ` M.Kustova
1 sibling, 1 reply; 16+ messages in thread
From: Fam Zheng @ 2014-08-08 8:50 UTC (permalink / raw)
To: Maria Kustova; +Cc: kwolf, qemu-devel, stefanha, Maria Kustova
On Wed, 08/06 17:12, Maria Kustova wrote:
> The purpose of the test runner is to prepare the test environment (e.g. create
> a work directory, a test image, etc), execute a program under test with
> parameters, indicate a test failure if the program was killed during the test
> execution and collect core dumps, logs and other test artifacts.
>
> The test runner doesn't depend on an image format or a program will be tested,
> so it can be used with any external image generator and program under test.
>
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
> tests/image-fuzzer/runner/runner.py | 405 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 405 insertions(+)
> create mode 100755 tests/image-fuzzer/runner/runner.py
>
> diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
> new file mode 100755
> index 0000000..3fa7fca
> --- /dev/null
> +++ b/tests/image-fuzzer/runner/runner.py
> @@ -0,0 +1,405 @@
> +#!/usr/bin/env python
> +
> +# Tool for running fuzz tests
> +#
> +# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
> +#
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation, either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import sys
> +import os
> +import signal
> +import subprocess
> +import random
> +import shutil
> +from itertools import count
> +import getopt
> +import StringIO
> +import resource
> +
> +try:
> + import json
> +except ImportError:
> + try:
> + import simplejson as json
> + except ImportError:
> + print >>sys.stderr, \
> + "Warning: Module for JSON processing is not found.\n" \
> + "'--config' and '--command' options are not supported."
> +
> +# Backing file sizes in MB
> +MAX_BACKING_FILE_SIZE = 10
> +MIN_BACKING_FILE_SIZE = 1
> +
> +
> +def multilog(msg, *output):
> + """ Write an object to all of specified file descriptors."""
> + for fd in output:
> + fd.write(msg)
> + fd.flush()
> +
> +
> +def str_signal(sig):
> + """ Convert a numeric value of a system signal to the string one
> + defined by the current operational system.
> + """
> + for k, v in signal.__dict__.items():
> + if v == sig:
> + return k
> +
> +
> +def run_app(fd, q_args):
> + """Start an application with specified arguments and return its exit code
> + or kill signal depending on the result of execution.
> + """
> + devnull = open('/dev/null', 'r+')
> + process = subprocess.Popen(q_args, stdin=devnull,
> + stdout=subprocess.PIPE,
> + stderr=subprocess.PIPE)
> + out, err = process.communicate()
> + fd.write(out)
> + fd.write(err)
> + return process.returncode
> +
> +
> +class TestException(Exception):
> + """Exception for errors risen by TestEnv objects."""
> + pass
> +
> +
> +class TestEnv(object):
> +
> + """Test object.
> +
> + The class sets up test environment, generates backing and test images
> + and executes application under tests with specified arguments and a test
> + image provided.
> +
> + All logs are collected.
> +
> + The summary log will contain short descriptions and statuses of tests in
> + a run.
> +
> + The test log will include application (e.g. 'qemu-img') logs besides info
> + sent to the summary log.
> + """
> +
> + def __init__(self, test_id, seed, work_dir, run_log,
> + cleanup=True, log_all=False):
> + """Set test environment in a specified work directory.
> +
> + Path to qemu-img and qemu-io will be retrieved from 'QEMU_IMG' and
> + 'QEMU_IO' environment variables.
> + """
> + if seed is not None:
> + self.seed = seed
> + else:
> + self.seed = str(random.randint(0, sys.maxint))
> + random.seed(self.seed)
> +
> + self.init_path = os.getcwd()
> + self.work_dir = work_dir
> + self.current_dir = os.path.join(work_dir, 'test-' + test_id)
> + self.qemu_img = os.environ.get('QEMU_IMG', 'qemu-img')\
> + .strip().split(' ')
Nitpicking. I think split(' ') doesn't make sense, this could instead be:
self.qemu_img = [os.environ.get('QEMU_IMG', 'qemu-img').strip()]
Otherwise user won't be able to pass in a QEMU_IMG path with space.
Corner case, though. Otherwise looks good,
Fam
> + self.qemu_io = os.environ.get('QEMU_IO', 'qemu-io').strip().split(' ')
> + self.commands = [['qemu-img', 'check', '-f', 'qcow2', '$test_img'],
> + ['qemu-img', 'info', '-f', 'qcow2', '$test_img'],
> + ['qemu-io', '$test_img', '-c', 'read $off $len'],
> + ['qemu-io', '$test_img', '-c', 'write $off $len'],
> + ['qemu-io', '$test_img', '-c',
> + 'aio_read $off $len'],
> + ['qemu-io', '$test_img', '-c',
> + 'aio_write $off $len'],
> + ['qemu-io', '$test_img', '-c', 'flush'],
> + ['qemu-io', '$test_img', '-c',
> + 'discard $off $len'],
> + ['qemu-io', '$test_img', '-c',
> + 'truncate $off']]
> + for fmt in ['raw', 'vmdk', 'vdi', 'cow', 'qcow2', 'file',
> + 'qed', 'vpc']:
> + self.commands.append(
> + ['qemu-img', 'convert', '-f', 'qcow2', '-O', fmt,
> + '$test_img', 'converted_image.' + fmt])
> +
> + try:
> + os.makedirs(self.current_dir)
> + except OSError, e:
> + print >>sys.stderr, \
> + "Error: The working directory '%s' cannot be used. Reason: %s"\
> + % (self.work_dir, e[1])
> + raise TestException
> + self.log = open(os.path.join(self.current_dir, "test.log"), "w")
> + self.parent_log = open(run_log, "a")
> + self.failed = False
> + self.cleanup = cleanup
> + self.log_all = log_all
> +
> + def _create_backing_file(self):
> + """Create a backing file in the current directory.
> +
> + Return a tuple of a backing file name and format.
> +
> + Format of a backing file is randomly chosen from all formats supported
> + by 'qemu-img create'.
> + """
> + # All formats supported by the 'qemu-img create' command.
> + backing_file_fmt = random.choice(['raw', 'vmdk', 'vdi', 'cow', 'qcow2',
> + 'file', 'qed', 'vpc'])
> + backing_file_name = 'backing_img.' + backing_file_fmt
> + backing_file_size = random.randint(MIN_BACKING_FILE_SIZE,
> + MAX_BACKING_FILE_SIZE) * (1 << 20)
> + cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
> + backing_file_name, str(backing_file_size)]
> + temp_log = StringIO.StringIO()
> + retcode = run_app(temp_log, cmd)
> + if retcode == 0:
> + temp_log.close()
> + return (backing_file_name, backing_file_fmt)
> + else:
> + multilog("Warning: The %s backing file was not created.\n\n"
> + % backing_file_fmt, sys.stderr, self.log, self.parent_log)
> + self.log.write("Log for the failure:\n" + temp_log.getvalue() +
> + '\n\n')
> + temp_log.close()
> + return (None, None)
> +
> + def execute(self, input_commands=None, fuzz_config=None):
> + """ Execute a test.
> +
> + The method creates backing and test images, runs test app and analyzes
> + its exit status. If the application was killed by a signal, the test
> + is marked as failed.
> + """
> + if input_commands is None:
> + commands = self.commands
> + else:
> + commands = input_commands
> +
> + os.chdir(self.current_dir)
> + backing_file_name, backing_file_fmt = self._create_backing_file()
> + img_size = image_generator.create_image('test.img',
> + backing_file_name,
> + backing_file_fmt,
> + fuzz_config)
> + for item in commands:
> + shutil.copy('test.img', 'copy.img')
> + # 'off' and 'len' are multiple of the sector size
> + sector_size = 512
> + start = random.randrange(0, img_size + 1, sector_size)
> + end = random.randrange(start, img_size + 1, sector_size)
> +
> + if item[0] == 'qemu-img':
> + current_cmd = list(self.qemu_img)
> + elif item[0] == 'qemu-io':
> + current_cmd = list(self.qemu_io)
> + else:
> + multilog("Warning: test command '%s' is not defined.\n" \
> + % item[0], sys.stderr, self.log, self.parent_log)
> + continue
> + # Replace all placeholders with their real values
> + for v in item[1:]:
> + c = (v
> + .replace('$test_img', 'copy.img')
> + .replace('$off', str(start))
> + .replace('$len', str(end - start)))
> + current_cmd.append(c)
> +
> + # Log string with the test header
> + test_summary = "Seed: %s\nCommand: %s\nTest directory: %s\n" \
> + "Backing file: %s\n" \
> + % (self.seed, " ".join(current_cmd),
> + self.current_dir, backing_file_name)
> +
> + temp_log = StringIO.StringIO()
> + try:
> + retcode = run_app(temp_log, current_cmd)
> + except OSError, e:
> + multilog(test_summary + "Error: Start of '%s' failed. " \
> + "Reason: %s\n\n" % (os.path.basename(
> + current_cmd[0]), e[1]),
> + sys.stderr, self.log, self.parent_log)
> + raise TestException
> +
> + if retcode < 0:
> + self.log.write(temp_log.getvalue())
> + multilog(test_summary + "FAIL: Test terminated by signal " +
> + "%s\n\n" % str_signal(-retcode), sys.stderr, self.log,
> + self.parent_log)
> + self.failed = True
> + else:
> + if self.log_all:
> + self.log.write(temp_log.getvalue())
> + multilog(test_summary + "PASS: Application exited with" +
> + " the code '%d'\n\n" % retcode, sys.stdout,
> + self.log, self.parent_log)
> + temp_log.close()
> + os.remove('copy.img')
> +
> + def finish(self):
> + """Restore the test environment after a test execution."""
> + self.log.close()
> + self.parent_log.close()
> + os.chdir(self.init_path)
> + if self.cleanup and not self.failed:
> + shutil.rmtree(self.current_dir)
> +
> +if __name__ == '__main__':
> +
> + def usage():
> + print """
> + Usage: runner.py [OPTION...] TEST_DIR IMG_GENERATOR
> +
> + Set up test environment in TEST_DIR and run a test in it. A module for
> + test image generation should be specified via IMG_GENERATOR.
> + Example:
> + runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test ../qcow2
> +
> + Optional arguments:
> + -h, --help display this help and exit
> + -c, --command=JSON run tests for all commands specified in
> + the JSON array
> + -s, --seed=STRING seed for a test image generation,
> + by default will be generated randomly
> + --config=JSON take fuzzer configuration from the JSON
> + array
> + -k, --keep_passed don't remove folders of passed tests
> + -v, --verbose log information about passed tests
> +
> + JSON:
> +
> + '--command' accepts a JSON array of commands. Each command presents
> + an application under test with all its paramaters as a list of strings,
> + e.g.
> + ["qemu-io", "$test_img", "-c", "write $off $len"]
> +
> + Supported application aliases: 'qemu-img' and 'qemu-io'.
> + Supported argument aliases: $test_img for the fuzzed image, $off
> + for an offset, $len for length.
> +
> + Values for $off and $len will be generated based on the virtual disk
> + size of the fuzzed image
> + Paths to 'qemu-img' and 'qemu-io' are retrevied from 'QEMU_IMG' and
> + 'QEMU_IO' environment variables
> +
> + '--config' accepts a JSON array of fields to be fuzzed, e.g.
> + '[["header"], ["header", "version"]]'
> + Each of the list elements can consist of a complex image element only
> + as ["header"] or ["feature_name_table"] or an exact field as
> + ["header", "version"]. In the first case random portion of the element
> + fields will be fuzzed, in the second one the specified field will be
> + fuzzed always.
> +
> + If '--config' argument is specified, fields not listed in
> + the configuration array will not be fuzzed.
> + """
> +
> + def run_test(test_id, seed, work_dir, run_log, cleanup, log_all,
> + command, fuzz_config):
> + """Setup environment for one test and execute this test."""
> + try:
> + test = TestEnv(test_id, seed, work_dir, run_log, cleanup,
> + log_all)
> + except TestException:
> + sys.exit(1)
> +
> + # Python 2.4 doesn't support 'finally' and 'except' in the same 'try'
> + # block
> + try:
> + try:
> + test.execute(command, fuzz_config)
> + except TestException:
> + sys.exit(1)
> + finally:
> + test.finish()
> +
> + try:
> + opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hs:kv',
> + ['command=', 'help', 'seed=', 'config=',
> + 'keep_passed', 'verbose'])
> + except getopt.error, e:
> + print >>sys.stderr, \
> + "Error: %s\n\nTry 'runner.py --help' for more information" % e
> + sys.exit(1)
> +
> + command = None
> + cleanup = True
> + log_all = False
> + seed = None
> + config = None
> + for opt, arg in opts:
> + if opt in ('-h', '--help'):
> + usage()
> + sys.exit()
> + elif opt in ('-c', '--command'):
> + try:
> + command = json.loads(arg)
> + except (TypeError, ValueError, NameError), e:
> + print >>sys.stderr, \
> + "Error: JSON array of test commands cannot be loaded.\n" \
> + "Reason: %s" % e
> + sys.exit(1)
> + elif opt in ('-k', '--keep_passed'):
> + cleanup = False
> + elif opt in ('-v', '--verbose'):
> + log_all = True
> + elif opt in ('-s', '--seed'):
> + seed = arg
> + elif opt == '--config':
> + try:
> + config = json.loads(arg)
> + except (TypeError, ValueError, NameError), e:
> + print >>sys.stderr, \
> + "Error: JSON array with the fuzzer configuration cannot" \
> + " be loaded\nReason: %s" % e
> + sys.exit(1)
> +
> + if not len(args) == 2:
> + print >>sys.stderr, \
> + "Expected two parameters\nTry 'runner.py --help'" \
> + " for more information."
> + sys.exit(1)
> +
> + work_dir = os.path.realpath(args[0])
> + # run_log is created in 'main', because multiple tests are expected to
> + # log in it
> + run_log = os.path.join(work_dir, 'run.log')
> +
> + # Add the path to the image generator module to sys.path
> + sys.path.append(os.path.realpath(os.path.dirname(args[1])))
> + # Remove a script extension from image generator module if any
> + generator_name = os.path.splitext(os.path.basename(args[1]))[0]
> +
> + try:
> + image_generator = __import__(generator_name)
> + except ImportError, e:
> + print >>sys.stderr, \
> + "Error: The image generator '%s' cannot be imported.\n" \
> + "Reason: %s" % (generator_name, e)
> + sys.exit(1)
> +
> + # Enable core dumps
> + resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
> + # If a seed is specified, only one test will be executed.
> + # Otherwise runner will terminate after a keyboard interruption
> + for test_id in count(1):
> + try:
> + run_test(str(test_id), seed, work_dir, run_log, cleanup,
> + log_all, command, config)
> + except (KeyboardInterrupt, SystemExit):
> + sys.exit(1)
> +
> + if seed is not None:
> + break
> --
> 1.9.3
>
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution
2014-08-08 8:50 ` Fam Zheng
@ 2014-08-08 8:58 ` M.Kustova
2014-08-08 11:08 ` Fam Zheng
0 siblings, 1 reply; 16+ messages in thread
From: M.Kustova @ 2014-08-08 8:58 UTC (permalink / raw)
To: Fam Zheng; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi
On Fri, Aug 8, 2014 at 12:50 PM, Fam Zheng <famz@redhat.com> wrote:
> On Wed, 08/06 17:12, Maria Kustova wrote:
>> The purpose of the test runner is to prepare the test environment (e.g. create
>> a work directory, a test image, etc), execute a program under test with
>> parameters, indicate a test failure if the program was killed during the test
>> execution and collect core dumps, logs and other test artifacts.
>>
>> The test runner doesn't depend on an image format or a program will be tested,
>> so it can be used with any external image generator and program under test.
>>
>> Signed-off-by: Maria Kustova <maria.k@catit.be>
>> ---
>> tests/image-fuzzer/runner/runner.py | 405 ++++++++++++++++++++++++++++++++++++
>> 1 file changed, 405 insertions(+)
>> create mode 100755 tests/image-fuzzer/runner/runner.py
>>
>> diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
>> new file mode 100755
>> index 0000000..3fa7fca
>> --- /dev/null
>> +++ b/tests/image-fuzzer/runner/runner.py
>> @@ -0,0 +1,405 @@
>> +#!/usr/bin/env python
>> +
>> +# Tool for running fuzz tests
>> +#
>> +# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
>> +#
>> +# This program is free software: you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation, either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
>> +#
>> +
>> +import sys
>> +import os
>> +import signal
>> +import subprocess
>> +import random
>> +import shutil
>> +from itertools import count
>> +import getopt
>> +import StringIO
>> +import resource
>> +
>> +try:
>> + import json
>> +except ImportError:
>> + try:
>> + import simplejson as json
>> + except ImportError:
>> + print >>sys.stderr, \
>> + "Warning: Module for JSON processing is not found.\n" \
>> + "'--config' and '--command' options are not supported."
>> +
>> +# Backing file sizes in MB
>> +MAX_BACKING_FILE_SIZE = 10
>> +MIN_BACKING_FILE_SIZE = 1
>> +
>> +
>> +def multilog(msg, *output):
>> + """ Write an object to all of specified file descriptors."""
>> + for fd in output:
>> + fd.write(msg)
>> + fd.flush()
>> +
>> +
>> +def str_signal(sig):
>> + """ Convert a numeric value of a system signal to the string one
>> + defined by the current operational system.
>> + """
>> + for k, v in signal.__dict__.items():
>> + if v == sig:
>> + return k
>> +
>> +
>> +def run_app(fd, q_args):
>> + """Start an application with specified arguments and return its exit code
>> + or kill signal depending on the result of execution.
>> + """
>> + devnull = open('/dev/null', 'r+')
>> + process = subprocess.Popen(q_args, stdin=devnull,
>> + stdout=subprocess.PIPE,
>> + stderr=subprocess.PIPE)
>> + out, err = process.communicate()
>> + fd.write(out)
>> + fd.write(err)
>> + return process.returncode
>> +
>> +
>> +class TestException(Exception):
>> + """Exception for errors risen by TestEnv objects."""
>> + pass
>> +
>> +
>> +class TestEnv(object):
>> +
>> + """Test object.
>> +
>> + The class sets up test environment, generates backing and test images
>> + and executes application under tests with specified arguments and a test
>> + image provided.
>> +
>> + All logs are collected.
>> +
>> + The summary log will contain short descriptions and statuses of tests in
>> + a run.
>> +
>> + The test log will include application (e.g. 'qemu-img') logs besides info
>> + sent to the summary log.
>> + """
>> +
>> + def __init__(self, test_id, seed, work_dir, run_log,
>> + cleanup=True, log_all=False):
>> + """Set test environment in a specified work directory.
>> +
>> + Path to qemu-img and qemu-io will be retrieved from 'QEMU_IMG' and
>> + 'QEMU_IO' environment variables.
>> + """
>> + if seed is not None:
>> + self.seed = seed
>> + else:
>> + self.seed = str(random.randint(0, sys.maxint))
>> + random.seed(self.seed)
>> +
>> + self.init_path = os.getcwd()
>> + self.work_dir = work_dir
>> + self.current_dir = os.path.join(work_dir, 'test-' + test_id)
>> + self.qemu_img = os.environ.get('QEMU_IMG', 'qemu-img')\
>> + .strip().split(' ')
>
> Nitpicking. I think split(' ') doesn't make sense, this could instead be:
>
> self.qemu_img = [os.environ.get('QEMU_IMG', 'qemu-img').strip()]
>
> Otherwise user won't be able to pass in a QEMU_IMG path with space.
>
> Corner case, though. Otherwise looks good,
This functionality was inherited from qemu-iotests/iotests.py.
Rationale for this splitting was that environment variables can
contain call arguments as well.
Ruining paths with white spaces was a reasonable trade-off.
Maria.
>> + self.qemu_io = os.environ.get('QEMU_IO', 'qemu-io').strip().split(' ')
>> + self.commands = [['qemu-img', 'check', '-f', 'qcow2', '$test_img'],
>> + ['qemu-img', 'info', '-f', 'qcow2', '$test_img'],
>> + ['qemu-io', '$test_img', '-c', 'read $off $len'],
>> + ['qemu-io', '$test_img', '-c', 'write $off $len'],
>> + ['qemu-io', '$test_img', '-c',
>> + 'aio_read $off $len'],
>> + ['qemu-io', '$test_img', '-c',
>> + 'aio_write $off $len'],
>> + ['qemu-io', '$test_img', '-c', 'flush'],
>> + ['qemu-io', '$test_img', '-c',
>> + 'discard $off $len'],
>> + ['qemu-io', '$test_img', '-c',
>> + 'truncate $off']]
>> + for fmt in ['raw', 'vmdk', 'vdi', 'cow', 'qcow2', 'file',
>> + 'qed', 'vpc']:
>> + self.commands.append(
>> + ['qemu-img', 'convert', '-f', 'qcow2', '-O', fmt,
>> + '$test_img', 'converted_image.' + fmt])
>> +
>> + try:
>> + os.makedirs(self.current_dir)
>> + except OSError, e:
>> + print >>sys.stderr, \
>> + "Error: The working directory '%s' cannot be used. Reason: %s"\
>> + % (self.work_dir, e[1])
>> + raise TestException
>> + self.log = open(os.path.join(self.current_dir, "test.log"), "w")
>> + self.parent_log = open(run_log, "a")
>> + self.failed = False
>> + self.cleanup = cleanup
>> + self.log_all = log_all
>> +
>> + def _create_backing_file(self):
>> + """Create a backing file in the current directory.
>> +
>> + Return a tuple of a backing file name and format.
>> +
>> + Format of a backing file is randomly chosen from all formats supported
>> + by 'qemu-img create'.
>> + """
>> + # All formats supported by the 'qemu-img create' command.
>> + backing_file_fmt = random.choice(['raw', 'vmdk', 'vdi', 'cow', 'qcow2',
>> + 'file', 'qed', 'vpc'])
>> + backing_file_name = 'backing_img.' + backing_file_fmt
>> + backing_file_size = random.randint(MIN_BACKING_FILE_SIZE,
>> + MAX_BACKING_FILE_SIZE) * (1 << 20)
>> + cmd = self.qemu_img + ['create', '-f', backing_file_fmt,
>> + backing_file_name, str(backing_file_size)]
>> + temp_log = StringIO.StringIO()
>> + retcode = run_app(temp_log, cmd)
>> + if retcode == 0:
>> + temp_log.close()
>> + return (backing_file_name, backing_file_fmt)
>> + else:
>> + multilog("Warning: The %s backing file was not created.\n\n"
>> + % backing_file_fmt, sys.stderr, self.log, self.parent_log)
>> + self.log.write("Log for the failure:\n" + temp_log.getvalue() +
>> + '\n\n')
>> + temp_log.close()
>> + return (None, None)
>> +
>> + def execute(self, input_commands=None, fuzz_config=None):
>> + """ Execute a test.
>> +
>> + The method creates backing and test images, runs test app and analyzes
>> + its exit status. If the application was killed by a signal, the test
>> + is marked as failed.
>> + """
>> + if input_commands is None:
>> + commands = self.commands
>> + else:
>> + commands = input_commands
>> +
>> + os.chdir(self.current_dir)
>> + backing_file_name, backing_file_fmt = self._create_backing_file()
>> + img_size = image_generator.create_image('test.img',
>> + backing_file_name,
>> + backing_file_fmt,
>> + fuzz_config)
>> + for item in commands:
>> + shutil.copy('test.img', 'copy.img')
>> + # 'off' and 'len' are multiple of the sector size
>> + sector_size = 512
>> + start = random.randrange(0, img_size + 1, sector_size)
>> + end = random.randrange(start, img_size + 1, sector_size)
>> +
>> + if item[0] == 'qemu-img':
>> + current_cmd = list(self.qemu_img)
>> + elif item[0] == 'qemu-io':
>> + current_cmd = list(self.qemu_io)
>> + else:
>> + multilog("Warning: test command '%s' is not defined.\n" \
>> + % item[0], sys.stderr, self.log, self.parent_log)
>> + continue
>> + # Replace all placeholders with their real values
>> + for v in item[1:]:
>> + c = (v
>> + .replace('$test_img', 'copy.img')
>> + .replace('$off', str(start))
>> + .replace('$len', str(end - start)))
>> + current_cmd.append(c)
>> +
>> + # Log string with the test header
>> + test_summary = "Seed: %s\nCommand: %s\nTest directory: %s\n" \
>> + "Backing file: %s\n" \
>> + % (self.seed, " ".join(current_cmd),
>> + self.current_dir, backing_file_name)
>> +
>> + temp_log = StringIO.StringIO()
>> + try:
>> + retcode = run_app(temp_log, current_cmd)
>> + except OSError, e:
>> + multilog(test_summary + "Error: Start of '%s' failed. " \
>> + "Reason: %s\n\n" % (os.path.basename(
>> + current_cmd[0]), e[1]),
>> + sys.stderr, self.log, self.parent_log)
>> + raise TestException
>> +
>> + if retcode < 0:
>> + self.log.write(temp_log.getvalue())
>> + multilog(test_summary + "FAIL: Test terminated by signal " +
>> + "%s\n\n" % str_signal(-retcode), sys.stderr, self.log,
>> + self.parent_log)
>> + self.failed = True
>> + else:
>> + if self.log_all:
>> + self.log.write(temp_log.getvalue())
>> + multilog(test_summary + "PASS: Application exited with" +
>> + " the code '%d'\n\n" % retcode, sys.stdout,
>> + self.log, self.parent_log)
>> + temp_log.close()
>> + os.remove('copy.img')
>> +
>> + def finish(self):
>> + """Restore the test environment after a test execution."""
>> + self.log.close()
>> + self.parent_log.close()
>> + os.chdir(self.init_path)
>> + if self.cleanup and not self.failed:
>> + shutil.rmtree(self.current_dir)
>> +
>> +if __name__ == '__main__':
>> +
>> + def usage():
>> + print """
>> + Usage: runner.py [OPTION...] TEST_DIR IMG_GENERATOR
>> +
>> + Set up test environment in TEST_DIR and run a test in it. A module for
>> + test image generation should be specified via IMG_GENERATOR.
>> + Example:
>> + runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test ../qcow2
>> +
>> + Optional arguments:
>> + -h, --help display this help and exit
>> + -c, --command=JSON run tests for all commands specified in
>> + the JSON array
>> + -s, --seed=STRING seed for a test image generation,
>> + by default will be generated randomly
>> + --config=JSON take fuzzer configuration from the JSON
>> + array
>> + -k, --keep_passed don't remove folders of passed tests
>> + -v, --verbose log information about passed tests
>> +
>> + JSON:
>> +
>> + '--command' accepts a JSON array of commands. Each command presents
>> + an application under test with all its paramaters as a list of strings,
>> + e.g.
>> + ["qemu-io", "$test_img", "-c", "write $off $len"]
>> +
>> + Supported application aliases: 'qemu-img' and 'qemu-io'.
>> + Supported argument aliases: $test_img for the fuzzed image, $off
>> + for an offset, $len for length.
>> +
>> + Values for $off and $len will be generated based on the virtual disk
>> + size of the fuzzed image
>> + Paths to 'qemu-img' and 'qemu-io' are retrevied from 'QEMU_IMG' and
>> + 'QEMU_IO' environment variables
>> +
>> + '--config' accepts a JSON array of fields to be fuzzed, e.g.
>> + '[["header"], ["header", "version"]]'
>> + Each of the list elements can consist of a complex image element only
>> + as ["header"] or ["feature_name_table"] or an exact field as
>> + ["header", "version"]. In the first case random portion of the element
>> + fields will be fuzzed, in the second one the specified field will be
>> + fuzzed always.
>> +
>> + If '--config' argument is specified, fields not listed in
>> + the configuration array will not be fuzzed.
>> + """
>> +
>> + def run_test(test_id, seed, work_dir, run_log, cleanup, log_all,
>> + command, fuzz_config):
>> + """Setup environment for one test and execute this test."""
>> + try:
>> + test = TestEnv(test_id, seed, work_dir, run_log, cleanup,
>> + log_all)
>> + except TestException:
>> + sys.exit(1)
>> +
>> + # Python 2.4 doesn't support 'finally' and 'except' in the same 'try'
>> + # block
>> + try:
>> + try:
>> + test.execute(command, fuzz_config)
>> + except TestException:
>> + sys.exit(1)
>> + finally:
>> + test.finish()
>> +
>> + try:
>> + opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hs:kv',
>> + ['command=', 'help', 'seed=', 'config=',
>> + 'keep_passed', 'verbose'])
>> + except getopt.error, e:
>> + print >>sys.stderr, \
>> + "Error: %s\n\nTry 'runner.py --help' for more information" % e
>> + sys.exit(1)
>> +
>> + command = None
>> + cleanup = True
>> + log_all = False
>> + seed = None
>> + config = None
>> + for opt, arg in opts:
>> + if opt in ('-h', '--help'):
>> + usage()
>> + sys.exit()
>> + elif opt in ('-c', '--command'):
>> + try:
>> + command = json.loads(arg)
>> + except (TypeError, ValueError, NameError), e:
>> + print >>sys.stderr, \
>> + "Error: JSON array of test commands cannot be loaded.\n" \
>> + "Reason: %s" % e
>> + sys.exit(1)
>> + elif opt in ('-k', '--keep_passed'):
>> + cleanup = False
>> + elif opt in ('-v', '--verbose'):
>> + log_all = True
>> + elif opt in ('-s', '--seed'):
>> + seed = arg
>> + elif opt == '--config':
>> + try:
>> + config = json.loads(arg)
>> + except (TypeError, ValueError, NameError), e:
>> + print >>sys.stderr, \
>> + "Error: JSON array with the fuzzer configuration cannot" \
>> + " be loaded\nReason: %s" % e
>> + sys.exit(1)
>> +
>> + if not len(args) == 2:
>> + print >>sys.stderr, \
>> + "Expected two parameters\nTry 'runner.py --help'" \
>> + " for more information."
>> + sys.exit(1)
>> +
>> + work_dir = os.path.realpath(args[0])
>> + # run_log is created in 'main', because multiple tests are expected to
>> + # log in it
>> + run_log = os.path.join(work_dir, 'run.log')
>> +
>> + # Add the path to the image generator module to sys.path
>> + sys.path.append(os.path.realpath(os.path.dirname(args[1])))
>> + # Remove a script extension from image generator module if any
>> + generator_name = os.path.splitext(os.path.basename(args[1]))[0]
>> +
>> + try:
>> + image_generator = __import__(generator_name)
>> + except ImportError, e:
>> + print >>sys.stderr, \
>> + "Error: The image generator '%s' cannot be imported.\n" \
>> + "Reason: %s" % (generator_name, e)
>> + sys.exit(1)
>> +
>> + # Enable core dumps
>> + resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
>> + # If a seed is specified, only one test will be executed.
>> + # Otherwise runner will terminate after a keyboard interruption
>> + for test_id in count(1):
>> + try:
>> + run_test(str(test_id), seed, work_dir, run_log, cleanup,
>> + log_all, command, config)
>> + except (KeyboardInterrupt, SystemExit):
>> + sys.exit(1)
>> +
>> + if seed is not None:
>> + break
>> --
>> 1.9.3
>>
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution
2014-08-08 8:58 ` M.Kustova
@ 2014-08-08 11:08 ` Fam Zheng
0 siblings, 0 replies; 16+ messages in thread
From: Fam Zheng @ 2014-08-08 11:08 UTC (permalink / raw)
To: M.Kustova; +Cc: Kevin Wolf, qemu-devel, Stefan Hajnoczi
On Fri, 08/08 12:58, M.Kustova wrote:
> On Fri, Aug 8, 2014 at 12:50 PM, Fam Zheng <famz@redhat.com> wrote:
> > On Wed, 08/06 17:12, Maria Kustova wrote:
> >> The purpose of the test runner is to prepare the test environment (e.g. create
> >> a work directory, a test image, etc), execute a program under test with
> >> parameters, indicate a test failure if the program was killed during the test
> >> execution and collect core dumps, logs and other test artifacts.
> >>
> >> The test runner doesn't depend on an image format or a program will be tested,
> >> so it can be used with any external image generator and program under test.
> >>
> >> Signed-off-by: Maria Kustova <maria.k@catit.be>
> >> ---
> >> tests/image-fuzzer/runner/runner.py | 405 ++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 405 insertions(+)
> >> create mode 100755 tests/image-fuzzer/runner/runner.py
> >>
> >> diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
> >> new file mode 100755
> >> index 0000000..3fa7fca
> >> --- /dev/null
> >> +++ b/tests/image-fuzzer/runner/runner.py
> >> @@ -0,0 +1,405 @@
> >> +#!/usr/bin/env python
> >> +
> >> +# Tool for running fuzz tests
> >> +#
> >> +# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
> >> +#
> >> +# This program is free software: you can redistribute it and/or modify
> >> +# it under the terms of the GNU General Public License as published by
> >> +# the Free Software Foundation, either version 2 of the License, or
> >> +# (at your option) any later version.
> >> +#
> >> +# This program is distributed in the hope that it will be useful,
> >> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> >> +# GNU General Public License for more details.
> >> +#
> >> +# You should have received a copy of the GNU General Public License
> >> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> >> +#
> >> +
> >> +import sys
> >> +import os
> >> +import signal
> >> +import subprocess
> >> +import random
> >> +import shutil
> >> +from itertools import count
> >> +import getopt
> >> +import StringIO
> >> +import resource
> >> +
> >> +try:
> >> + import json
> >> +except ImportError:
> >> + try:
> >> + import simplejson as json
> >> + except ImportError:
> >> + print >>sys.stderr, \
> >> + "Warning: Module for JSON processing is not found.\n" \
> >> + "'--config' and '--command' options are not supported."
> >> +
> >> +# Backing file sizes in MB
> >> +MAX_BACKING_FILE_SIZE = 10
> >> +MIN_BACKING_FILE_SIZE = 1
> >> +
> >> +
> >> +def multilog(msg, *output):
> >> + """ Write an object to all of specified file descriptors."""
> >> + for fd in output:
> >> + fd.write(msg)
> >> + fd.flush()
> >> +
> >> +
> >> +def str_signal(sig):
> >> + """ Convert a numeric value of a system signal to the string one
> >> + defined by the current operational system.
> >> + """
> >> + for k, v in signal.__dict__.items():
> >> + if v == sig:
> >> + return k
> >> +
> >> +
> >> +def run_app(fd, q_args):
> >> + """Start an application with specified arguments and return its exit code
> >> + or kill signal depending on the result of execution.
> >> + """
> >> + devnull = open('/dev/null', 'r+')
> >> + process = subprocess.Popen(q_args, stdin=devnull,
> >> + stdout=subprocess.PIPE,
> >> + stderr=subprocess.PIPE)
> >> + out, err = process.communicate()
> >> + fd.write(out)
> >> + fd.write(err)
> >> + return process.returncode
> >> +
> >> +
> >> +class TestException(Exception):
> >> + """Exception for errors risen by TestEnv objects."""
> >> + pass
> >> +
> >> +
> >> +class TestEnv(object):
> >> +
> >> + """Test object.
> >> +
> >> + The class sets up test environment, generates backing and test images
> >> + and executes application under tests with specified arguments and a test
> >> + image provided.
> >> +
> >> + All logs are collected.
> >> +
> >> + The summary log will contain short descriptions and statuses of tests in
> >> + a run.
> >> +
> >> + The test log will include application (e.g. 'qemu-img') logs besides info
> >> + sent to the summary log.
> >> + """
> >> +
> >> + def __init__(self, test_id, seed, work_dir, run_log,
> >> + cleanup=True, log_all=False):
> >> + """Set test environment in a specified work directory.
> >> +
> >> + Path to qemu-img and qemu-io will be retrieved from 'QEMU_IMG' and
> >> + 'QEMU_IO' environment variables.
> >> + """
> >> + if seed is not None:
> >> + self.seed = seed
> >> + else:
> >> + self.seed = str(random.randint(0, sys.maxint))
> >> + random.seed(self.seed)
> >> +
> >> + self.init_path = os.getcwd()
> >> + self.work_dir = work_dir
> >> + self.current_dir = os.path.join(work_dir, 'test-' + test_id)
> >> + self.qemu_img = os.environ.get('QEMU_IMG', 'qemu-img')\
> >> + .strip().split(' ')
> >
> > Nitpicking. I think split(' ') doesn't make sense, this could instead be:
> >
> > self.qemu_img = [os.environ.get('QEMU_IMG', 'qemu-img').strip()]
> >
> > Otherwise user won't be able to pass in a QEMU_IMG path with space.
> >
> > Corner case, though. Otherwise looks good,
>
> This functionality was inherited from qemu-iotests/iotests.py.
> Rationale for this splitting was that environment variables can
> contain call arguments as well.
> Ruining paths with white spaces was a reasonable trade-off.
OK, thanks for explanation.
Fam
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH V5 3/5] fuzz: Fuzzing functions for qcow2 images
2014-08-06 13:12 [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Maria Kustova
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 1/5] docs: Specification for the image fuzzer Maria Kustova
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 2/5] runner: Tool for fuzz tests execution Maria Kustova
@ 2014-08-06 13:12 ` Maria Kustova
2014-08-08 13:06 ` Stefan Hajnoczi
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 4/5] layout: Generator of fuzzed " Maria Kustova
` (2 subsequent siblings)
5 siblings, 1 reply; 16+ messages in thread
From: Maria Kustova @ 2014-08-06 13:12 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, famz, Maria Kustova, stefanha
The fuzz submodule of the qcow2 image generator contains fuzzing functions for
image fields.
Each fuzzing function contains a list of constraints and a call of a helper
function that randomly selects a fuzzed value satisfied to one of constraints.
For now constraints include only known as invalid or potentially dangerous
values. But after investigation of code coverage by fuzz tests they will be
expanded by heuristic values based on inner checks and flows of a program
under test.
Now fuzzing of a header, header extensions and a backing file name is
supported.
Signed-off-by: Maria Kustova <maria.k@catit.be>
---
tests/image-fuzzer/qcow2/fuzz.py | 327 +++++++++++++++++++++++++++++++++++++++
1 file changed, 327 insertions(+)
create mode 100644 tests/image-fuzzer/qcow2/fuzz.py
diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
new file mode 100644
index 0000000..a53c84f
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -0,0 +1,327 @@
+# Fuzzing functions for qcow2 fields
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import random
+
+
+UINT8 = 0xff
+UINT32 = 0xffffffff
+UINT64 = 0xffffffffffffffff
+# Most significant bit orders
+UINT32_M = 31
+UINT64_M = 63
+# Fuzz vectors
+UINT8_V = [0, 0x10, UINT8/4, UINT8/2 - 1, UINT8/2, UINT8/2 + 1, UINT8 - 1,
+ UINT8]
+UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1,
+ UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32]
+UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4,
+ UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1,
+ UINT64]
+STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
+ '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
+ '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p',
+ '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
+ '%s x 129', '%x x 257']
+
+
+def random_from_intervals(intervals):
+ """Select a random integer number from the list of specified intervals.
+
+ Each interval is a tuple of lower and upper limits of the interval. The
+ limits are included. Intervals in a list should not overlap.
+ """
+ total = reduce(lambda x, y: x + y[1] - y[0] + 1, intervals, 0)
+ r = random.randint(0, total - 1) + intervals[0][0]
+ for x in zip(intervals, intervals[1:]):
+ r = r + (r > x[0][1]) * (x[1][0] - x[0][1] - 1)
+ return r
+
+
+def random_bits(bit_ranges):
+ """Generate random binary mask with ones in the specified bit ranges.
+
+ Each bit_ranges is a list of tuples of lower and upper limits of bit
+ positions will be fuzzed. The limits are included. Random amount of bits
+ in range limits will be set to ones. The mask is returned in decimal
+ integer format.
+ """
+ bit_numbers = []
+ # Select random amount of random positions in bit_ranges
+ for rng in bit_ranges:
+ bit_numbers += random.sample(range(rng[0], rng[1] + 1),
+ random.randint(0, rng[1] - rng[0] + 1))
+ val = 0
+ # Set bits on selected positions to ones
+ for bit in bit_numbers:
+ val |= 1 << bit
+ return val
+
+
+def truncate_string(strings, length):
+ """Return strings truncated to specified length."""
+ if type(strings) == list:
+ return [s[:length] for s in strings]
+ else:
+ return strings[:length]
+
+
+def validator(current, pick, choices):
+ """Return a value not equal to the current selected by the pick
+ function from choices.
+ """
+ while True:
+ val = pick(choices)
+ if not val == current:
+ return val
+
+
+def int_validator(current, intervals):
+ """Return a random value from intervals not equal to the current.
+
+ This function is useful for selection from valid values except current one.
+ """
+ return validator(current, random_from_intervals, intervals)
+
+
+def bit_validator(current, bit_ranges):
+ """Return a random bit mask not equal to the current.
+
+ This function is useful for selection from valid values except current one.
+ """
+ return validator(current, random_bits, bit_ranges)
+
+
+def string_validator(current, strings):
+ """Return a random string value from the list not equal to the current.
+
+ This function is useful for selection from valid values except current one.
+ """
+ return validator(current, random.choice, strings)
+
+
+def selector(current, constraints, validate=int_validator):
+ """Select one value from all defined by constraints.
+
+ Each constraint produces one random value satisfying to it. The function
+ randomly selects one value satisfying at least one constraint (depending on
+ constraints overlaps).
+ """
+ def iter_validate(c):
+ """Apply validate() only to constraints represented as lists.
+
+ This auxiliary function replaces short circuit conditions not supported
+ in Python 2.4
+ """
+ if type(c) == list:
+ return validate(current, c)
+ else:
+ return c
+
+ fuzz_values = [iter_validate(c) for c in constraints]
+ # Remove current for cases it's implicitly specified in constraints
+ # Duplicate validator functionality to prevent decreasing of probability
+ # to get one of allowable values
+ # TODO: remove validators after implementation of intelligent selection
+ # of fields will be fuzzed
+ try:
+ fuzz_values.remove(current)
+ except ValueError:
+ pass
+ return random.choice(fuzz_values)
+
+
+def magic(current):
+ """Fuzz magic header field.
+
+ The function just returns the current magic value and provides uniformity
+ of calls for all fuzzing functions.
+ """
+ return current
+
+
+def version(current):
+ """Fuzz version header field."""
+ constraints = UINT32_V + [
+ [(2, 3)], # correct values
+ [(0, 1), (4, UINT32)]
+ ]
+ return selector(current, constraints)
+
+
+def backing_file_offset(current):
+ """Fuzz backing file offset header field."""
+ constraints = UINT64_V
+ return selector(current, constraints)
+
+
+def backing_file_size(current):
+ """Fuzz backing file size header field."""
+ constraints = UINT32_V
+ return selector(current, constraints)
+
+
+def cluster_bits(current):
+ """Fuzz cluster bits header field."""
+ constraints = UINT32_V + [
+ [(9, 20)], # correct values
+ [(0, 9), (20, UINT32)]
+ ]
+ return selector(current, constraints)
+
+
+def size(current):
+ """Fuzz image size header field."""
+ constraints = UINT64_V
+ return selector(current, constraints)
+
+
+def crypt_method(current):
+ """Fuzz crypt method header field."""
+ constraints = UINT32_V + [
+ 1,
+ [(2, UINT32)]
+ ]
+ return selector(current, constraints)
+
+
+def l1_size(current):
+ """Fuzz L1 table size header field."""
+ constraints = UINT32_V
+ return selector(current, constraints)
+
+
+def l1_table_offset(current):
+ """Fuzz L1 table offset header field."""
+ constraints = UINT64_V
+ return selector(current, constraints)
+
+
+def refcount_table_offset(current):
+ """Fuzz refcount table offset header field."""
+ constraints = UINT64_V
+ return selector(current, constraints)
+
+
+def refcount_table_clusters(current):
+ """Fuzz refcount table clusters header field."""
+ constraints = UINT32_V
+ return selector(current, constraints)
+
+
+def nb_snapshots(current):
+ """Fuzz number of snapshots header field."""
+ constraints = UINT32_V
+ return selector(current, constraints)
+
+
+def snapshots_offset(current):
+ """Fuzz snapshots offset header field."""
+ constraints = UINT64_V
+ return selector(current, constraints)
+
+
+def incompatible_features(current):
+ """Fuzz incompatible features header field."""
+ constraints = [
+ [(0, 1)], # allowable values
+ [(0, UINT64_M)]
+ ]
+ return selector(current, constraints, bit_validator)
+
+
+def compatible_features(current):
+ """Fuzz compatible features header field."""
+ constraints = [
+ [(0, UINT64_M)]
+ ]
+ return selector(current, constraints, bit_validator)
+
+
+def autoclear_features(current):
+ """Fuzz autoclear features header field."""
+ constraints = [
+ [(0, UINT64_M)]
+ ]
+ return selector(current, constraints, bit_validator)
+
+
+def refcount_order(current):
+ """Fuzz number of refcount order header field."""
+ constraints = UINT32_V
+ return selector(current, constraints)
+
+
+def header_length(current):
+ """Fuzz number of refcount order header field."""
+ constraints = UINT32_V + [
+ 72,
+ 104,
+ [(0, UINT32)]
+ ]
+ return selector(current, constraints)
+
+
+def bf_name(current):
+ """Fuzz the backing file name."""
+ constraints = [
+ truncate_string(STRING_V, len(current))
+ ]
+ return selector(current, constraints, string_validator)
+
+
+def ext_magic(current):
+ """Fuzz magic field of a header extension."""
+ constraints = UINT32_V
+ return selector(current, constraints)
+
+
+def ext_length(current):
+ """Fuzz length field of a header extension."""
+ constraints = UINT32_V
+ return selector(current, constraints)
+
+
+def bf_format(current):
+ """Fuzz backing file format in the corresponding header extension."""
+ constraints = [
+ truncate_string(STRING_V, len(current)),
+ truncate_string(STRING_V, (len(current) + 7) & ~7) # Fuzz padding
+ ]
+ return selector(current, constraints, string_validator)
+
+
+def feature_type(current):
+ """Fuzz feature type field of a feature name table header extension."""
+ constraints = UINT8_V
+ return selector(current, constraints)
+
+
+def feature_bit_number(current):
+ """Fuzz bit number field of a feature name table header extension."""
+ constraints = UINT8_V
+ return selector(current, constraints)
+
+
+def feature_name(current):
+ """Fuzz feature name field of a feature name table header extension."""
+ constraints = [
+ truncate_string(STRING_V, len(current)),
+ truncate_string(STRING_V, 46) # Fuzz padding (field length = 46)
+ ]
+ return selector(current, constraints, string_validator)
--
1.9.3
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 3/5] fuzz: Fuzzing functions for qcow2 images
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 3/5] fuzz: Fuzzing functions for qcow2 images Maria Kustova
@ 2014-08-08 13:06 ` Stefan Hajnoczi
0 siblings, 0 replies; 16+ messages in thread
From: Stefan Hajnoczi @ 2014-08-08 13:06 UTC (permalink / raw)
To: Maria Kustova; +Cc: kwolf, famz, qemu-devel, stefanha, Maria Kustova
[-- Attachment #1: Type: text/plain, Size: 957 bytes --]
On Wed, Aug 06, 2014 at 05:12:48PM +0400, Maria Kustova wrote:
> The fuzz submodule of the qcow2 image generator contains fuzzing functions for
> image fields.
> Each fuzzing function contains a list of constraints and a call of a helper
> function that randomly selects a fuzzed value satisfied to one of constraints.
> For now constraints include only known as invalid or potentially dangerous
> values. But after investigation of code coverage by fuzz tests they will be
> expanded by heuristic values based on inner checks and flows of a program
> under test.
>
> Now fuzzing of a header, header extensions and a backing file name is
> supported.
>
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
> tests/image-fuzzer/qcow2/fuzz.py | 327 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 327 insertions(+)
> create mode 100644 tests/image-fuzzer/qcow2/fuzz.py
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH V5 4/5] layout: Generator of fuzzed qcow2 images
2014-08-06 13:12 [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Maria Kustova
` (2 preceding siblings ...)
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 3/5] fuzz: Fuzzing functions for qcow2 images Maria Kustova
@ 2014-08-06 13:12 ` Maria Kustova
2014-08-08 13:14 ` Stefan Hajnoczi
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 5/5] package: Public API for image-fuzzer/runner/runner.py Maria Kustova
2014-08-08 6:54 ` [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Stefan Hajnoczi
5 siblings, 1 reply; 16+ messages in thread
From: Maria Kustova @ 2014-08-06 13:12 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, famz, Maria Kustova, stefanha
The layout submodule of the qcow2 package creates a random valid image,
randomly selects some amount of its fields, fuzzes them and write the fuzzed
image to the file. Fuzzing process can be controlled by an external
configuration.
Signed-off-by: Maria Kustova <maria.k@catit.be>
---
tests/image-fuzzer/qcow2/layout.py | 369 +++++++++++++++++++++++++++++++++++++
1 file changed, 369 insertions(+)
create mode 100644 tests/image-fuzzer/qcow2/layout.py
diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
new file mode 100644
index 0000000..4c08202
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -0,0 +1,369 @@
+# Generator of fuzzed qcow2 images
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import random
+import struct
+import fuzz
+
+MAX_IMAGE_SIZE = 10 * (1 << 20)
+# Standard sizes
+UINT32_S = 4
+UINT64_S = 8
+
+
+class Field(object):
+
+ """Atomic image element (field).
+
+ The class represents an image field as quadruple of a data format
+ of value necessary for its packing to binary form, an offset from
+ the beginning of the image, a value and a name.
+
+ The field can be iterated as a list [format, offset, value].
+ """
+
+ __slots__ = ('fmt', 'offset', 'value', 'name')
+
+ def __init__(self, fmt, offset, val, name):
+ self.fmt = fmt
+ self.offset = offset
+ self.value = val
+ self.name = name
+
+ def __iter__(self):
+ return iter([self.fmt, self.offset, self.value])
+
+ def __repr__(self):
+ return "Field(fmt='%s', offset=%d, value=%s, name=%s)" % \
+ (self.fmt, self.offset, str(self.value), self.name)
+
+
+class FieldsList(object):
+
+ """List of fields.
+
+ The class allows access to a field in the list by its name and joins
+ several list in one via in-place addition.
+ """
+
+ def __init__(self, meta_data=None):
+ if meta_data is None:
+ self.data = []
+ else:
+ self.data = [Field(f[0], f[1], f[2], f[3])
+ for f in meta_data]
+
+ def __getitem__(self, name):
+ return [x for x in self.data if x.name == name]
+
+ def __iter__(self):
+ return iter(self.data)
+
+ def __iadd__(self, other):
+ self.data += other.data
+ return self
+
+ def __len__(self):
+ return len(self.data)
+
+
+class Image(object):
+
+ """ Qcow2 image object.
+
+ This class allows to create qcow2 images with random valid structures and
+ values, fuzz them via external qcow2.fuzz module and write the result to
+ a file.
+ """
+
+ @staticmethod
+ def _size_params():
+ """Generate a random image size aligned to a random correct
+ cluster size.
+ """
+ cluster_bits = random.randrange(9, 21)
+ cluster_size = 1 << cluster_bits
+ img_size = random.randrange(0, MAX_IMAGE_SIZE + 1, cluster_size)
+ return (cluster_bits, img_size)
+
+ @staticmethod
+ def _header(cluster_bits, img_size, backing_file_name=None):
+ """Generate a random valid header."""
+ meta_header = [
+ ['>4s', 0, "QFI\xfb", 'magic'],
+ ['>I', 4, random.randint(2, 3), 'version'],
+ ['>Q', 8, 0, 'backing_file_offset'],
+ ['>I', 16, 0, 'backing_file_size'],
+ ['>I', 20, cluster_bits, 'cluster_bits'],
+ ['>Q', 24, img_size, 'size'],
+ ['>I', 32, 0, 'crypt_method'],
+ ['>I', 36, 0, 'l1_size'],
+ ['>Q', 40, 0, 'l1_table_offset'],
+ ['>Q', 48, 0, 'refcount_table_offset'],
+ ['>I', 56, 0, 'refcount_table_clusters'],
+ ['>I', 60, 0, 'nb_snapshots'],
+ ['>Q', 64, 0, 'snapshots_offset'],
+ ['>Q', 72, 0, 'incompatible_features'],
+ ['>Q', 80, 0, 'compatible_features'],
+ ['>Q', 88, 0, 'autoclear_features'],
+ # Only refcount_order = 4 is supported by current (07.2014)
+ # implementation of QEMU
+ ['>I', 96, 4, 'refcount_order'],
+ ['>I', 100, 0, 'header_length']
+ ]
+ v_header = FieldsList(meta_header)
+
+ if v_header['version'][0].value == 2:
+ v_header['header_length'][0].value = 72
+ else:
+ v_header['incompatible_features'][0].value = random.getrandbits(2)
+ v_header['compatible_features'][0].value = random.getrandbits(1)
+ v_header['header_length'][0].value = 104
+
+ max_header_len = struct.calcsize(v_header['header_length'][0].fmt) + \
+ v_header['header_length'][0].offset
+ end_of_extension_area_len = 2 * UINT32_S
+ free_space = (1 << cluster_bits) - (max_header_len +
+ end_of_extension_area_len)
+ # If the backing file name specified and there is enough space for it
+ # in the first cluster, then it's placed in the very end of the first
+ # cluster.
+ if (backing_file_name is not None) and \
+ (free_space >= len(backing_file_name)):
+ v_header['backing_file_size'][0].value = len(backing_file_name)
+ v_header['backing_file_offset'][0].value = (1 << cluster_bits) - \
+ len(backing_file_name)
+
+ return v_header
+
+ @staticmethod
+ def _backing_file_name(header, backing_file_name=None):
+ """Add the name of the backing file at the offset specified
+ in the header.
+ """
+ if (backing_file_name is not None) and \
+ (not header['backing_file_offset'][0].value == 0):
+ data_len = len(backing_file_name)
+ data_fmt = '>' + str(data_len) + 's'
+ data_field = FieldsList([
+ [data_fmt, header['backing_file_offset'][0].value,
+ backing_file_name, 'bf_name']
+ ])
+ else:
+ data_field = FieldsList()
+
+ return data_field
+
+ @staticmethod
+ def _backing_file_format(header, backing_file_fmt=None):
+ """Generate the header extension for the backing file
+ format.
+ """
+ ext = FieldsList()
+ offset = struct.calcsize(header['header_length'][0].fmt) + \
+ header['header_length'][0].offset
+
+ if backing_file_fmt is not None:
+ # Calculation of the free space available in the first cluster
+ end_of_extension_area_len = 2 * UINT32_S
+ high_border = (header['backing_file_offset'][0].value or
+ ((1 << header['cluster_bits'][0].value) - 1)) - \
+ end_of_extension_area_len
+ free_space = high_border - offset
+ ext_size = 2 * UINT32_S + ((len(backing_file_fmt) + 7) & ~7)
+
+ if free_space >= ext_size:
+ ext_data_len = len(backing_file_fmt)
+ ext_data_fmt = '>' + str(ext_data_len) + 's'
+ ext_padding_len = 7 - (ext_data_len - 1) % 8
+ ext = FieldsList([
+ ['>I', offset, 0xE2792ACA, 'ext_magic'],
+ ['>I', offset + UINT32_S, ext_data_len, 'ext_length'],
+ [ext_data_fmt, offset + UINT32_S * 2, backing_file_fmt,
+ 'bf_format']
+ ])
+ offset = ext['bf_format'][0].offset + \
+ struct.calcsize(ext['bf_format'][0].fmt) + \
+ ext_padding_len
+ return (ext, offset)
+
+ @staticmethod
+ def _feature_name_table(header, offset):
+ """Generate a random header extension for names of features used in
+ the image.
+ """
+ def gen_feat_ids():
+ """Return random feature type and feature bit."""
+ return (random.randint(0, 2), random.randint(0, 63))
+
+ end_of_extension_area_len = 2 * UINT32_S
+ high_border = (header['backing_file_offset'][0].value or
+ (1 << header['cluster_bits'][0].value) - 1) - \
+ end_of_extension_area_len
+ free_space = high_border - offset
+ # Sum of sizes of 'magic' and 'length' header extension fields
+ ext_header_len = 2 * UINT32_S
+ fnt_entry_size = 6 * UINT64_S
+ num_fnt_entries = min(10, (free_space - ext_header_len) /
+ fnt_entry_size)
+ if not num_fnt_entries == 0:
+ feature_tables = []
+ feature_ids = []
+ inner_offset = offset + ext_header_len
+ feat_name = 'some cool feature'
+ while len(feature_tables) < num_fnt_entries * 3:
+ feat_type, feat_bit = gen_feat_ids()
+ # Remove duplicates
+ while (feat_type, feat_bit) in feature_ids:
+ feat_type, feat_bit = gen_feat_ids()
+ feature_ids.append((feat_type, feat_bit))
+ feat_fmt = '>' + str(len(feat_name)) + 's'
+ feature_tables += [['B', inner_offset,
+ feat_type, 'feature_type'],
+ ['B', inner_offset + 1, feat_bit,
+ 'feature_bit_number'],
+ [feat_fmt, inner_offset + 2,
+ feat_name, 'feature_name']
+ ]
+ inner_offset += fnt_entry_size
+ # No padding for the extension is necessary, because
+ # the extension length is multiple of 8
+ ext = FieldsList([
+ ['>I', offset, 0x6803f857, 'ext_magic'],
+ # One feature table contains 3 fields and takes 48 bytes
+ ['>I', offset + UINT32_S, len(feature_tables) / 3 * 48,
+ 'ext_length']
+ ] + feature_tables)
+ offset = inner_offset
+ else:
+ ext = FieldsList()
+
+ return (ext, offset)
+
+ @staticmethod
+ def _end_of_extension_area(offset):
+ """Generate a mandatory header extension marking end of header
+ extensions.
+ """
+ ext = FieldsList([
+ ['>I', offset, 0, 'ext_magic'],
+ ['>I', offset + UINT32_S, 0, 'ext_length']
+ ])
+ return ext
+
+ def __init__(self, backing_file_name=None, backing_file_fmt=None):
+ """Create a random valid qcow2 image with the correct inner structure
+ and allowable values.
+ """
+ # Image size is saved as an attribute for the runner needs
+ cluster_bits, self.image_size = self._size_params()
+ # Saved as an attribute, because it's necessary for writing
+ self.cluster_size = 1 << cluster_bits
+ self.header = self._header(cluster_bits, self.image_size,
+ backing_file_name)
+ self.backing_file_name = self._backing_file_name(self.header,
+ backing_file_name)
+ self.backing_file_format, \
+ offset = self._backing_file_format(self.header,
+ backing_file_fmt)
+ self.feature_name_table, \
+ offset = self._feature_name_table(self.header, offset)
+ self.end_of_extension_area = self._end_of_extension_area(offset)
+ # Container for entire image
+ self.data = FieldsList()
+ # Percentage of fields will be fuzzed
+ self.bias = random.uniform(0.2, 0.5)
+
+ def __iter__(self):
+ return iter([self.header,
+ self.backing_file_format,
+ self.feature_name_table,
+ self.end_of_extension_area,
+ self.backing_file_name])
+
+ def _join(self):
+ """Join all image structure elements as header, tables, etc in one
+ list of fields.
+ """
+ if len(self.data) == 0:
+ for v in self:
+ self.data += v
+
+ def fuzz(self, fields_to_fuzz=None):
+ """Fuzz an image by corrupting values of a random subset of its fields.
+
+ Without parameters the method fuzzes an entire image.
+ If 'fields_to_fuzz' is specified then only fields in this list will be
+ fuzzed. 'fields_to_fuzz' can contain both individual fields and more
+ general image elements as a header or tables.
+ In the first case the field will be fuzzed always.
+ In the second a random subset of fields will be selected and fuzzed.
+ """
+ def coin():
+ """Return boolean value proportional to a portion of fields to be
+ fuzzed.
+ """
+ return random.random() < self.bias
+
+ if fields_to_fuzz is None:
+ self._join()
+ for field in self.data:
+ if coin():
+ field.value = getattr(fuzz, field.name)(field.value)
+ else:
+ for item in fields_to_fuzz:
+ if len(item) == 1:
+ for field in getattr(self, item[0]):
+ if coin():
+ field.value = getattr(fuzz,
+ field.name)(field.value)
+ else:
+ for field in getattr(self, item[0])[item[1]]:
+ try:
+ field.value = getattr(fuzz, field.name)(
+ field.value)
+ except AttributeError:
+ # Some fields can be skipped depending on
+ # references, e.g. FNT header extension is not
+ # generated for a feature mask header field
+ # equal to zero
+ pass
+
+ def write(self, filename):
+ """Write an entire image to the file."""
+ image_file = open(filename, 'w')
+ self._join()
+ for field in self.data:
+ image_file.seek(field.offset)
+ image_file.write(struct.pack(field.fmt, field.value))
+ image_file.seek(0, 2)
+ size = image_file.tell()
+ rounded = (size + self.cluster_size - 1) & ~(self.cluster_size - 1)
+ if rounded > size:
+ image_file.seek(rounded - 1)
+ image_file.write("\0")
+ image_file.close()
+
+
+def create_image(test_img_path, backing_file_name=None, backing_file_fmt=None,
+ fields_to_fuzz=None):
+ """Create a fuzzed image and write it to the specified file."""
+ image = Image(backing_file_name, backing_file_fmt)
+ image.fuzz(fields_to_fuzz)
+ image.write(test_img_path)
+ return image.image_size
--
1.9.3
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 4/5] layout: Generator of fuzzed qcow2 images
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 4/5] layout: Generator of fuzzed " Maria Kustova
@ 2014-08-08 13:14 ` Stefan Hajnoczi
0 siblings, 0 replies; 16+ messages in thread
From: Stefan Hajnoczi @ 2014-08-08 13:14 UTC (permalink / raw)
To: Maria Kustova; +Cc: kwolf, famz, qemu-devel, stefanha, Maria Kustova
[-- Attachment #1: Type: text/plain, Size: 603 bytes --]
On Wed, Aug 06, 2014 at 05:12:49PM +0400, Maria Kustova wrote:
> The layout submodule of the qcow2 package creates a random valid image,
> randomly selects some amount of its fields, fuzzes them and write the fuzzed
> image to the file. Fuzzing process can be controlled by an external
> configuration.
>
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
> tests/image-fuzzer/qcow2/layout.py | 369 +++++++++++++++++++++++++++++++++++++
> 1 file changed, 369 insertions(+)
> create mode 100644 tests/image-fuzzer/qcow2/layout.py
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH V5 5/5] package: Public API for image-fuzzer/runner/runner.py
2014-08-06 13:12 [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Maria Kustova
` (3 preceding siblings ...)
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 4/5] layout: Generator of fuzzed " Maria Kustova
@ 2014-08-06 13:12 ` Maria Kustova
2014-08-08 13:14 ` Stefan Hajnoczi
2014-08-08 6:54 ` [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Stefan Hajnoczi
5 siblings, 1 reply; 16+ messages in thread
From: Maria Kustova @ 2014-08-06 13:12 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, famz, Maria Kustova, stefanha
__init__.py provides the public API required by the test runner
Signed-off-by: Maria Kustova <maria.k@catit.be>
---
tests/image-fuzzer/qcow2/__init__.py | 1 +
1 file changed, 1 insertion(+)
create mode 100644 tests/image-fuzzer/qcow2/__init__.py
diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-fuzzer/qcow2/__init__.py
new file mode 100644
index 0000000..e2ebe19
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/__init__.py
@@ -0,0 +1 @@
+from layout import create_image
--
1.9.3
^ permalink raw reply related [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 5/5] package: Public API for image-fuzzer/runner/runner.py
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 5/5] package: Public API for image-fuzzer/runner/runner.py Maria Kustova
@ 2014-08-08 13:14 ` Stefan Hajnoczi
0 siblings, 0 replies; 16+ messages in thread
From: Stefan Hajnoczi @ 2014-08-08 13:14 UTC (permalink / raw)
To: Maria Kustova; +Cc: kwolf, famz, qemu-devel, stefanha, Maria Kustova
[-- Attachment #1: Type: text/plain, Size: 389 bytes --]
On Wed, Aug 06, 2014 at 05:12:50PM +0400, Maria Kustova wrote:
> __init__.py provides the public API required by the test runner
>
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
> tests/image-fuzzer/qcow2/__init__.py | 1 +
> 1 file changed, 1 insertion(+)
> create mode 100644 tests/image-fuzzer/qcow2/__init__.py
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support
2014-08-06 13:12 [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Maria Kustova
` (4 preceding siblings ...)
2014-08-06 13:12 ` [Qemu-devel] [PATCH V5 5/5] package: Public API for image-fuzzer/runner/runner.py Maria Kustova
@ 2014-08-08 6:54 ` Stefan Hajnoczi
2014-08-08 6:58 ` M.Kustova
5 siblings, 1 reply; 16+ messages in thread
From: Stefan Hajnoczi @ 2014-08-08 6:54 UTC (permalink / raw)
To: Maria Kustova; +Cc: kwolf, famz, qemu-devel, stefanha, Maria Kustova
[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]
On Wed, Aug 06, 2014 at 05:12:45PM +0400, Maria Kustova wrote:
>
> This patch series introduces the image fuzzer, a tool for stability and
> reliability testing.
> Its approach is to run large amount of tests in background. During every test a
> program (e.g. qemu-img) is called to read or modify an invalid test image.
> A test image has valid inner structure defined by its format specification with
> some fields having random invalid values.
>
> Patch 1 contains documentation for the image fuzzer, patch 2 is the test runner
> and remaining ones relate to the image generator for qcow2 format.
>
> This patch series was created for the 'block-next' branch.
>
> v4 -> v5:
> Runner:
> * Added a warning message if a backing file failed to be created (based on
> the review of Fam Zheng)
> * Back up a test image before every test command
> * Fixed always logged messages of a program under test
> * Added a warning message if a wrong name of a program under test is specified
> * Made offset and length qemu-io arguments multiple of a sector size
> * Print all errors to stderr
Can runner.py be moved to tests/image-fuzzer/runner.py?
That way it's immediately clear how to launch the fuzzer when looking at
the image-fuzzer/ directory.
Stefan
[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support
2014-08-08 6:54 ` [Qemu-devel] [PATCH V5 0/5] tests: Add the image fuzzer with qcow2 support Stefan Hajnoczi
@ 2014-08-08 6:58 ` M.Kustova
0 siblings, 0 replies; 16+ messages in thread
From: M.Kustova @ 2014-08-08 6:58 UTC (permalink / raw)
To: Stefan Hajnoczi; +Cc: Kevin Wolf, Fam Zheng, qemu-devel, Stefan Hajnoczi
On Fri, Aug 8, 2014 at 10:54 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Wed, Aug 06, 2014 at 05:12:45PM +0400, Maria Kustova wrote:
>>
>> This patch series introduces the image fuzzer, a tool for stability and
>> reliability testing.
>> Its approach is to run large amount of tests in background. During every test a
>> program (e.g. qemu-img) is called to read or modify an invalid test image.
>> A test image has valid inner structure defined by its format specification with
>> some fields having random invalid values.
>>
>> Patch 1 contains documentation for the image fuzzer, patch 2 is the test runner
>> and remaining ones relate to the image generator for qcow2 format.
>>
>> This patch series was created for the 'block-next' branch.
>>
>> v4 -> v5:
>> Runner:
>> * Added a warning message if a backing file failed to be created (based on
>> the review of Fam Zheng)
>> * Back up a test image before every test command
>> * Fixed always logged messages of a program under test
>> * Added a warning message if a wrong name of a program under test is specified
>> * Made offset and length qemu-io arguments multiple of a sector size
>> * Print all errors to stderr
>
> Can runner.py be moved to tests/image-fuzzer/runner.py?
>
> That way it's immediately clear how to launch the fuzzer when looking at
> the image-fuzzer/ directory.
>
Sure. There is no special requirements for the runner location.
> Stefan
^ permalink raw reply [flat|nested] 16+ messages in thread