[Qemu-devel] [PATCH 0/5] tests: Add the image fuzzer with qcow2 support

[Qemu-devel] [PATCH 0/5] tests: Add the image fuzzer with qcow2 support

[Maria Kustova]

This patch series introduces the image fuzzer, a tool for stability and
reliability testing.
Its approach is to run large amount of tests in background. During every test a
program (e.g. qemu-img) is called to read or modify an invalid test image.
A test image has valid inner structure defined by its format specification with
some fields having random invalid values.

Patch 1 contains documentation for the image fuzzer, patch 2 is the test runner
and remaining ones relate to the image generator for qcow2 format.

Maria Kustova (5):
  docs: Specification for the image fuzzer
  runner: Tool for fuzz tests execution
  fuzz: Fuzzing functions for qcow2 images
  layout: Generator of fuzzed qcow2 images
  package: Public API for image-fuzzer/runner/runner.py

 tests/image-fuzzer/docs/image-fuzzer.txt | 176 +++++++++++++++++
 tests/image-fuzzer/qcow2/__init__.py     |   1 +
 tests/image-fuzzer/qcow2/fuzz.py         | 329 +++++++++++++++++++++++++++++++
 tests/image-fuzzer/qcow2/layout.py       | 250 +++++++++++++++++++++++
 tests/image-fuzzer/runner/runner.py      | 270 +++++++++++++++++++++++++
 5 files changed, 1026 insertions(+)
 create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt
 create mode 100644 tests/image-fuzzer/qcow2/__init__.py
 create mode 100644 tests/image-fuzzer/qcow2/fuzz.py
 create mode 100644 tests/image-fuzzer/qcow2/layout.py
 create mode 100755 tests/image-fuzzer/runner/runner.py

-- 
1.9.3

[Qemu-devel] [PATCH 1/5] docs: Specification for the image fuzzer

[Maria Kustova]

'Overall fuzzer requirements' chapter contains the current product vision and
features done and to be done. This chapter is still in progress.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/docs/image-fuzzer.txt | 176 +++++++++++++++++++++++++++++++
 1 file changed, 176 insertions(+)
 create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt

diff --git a/tests/image-fuzzer/docs/image-fuzzer.txt b/tests/image-fuzzer/docs/image-fuzzer.txt
new file mode 100644
index 0000000..a9ed4b6
--- /dev/null
+++ b/tests/image-fuzzer/docs/image-fuzzer.txt
@@ -0,0 +1,176 @@
+Image fuzzer
+============
+
+Description
+-----------
+
+The goal of the image fuzzer is to catch crashes of qemu-io/qemu-img providing
+to them randomly corrupted images.
+Test images are generated from scratch and have valid inner structure with some
+elements, e.g. L1/L2 tables, having random invalid values.
+
+
+Test runner
+-----------
+
+The test runner generates test images, executes tests utilizing generated
+images, indicates their results and collect all test related artifacts (logs,
+core dumps, test images).
+The test means one start of a system under test (SUT), e.g. qemu-io, with
+specified arguments and one test image.
+By default, the test runner generates new tests and executes them till
+keyboard interruption. But if a test seed is specified via '-s' runner
+parameter, then only one test with this seed will be executed, after its finish
+the runner will exit.
+
+The runner uses an external image fuzzer to generate test images. An image
+generator should be specified as a mandatory parameter of the test runner.
+Details about interactions between the runner and fuzzers see "Module
+interfaces".
+
+The runner activates generation of core dumps during test executions, but it
+assumes that core dumps will be generated in the current working directory.
+For comprehensive test results, please, set up your test environment
+properly.
+
+Path to a SUT can be specified via environment variables (for now only
+qemu-img) or via '--binary' parameter of the test runner. For details about
+environment variables see qemu-iotests/check.
+
+
+Qcow2 image generator
+---------------------
+
+The 'qcow2' generator is a Python package providing 'create_image' method as
+a single public API. See details in 'Test runner/image fuzzer' in 'Module
+interfaces'.
+
+Qcow2 contains two submodules: fuzz.py and layout.py.
+
+'fuzz.py' contains all fuzzing functions one per image field. It's assumed that
+after code analysis every field will have own constraints for its value. For
+now only universal potentially dangerous values are used, e.g. type limits for
+integers or unsafe symbols as '%s' for strings. For bitmasks random amount of
+bits are set to ones. All fuzzed values are checked on non-equality to the
+current valid value of the field. In case of equality the value will be
+regenerated.
+
+'layout.py' creates a random valid image, fuzzes a random subset of the image
+fields by 'fuzz.py' module and writes a fuzzed image to the file specified.
+
+For now only header fields and header extensions are generated, the remaining
+file is filled with zeros.
+
+
+Module interfaces
+-----------------
+
+* Test runner/image fuzzer
+
+The runner calls an image generator specifying path to a test image file.
+An image generator is expected to provide 'create_image(test_img_path)' method
+that creates a test image and writes it to the specified file. The file should
+be created if it doesn't exist or overwritten otherwise.
+Random seed is set by the runner at every test execution for the regression
+purpose, so an image generator is not recommended to modify it internally.
+
+* Test runner/SUT
+
+A full test command is composed from the SUT, its arguments specified
+via '-c' runner parameter and the name of generated image. To indicate where
+a test image name should be placed TEST_IMG placeholder can be used.
+For example, for the next test command
+
+ qemu-img convert -O vmdk fuzzed_image test.vmdk
+
+a call of the image fuzzer will be following:
+
+ ./runner.py -b qemu-img -c 'convert -O vmdk TEST_IMG test.vmdk' work_dir qcow2
+
+
+Overall fuzzer requirements
+===========================
+
+Input data:
+----------
+
+ - image template (generator)
+ - work directory
+ - test run duration (optional)
+ - action vector (optional)
+ - seed (optional)
+ - SUT and its arguments (optional)
+
+
+Fuzzer requirements:
+-------------------
+
+1.  Should be able to inject random data
+2.  Should be able to select a random value from the manually pregenerated
+    vector (boundary values, e.g. max/min cluster size)
+3.  Image template should describe a general structure invariant for all
+    test images (image format description)
+4.  Image template should be autonomous and other fuzzer parts should not
+    relate on it
+5.  Image template should contain reference rules (not only block+size
+    description)
+6.  Should generate the test image with the correct structure based on an image
+    template
+7.  Should accept a seed as an argument (for regression purpose)
+8.  Should generate a seed if it is not specified as an input parameter.
+9.  The same seed should generate the same image, if no or the same action
+    vector is specified
+10. Should accept a vector of actions as an argument (for test reproducing and
+    for test case specification, e.g. group of tests for header structure,
+    group of test for snapshots, etc)
+11. Action vector should be randomly generated from the pool of available
+    actions, if it is not specified as an input parameter
+12. Pool of actions should be defined automatically based on an image template
+13. Should accept a SUT and its call parameters as an argument or select them
+    randomly otherwise. As far as it's expected to be rarely changed, the list
+    of all possible test commands can be available in the test runner
+    internally.
+14. Should accept a test run duration as an argument. Tests should be executed
+    during a minimum period from a test run duration and time while fuzzer can
+    generate different test images
+15. Should support an external cancellation of a test run
+16. Seed and action vector should be logged (for regression purpose)
+17. All files related to test result should be collected: a test image,
+    SUT logs, fuzzer logs and crash dumps
+18. Should be compatible with python version 2.4-2.7
+19. Usage of external libraries should be limited as much as possible.
+
+
+Image formats:
+-------------
+
+Main target image format is qcow2, but support of image templates should
+provide an ability to add any other image format.
+
+
+Effectiveness:
+-------------
+
+Fuzzer can be controlled via template, seed and action vector;
+it makes the fuzzer itself invariant to an image format and test logic.
+It should be able to perform rather complex and precise tests, that can be
+specified via action vector. Otherwise, knowledge about an image structure
+allows the fuzzer to generate the pool of all available areas can be fuzzed
+and randomly select some of them and so compose its own action vector.
+Also complexity of a template defines complexity of the fuzzer, so its
+functionality can be varied from simple model-independent fuzzing to smart
+model-based one.
+
+
+Glossary:
+--------
+
+Action vector is a sequence of structure elements retrieved from an image
+format, each of them will be fuzzed for the test image. It's a subset of
+elements of the action pool. Example: header, refcount table, etc.
+Action pool is all available elements of an image structure that generated
+automatically from an image template.
+Image template is a formal description of an image structure and relations
+between image blocks.
+Test image is an output image of the fuzzer defined by the current seed and
+action vector.
-- 
1.9.3

[Qemu-devel] [PATCH 2/5] runner: Tool for fuzz tests execution

[Maria Kustova]

The purpose of the test runner is to prepare test environment (e.g. create a
work directory, a test image, etc), execute the program under test with
parameters, indicate a test failure if the program was killed during test
execution and collect core dumps, logs and other test artifacts.

The test runner doesn't depend on image format or a program will be tested, so
it can be used with any external image generator and program under test.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/runner/runner.py | 270 ++++++++++++++++++++++++++++++++++++
 1 file changed, 270 insertions(+)
 create mode 100755 tests/image-fuzzer/runner/runner.py

diff --git a/tests/image-fuzzer/runner/runner.py b/tests/image-fuzzer/runner/runner.py
new file mode 100755
index 0000000..21de78e
--- /dev/null
+++ b/tests/image-fuzzer/runner/runner.py
@@ -0,0 +1,270 @@
+#!/usr/bin/env python
+
+# Tool for running fuzz tests
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import sys, os, signal
+from time import time
+import subprocess
+import random
+from itertools import count
+from shutil import rmtree
+import getopt
+import resource
+resource.setrlimit(resource.RLIMIT_CORE, (-1, -1))
+
+
+def multilog(msg, *output):
+    """ Write an object to all of specified file descriptors
+    """
+
+    for fd in output:
+        fd.write(msg)
+        fd.flush()
+
+
+def str_signal(sig):
+    """ Convert a numeric value of a system signal to the string one
+    defined by the current operational system
+    """
+
+    for k, v in signal.__dict__.items():
+        if v == sig:
+            return k
+
+
+class TestException(Exception):
+    """Exception for errors risen by TestEnv objects"""
+    pass
+
+
+class TestEnv(object):
+    """ Trivial test object
+
+    The class sets up test environment, generates a test image and executes
+    application under tests with specified arguments and a test image provided.
+    All logs are collected.
+    Summary log will contain short descriptions and statuses of tests in
+    a run.
+    Test log will include application (e.g. 'qemu-img') logs besides info sent
+    to the summary log.
+    """
+
+    def __init__(self, test_id, seed, work_dir, run_log, exec_bin=None,
+                 cleanup=True, log_all=False):
+        """Set test environment in a specified work directory.
+
+        Path to qemu_img will be retrieved from 'QEMU_IMG' environment
+        variable, if a test binary is not specified.
+        """
+
+        if seed is not None:
+            self.seed = seed
+        else:
+            self.seed = hash(time())
+
+        self.init_path = os.getcwd()
+        self.work_dir = work_dir
+        self.current_dir = os.path.join(work_dir, 'test-' + test_id)
+        if exec_bin is not None:
+            self.exec_bin = exec_bin.strip().split(' ')
+        else:
+            self.exec_bin = \
+                os.environ.get('QEMU_IMG', 'qemu-img').strip().split(' ')
+
+        try:
+            os.makedirs(self.current_dir)
+        except OSError:
+            e = sys.exc_info()[1]
+            print >>sys.stderr, \
+                "Error: The working directory '%s' cannot be used. Reason: %s"\
+                % (self.work_dir, e[1])
+            raise TestException
+        self.log = open(os.path.join(self.current_dir, "test.log"), "w")
+        self.parent_log = open(run_log, "a")
+        self.result = False
+        self.cleanup = cleanup
+        self.log_all = log_all
+
+    def _test_app(self, q_args):
+        """ Start application under test with specified arguments and return
+        an exit code or a kill signal depending on result of an execution.
+        """
+        devnull = open('/dev/null', 'r+')
+        return subprocess.call(self.exec_bin + q_args,
+                               stdin=devnull, stdout=self.log, stderr=self.log)
+
+    def execute(self, q_args):
+        """ Execute a test.
+
+        The method creates a test image, runs test app and analyzes its exit
+        status. If the application was killed by a signal, the test is marked
+        as failed.
+        """
+        os.chdir(self.current_dir)
+        # Seed initialization should be as close to image generation call
+        # as posssible to avoid a corruption of random sequence
+        random.seed(self.seed)
+        image_generator.create_image('test_image')
+        test_summary = "Seed: %s\nCommand: %s\nTest directory: %s\n" \
+                       % (self.seed, " ".join(q_args), self.current_dir)
+        try:
+            retcode = self._test_app(q_args)
+        except OSError:
+            e = sys.exc_info()[1]
+            multilog(test_summary + "Error: Start of '%s' failed. " \
+                     "Reason: %s\n\n" % (os.path.basename(self.exec_bin[0]),
+                                         e[1]),
+                     sys.stderr, self.log, self.parent_log)
+            raise TestException
+
+        if retcode < 0:
+            multilog(test_summary + "FAIL: Test terminated by signal %s\n\n"
+                     % str_signal(-retcode), sys.stderr, self.log,
+                     self.parent_log)
+        elif self.log_all:
+            multilog(test_summary + "PASS: Application exited with the code" +
+                     " '%d'\n\n" % retcode, sys.stdout, self.log,
+                     self.parent_log)
+            self.result = True
+        else:
+            self.result = True
+
+    def finish(self):
+        """ Restore environment after a test execution. Remove folders of
+        passed tests
+        """
+        self.log.close()
+        self.parent_log.close()
+        os.chdir(self.init_path)
+        if self.result and self.cleanup:
+            rmtree(self.current_dir)
+
+if __name__ == '__main__':
+
+    def usage():
+        print """
+        Usage: runner.py [OPTION...] TEST_DIR IMG_GENERATOR
+
+        Set up test environment in TEST_DIR and run a test in it. A module for
+        test image generation should be specified via IMG_GENERATOR. Use
+        TEST_IMG alias to mark the position in the command where a test image
+        name should be placed.
+        Example:
+        python runner.py -b ./qemu-img -c 'info TEST_IMG' /tmp/test ../qcow2
+
+        Optional arguments:
+          -h, --help                    display this help and exit
+          -b, --binary=PATH             path to the application under test,
+                                        by default "qemu-img" in PATH or
+                                        QEMU_IMG environment variables
+          -c, --command=STRING          execute the tested application
+                                        with arguments specified,
+                                        by default STRING="check"
+          -s, --seed=STRING             seed for a test image generation,
+                                        by default will be generated randomly
+          -k, --keep_passed             don't remove folders of passed tests
+          -v, --verbose                 log information about passed tests
+        """
+
+    def run_test(test_id, seed, work_dir, run_log, test_bin, cleanup, log_all,
+                 command):
+        """Setup environment for one test and execute this test"""
+        try:
+            test = TestEnv(test_id, seed, work_dir, run_log, test_bin, cleanup,
+                           log_all)
+        except TestException:
+            sys.exit(1)
+
+        # Python 2.4 doesn't support 'finally' and 'except' in the same 'try'
+        # block
+        try:
+            try:
+                test.execute(command)
+            # Silent exit on user break
+            except TestException:
+                sys.exit(1)
+        finally:
+            test.finish()
+
+    try:
+        opts, args = getopt.gnu_getopt(sys.argv[1:], 'c:hb:s:kv',
+                                       ['command=', 'help', 'binary=', 'seed=',
+                                        'keep_passed', 'verbose'])
+    except getopt.error:
+        e = sys.exc_info()[1]
+        print "Error: %s\n\nTry 'runner.py --help' for more information" % e
+        sys.exit(1)
+
+    command = ['check']
+    cleanup = True
+    log_all = False
+    test_bin = None
+    seed = None
+    for opt, arg in opts:
+        if opt in ('-h', '--help'):
+            usage()
+            sys.exit()
+        elif opt in ('-c', '--command'):
+            command = arg.split(" ")
+        elif opt in ('-k', '--keep_passed'):
+            cleanup = False
+        elif opt in ('-v', '--verbose'):
+            log_all = True
+        elif opt in ('-b', '--binary'):
+            test_bin = os.path.realpath(arg)
+        elif opt in ('-s', '--seed'):
+            seed = arg
+
+    if not len(args) == 2:
+        print "Missed parameter\nTry 'runner.py --help' " \
+            "for more information"
+        sys.exit(1)
+
+    work_dir = os.path.realpath(args[0])
+    # run_log is created in 'main', because multiple tests are expected to
+    # log in it
+    run_log = os.path.join(work_dir, 'run.log')
+
+    # Add the path to the image generator module to sys.path
+    sys.path.append(os.path.dirname(os.path.realpath(args[1])))
+    # Remove a script extension from image generator module if any
+    generator_name = os.path.splitext(os.path.basename(args[1]))[0]
+    try:
+        image_generator = __import__(generator_name)
+    except ImportError:
+        e = sys.exc_info()[1]
+        print "Error: The image generator '%s' cannot be imported.\n" \
+            "Reason: %s" % (generator_name, e)
+        sys.exit(1)
+
+    # Replace test image alias with its real name
+    for idx, item in enumerate(command):
+        if item == 'TEST_IMG':
+            command[idx] = 'test_image'
+    # If a seed is specified, only one test will be executed.
+    # Otherwise runner will terminate after a keyboard interruption
+    for test_id in count(1):
+        try:
+            run_test(str(test_id), seed, work_dir, run_log, test_bin, cleanup,
+                     log_all, command)
+        except (KeyboardInterrupt, SystemExit):
+            sys.exit(1)
+
+        if seed is not None:
+            break
-- 
1.9.3

[Qemu-devel] [PATCH 3/5] fuzz: Fuzzing functions for qcow2 images

[Maria Kustova]

Fuzz submodule of qcow2 image generator contains fuzzing functions for image
fields.
Each fuzzing function contains list of constraints and call of a helper
function that randomly selects a fuzzed value satisfied to one of constraints.
For now constraints are only known as invalid or potentially dangerous values.
But after investigation of code coverage by fuzz tests they will be expanded
by heuristic values based on inner checks and flows of a program under test.

Now fuzzing of header and header extensions is only supported.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/qcow2/fuzz.py | 329 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 329 insertions(+)
 create mode 100644 tests/image-fuzzer/qcow2/fuzz.py

diff --git a/tests/image-fuzzer/qcow2/fuzz.py b/tests/image-fuzzer/qcow2/fuzz.py
new file mode 100644
index 0000000..a81296e
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/fuzz.py
@@ -0,0 +1,329 @@
+# Fuzzing functions for qcow2 fields
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import random
+
+
+UINT32 = 0xffffffff
+UINT64 = 0xffffffffffffffff
+# Most significant bit orders
+UINT32_M = 31
+UINT64_M = 63
+# Fuzz vectors
+UINT32_V = [0, 0x100, 0x1000, 0x10000, 0x100000, UINT32/4, UINT32/2 - 1,
+            UINT32/2, UINT32/2 + 1, UINT32 - 1, UINT32]
+UINT64_V = UINT32_V + [0x1000000, 0x10000000, 0x100000000, UINT64/4,
+                       UINT64/2 - 1, UINT64/2, UINT64/2 + 1, UINT64 - 1,
+                       UINT64]
+STRING_V = ['%s%p%x%d', '.1024d', '%.2049d', '%p%p%p%p', '%x%x%x%x',
+            '%d%d%d%d', '%s%s%s%s', '%99999999999s', '%08x', '%%20d', '%%20n',
+            '%%20x', '%%20s', '%s%s%s%s%s%s%s%s%s%s', '%p%p%p%p%p%p%p%p%p%p',
+            '%#0123456x%08x%x%s%p%d%n%o%u%c%h%l%q%j%z%Z%t%i%e%g%f%a%C%S%08x%%',
+            '%s x 129', '%x x 257']
+
+
+def random_from_intervals(intervals):
+    """Select a random integer number from the list of specified intervals
+
+    Each interval is a tuple of lower and upper limits of the interval. The
+    limits are included. Intervals in a list should not overlap.
+    """
+    total = reduce(lambda x, y: x + y[1] - y[0] + 1, intervals, 0)
+    r = random.randint(0, total-1) + intervals[0][0]
+    temp = zip(intervals, intervals[1:])
+    for x in temp:
+        r = r + (r > x[0][1])*(x[1][0] - x[0][1] - 1)
+    return r
+
+
+def random_bits(bit_ranges):
+    """Generate random binary mask with ones in the specified bit ranges
+
+    Each bit_ranges is a list of tuples of lower and upper limits of bit
+    positions will be fuzzed. The limits are included. Random amount of bits
+    in range limits will be set to ones. The mask is returned in decimal
+    integer format.
+    """
+    bit_numbers = []
+    # Select random amount of random positions in bit_ranges
+    for rng in bit_ranges:
+        bit_numbers += random.sample(range(rng[0], rng[1] + 1),
+                                     random.randint(0, rng[1] - rng[0] + 1))
+    val = 0
+    # Set bits on selected possitions to ones
+    for bit in bit_numbers:
+        val |= 1 << bit
+    return val
+
+
+def truncate_string(strings, length):
+    """Return strings truncated to specified length"""
+    if type(strings) == list:
+        return [s[length:] for s in strings]
+    else:
+        return strings[length:]
+
+
+def int_validator(current, intervals):
+    """Return a random value from intervals not equal to the current.
+
+    This function is useful for selection from valid values except current one.
+    """
+    val = random_from_intervals(intervals)
+    if val == current:
+        return int_validator(current, intervals)
+    else:
+        return val
+
+
+def bit_validator(current, bit_ranges):
+    """Return a random bit mask not equal to the current.
+
+    This function is useful for selection from valid values except current one.
+    """
+
+    val = random_bits(bit_ranges)
+    if val == current:
+        return bit_validator(current, bit_ranges)
+    else:
+        return val
+
+
+def string_validator(current, strings):
+    """Return a random string value from the list not equal to the current.
+
+    This function is useful for selection from valid values except current one.
+    """
+
+    val = random.choice(strings)
+    if val == current:
+        return string_validator(current, strings)
+    else:
+        return val
+
+
+def selector(current, constraints, fmt=None):
+    """Select one value from all defined by constraints
+
+    Each constraint produces one random value satisfying to it. The function
+    randomly selects one value satisfying at least one constraint (depending on
+    constraints overlaps).
+    """
+    validate = {
+        'bitmask': bit_validator,
+        'string': string_validator
+    }.get(fmt, int_validator)
+
+    def iter_validate(c):
+        """Apply validate() only to constraints represented as lists
+
+        This auxiliary function replaces short circuit conditions not supported
+        in Python 2.4
+        """
+        if type(c) == list:
+            return validate(current, c)
+        else:
+            return c
+    fuzz_values = [iter_validate(c) for c in constraints]
+    # Remove current for cases it's implicitly specified in constraints
+    # Duplicate validator functionality to prevent decreasing of probability
+    # to get one of allowable values
+    # TODO: remove validators after implementation of intelligent selection
+    # of fields will be fuzzed
+    try:
+        fuzz_values.remove(current)
+    except ValueError:
+        pass
+    return random.choice(fuzz_values)
+
+
+def magic(current):
+    """Fuzz magic header field
+
+    The function just returns the current magic value and provides uniformity
+    of calls for all fuzzing functions
+    """
+    return current
+
+
+def version(current):
+    """Fuzz version header field"""
+    constraints = UINT32_V + [
+        [(2, 3)],  # correct values
+        [(0, 1), (4, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def backing_file_offset(current):
+    """Fuzz backing file offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def backing_file_size(current):
+    """Fuzz backing file size header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def cluster_bits(current):
+    """Fuzz cluster bits header field"""
+    constraints = UINT32_V + [
+        [(9, 20)],  # correct values
+        [(0, 9), (20, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def size(current):
+    """Fuzz image size header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def crypt_method(current):
+    """Fuzz crypt method header field"""
+    constraints = UINT32_V + [
+        1,
+        [(2, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def l1_size(current):
+    """Fuzz L1 table size header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def l1_table_offset(current):
+    """Fuzz L1 table offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def refcount_table_offset(current):
+    """Fuzz refcount table offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def refcount_table_clusters(current):
+    """Fuzz refcount table clusters header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def nb_snapshots(current):
+    """Fuzz number of snapshots header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def snapshots_offset(current):
+    """Fuzz snapshots offset header field"""
+    constraints = UINT64_V
+    return selector(current, constraints)
+
+
+def incompatible_features(current):
+    """Fuzz incompatible features header field"""
+    constraints = [
+        [(0, 1)],  # allowable values
+        [(0, UINT64_M)]
+    ]
+    return selector(current, constraints, 'bitmask')
+
+
+def compatible_features(current):
+    """Fuzz compatible features header field"""
+    constraints = [
+        [(0, UINT64_M)]
+    ]
+    return selector(current, constraints, 'bitmask')
+
+
+def autoclear_features(current):
+    """Fuzz autoclear features header field"""
+    constraints = [
+        [(0, UINT64_M)]
+    ]
+    return selector(current, constraints, 'bitmask')
+
+
+def refcount_order(current):
+    """Fuzz number of refcount order header field"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def header_length(current):
+    """Fuzz number of refcount order header field"""
+    constraints = UINT32_V + [
+        72,
+        104,
+        [(0, UINT32)]
+    ]
+    return selector(current, constraints)
+
+
+def ext_magic(current):
+    """Fuzz magic field of a header extension"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def ext_length(current):
+    """Fuzz length field of a header extension"""
+    constraints = UINT32_V
+    return selector(current, constraints)
+
+
+def bf_data(current):
+    """Fuzz backing file format in the corresponding header extension"""
+    constraints = [
+        truncate_string(STRING_V, len(current)),
+        truncate_string(STRING_V, (len(current) + 7) & ~7)  # Fuzz padding
+    ]
+    return selector(current, constraints, 'string')
+
+
+def feat_type(current):
+    """Fuzz feature type field of a feature name table header extension"""
+    constraints = [
+        [(0, 255)]
+    ]
+    return selector(current, constraints)
+
+
+def feat_bit_number(current):
+    """Fuzz bit number field of a feature name table header extension"""
+    constraints = [
+        [(0, 255)]
+    ]
+    return selector(current, constraints)
+
+
+def feat_name(current):
+    """Fuzz feature name field of a feature name table header extension"""
+    constraints = [
+        truncate_string(STRING_V, len(current)),
+        truncate_string(STRING_V, 48)  # Fuzz padding (field length = 48)
+    ]
+    return selector(current, constraints, 'string')
-- 
1.9.3

[Qemu-devel] [PATCH 4/5] layout: Generator of fuzzed qcow2 images

[Maria Kustova]

Layout submodule of qcow2 package creates a random valid image, randomly
selects some amount of its fields, fuzzes them and write the fuzzed image to
the file.
Now only header and header extensions are generated, remaining file is filled
by zeroes.

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/qcow2/layout.py | 250 +++++++++++++++++++++++++++++++++++++
 1 file changed, 250 insertions(+)
 create mode 100644 tests/image-fuzzer/qcow2/layout.py

diff --git a/tests/image-fuzzer/qcow2/layout.py b/tests/image-fuzzer/qcow2/layout.py
new file mode 100644
index 0000000..b296979
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/layout.py
@@ -0,0 +1,250 @@
+# Generator of fuzzed qcow2 images
+#
+# Copyright (C) 2014 Maria Kustova <maria.k@catit.be>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import random
+import struct
+import fuzz
+from itertools import repeat
+import copy
+
+MAX_IMAGE_SIZE = 10*2**20
+# Standard sizes
+UINT32_S = 4
+UINT64_S = 8
+
+# Percentage of fields will be fuzzed
+BIAS = random.uniform(0.1, 0.4)
+
+
+class Field(object):
+    """Describes an image field as a triplet of a data format necessary for
+    packing, an offset to the beginning of the image and value of the field.
+
+    Can be iterated as a list [format, offset, value]
+    """
+    __slots__ = ('fmt', 'offset', 'value')
+
+    def __init__(self, fmt, offset, val):
+        self.fmt = fmt
+        self.offset = offset
+        self.value = val
+
+    def __iter__(self):
+        return (x for x in [self.fmt, self.offset, self.value])
+
+    def __repr__(self):
+        return "Field(fmt='%s', offset=%d, value=%s)" % (self.fmt, self.offset,
+                                                         str(self.value))
+
+
+def walk(v_struct, func):
+    """Walk via structure and apply the specified function to all Field()
+    elements
+    """
+    if isinstance(v_struct, list):
+        for item in v_struct:
+            walk(item, func)
+    else:
+        for k, v in v_struct.items():
+            if isinstance(v, list):
+                walk(v, func)
+            else:
+                func(v, k)
+
+
+def fuzz_struct(structure):
+    """Select part of fields in the specified structure and assign them invalid
+    values
+
+    """
+    def coin():
+        """Return boolean value proportional to a portion of fields to be
+        fuzzed
+        """
+        return random.random() < BIAS
+
+    def iter_fuzz(field, name):
+        """Fuzz field value if it's selected """
+        if coin():
+            field.value = getattr(fuzz, name)(field.value)
+
+    tmp = copy.deepcopy(structure)
+    walk(tmp, iter_fuzz)
+
+    return tmp
+
+
+def image_size():
+    """Generate a random file size aligned to a random correct cluster size"""
+    cluster_bits = random.randrange(9, 21)
+    cluster_size = 1 << cluster_bits
+    file_size = random.randrange(5*cluster_size, MAX_IMAGE_SIZE + 1,
+                                 cluster_size)
+    return [cluster_bits, file_size]
+
+
+def header(cluster_bits, img_size):
+    """Generate a random valid header"""
+    meta_header = [
+        ['>4s', 0, 'magic'],
+        ['>I', 4, 'version'],
+        ['>Q', 8, 'backing_file_offset'],
+        ['>I', 16, 'backing_file_size'],
+        ['>I', 20, 'cluster_bits'],
+        ['>Q', 24, 'size'],
+        ['>I', 32, 'crypt_method'],
+        ['>I', 36, 'l1_size'],
+        ['>Q', 40, 'l1_table_offset'],
+        ['>Q', 48, 'refcount_table_offset'],
+        ['>I', 56, 'refcount_table_clusters'],
+        ['>I', 60, 'nb_snapshots'],
+        ['>Q', 64, 'snapshots_offset'],
+        ['>Q', 72, 'incompatible_features'],
+        ['>Q', 80, 'compatible_features'],
+        ['>Q', 88, 'autoclear_features'],
+        ['>I', 96, 'refcount_order'],
+        ['>I', 100, 'header_length']
+    ]
+    values = repeat(0)
+    v_header = dict((f[2], Field(f[0], f[1], values.next()))
+                    for f in meta_header)
+
+    # Setup of valid values
+    v_header['magic'].value = "QFI\xfb"
+    v_header['version'].value = random.randint(2, 3)
+    v_header['cluster_bits'].value = cluster_bits
+    v_header['size'].value = img_size
+    if v_header['version'].value == 2:
+        v_header['header_length'].value = 72
+    else:
+        v_header['incompatible_features'].value = random.getrandbits(2)
+        v_header['compatible_features'].value = random.getrandbits(1)
+        v_header['header_length'].value = 104
+
+    # From the e-mail thread for [PATCH] docs: Define refcount_bits value:
+    # Only refcount_order = 4 is supported by QEMU at the moment
+    v_header['refcount_order'].value = 4
+
+    return v_header
+
+
+def header_extensions(header):
+    """Generate a random valid header"""
+    output = []
+    start_offset = struct.calcsize(header['header_length'].fmt) + \
+                   header['header_length'].offset
+
+    # Backing file format
+    if not header['backing_file_offset'].value == 0:
+        # Till real backup image is not supported, a random valid fmt is set
+        backing_img_fmt = random.choice(['raw', 'qcow', 'qcow2', 'qed',
+                                         'cow', 'vdi', 'vmdk', 'vpc',
+                                         'vhdx', 'bochs', 'cloop',
+                                         'dmg', 'parallels'])
+        bi_fmt_len = len(backing_img_fmt)
+        bf_fmt_padded = '>' + str(bi_fmt_len) + 's' + \
+                        str(7 - (bi_fmt_len - 1) % 8) + 'x'
+        bf_ext = dict([('ext_magic', Field('>I', start_offset, 0xE2792ACA)),
+                       ('ext_length', Field('>I', start_offset + UINT32_S,
+                                            bi_fmt_len)),
+                       ('bf_data', Field(bf_fmt_padded, start_offset +
+                                         UINT32_S*2, backing_img_fmt))])
+        output.append(bf_ext)
+        start_offset = bf_ext['bf_data'].offset + \
+                       struct.calcsize(bf_ext['bf_data'].fmt)
+
+    feature_tables = []
+    # Current offset + magic and length fields of the feature table extension
+    inner_offset = start_offset + 2*UINT32_S
+
+    # Each tuple is (bit value in the corresponding header field, feature type,
+    # number of the bit in the header field, feature name)
+    feature_list = [
+        (header['incompatible_features'].value & 1, 0, 1, 'dirty bit'),
+        (header['incompatible_features'].value & 2, 0, 2, 'corrupt bit'),
+        (header['compatible_features'].value & 1, 1, 1, 'lazy refcounts bit')
+    ]
+    for item in feature_list:
+        if not item[0] == 0:
+            name_len = len(item[3])
+            name_padded_fmt = '>' + str(name_len) + 's' + \
+                              str(46 - name_len) + 'x'
+            feature = dict([
+                ('feat_type', Field('B', inner_offset, item[1])),
+                ('feat_bit_number', Field('B', inner_offset + 1, item[2])),
+                ('feat_name', Field(name_padded_fmt, inner_offset + 2,
+                                    item[3])),
+            ])
+            feature_tables.append(feature)
+            inner_offset = feature['feat_name'].offset + \
+                           struct.calcsize(feature['feat_name'].fmt)
+
+    if not len(feature_tables) == 0:
+        # No padding for the extension is necessary, because
+        # the extension length = 8 + 48*N is multiple of 8
+        fnt_ext = dict([('ext_magic', Field('>I', start_offset, 0x6803f857)),
+                        ('ext_length', Field('>I', start_offset + UINT32_S,
+                                             len(feature_tables)*48)),
+                        ('fnt_data', feature_tables)])
+        output.append(fnt_ext)
+        start_offset = inner_offset
+
+    end_ext = dict([
+        ('ext_magic', Field('>I', start_offset, 0)),
+        ('ext_length', Field('>I', start_offset + UINT32_S, 0))
+    ])
+    output.append(end_ext)
+
+    return output
+
+
+def create_image(test_img_path):
+    """Write a fuzzed image to the specified file"""
+    image_file = open(test_img_path, 'w')
+    cluster_bits, v_image_size = image_size()
+    # Create an empty image
+    # (sparse if FS supports it or preallocated otherwise)
+    image_file.seek(v_image_size - 1)
+    image_file.write("\0")
+    # Image structure defined as references between image blocks
+    # List is used because of its order.
+    meta_img = [
+        lambda x: header(cluster_bits, v_image_size),
+        lambda x: header_extensions(x[0])
+    ]
+
+    # Valid image
+    img = []
+
+    # On every step all prerequisites for the current image block
+    # are already evaluated, e.g. the header (img[0]) is generated and
+    # available for header_extensions() function that will produce img[1]
+    for ref in meta_img:
+        img.append(ref(img))
+
+    # Fuzzed image
+    img_fuzzed = [fuzz_struct(substruct) for substruct in img]
+
+    # Writing to file
+    def write_field(field, _):
+        """Write a field to file"""
+        image_file.seek(field.offset)
+        image_file.write(struct.pack(field.fmt, field.value))
+
+    walk(img_fuzzed, write_field)
+    image_file.close()
-- 
1.9.3

[Qemu-devel] [PATCH 5/5] package: Public API for image-fuzzer/runner/runner.py

[Maria Kustova]

__init__.py provides the public API required by the test runner

Signed-off-by: Maria Kustova <maria.k@catit.be>
---
 tests/image-fuzzer/qcow2/__init__.py | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 tests/image-fuzzer/qcow2/__init__.py

diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-fuzzer/qcow2/__init__.py
new file mode 100644
index 0000000..e2ebe19
--- /dev/null
+++ b/tests/image-fuzzer/qcow2/__init__.py
@@ -0,0 +1 @@
+from layout import create_image
-- 
1.9.3

Re: [Qemu-devel] [PATCH 1/5] docs: Specification for the image fuzzer

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 9165 bytes --]

On 06/30/2014 05:48 AM, Maria Kustova wrote:
> 'Overall fuzzer requirements' chapter contains the current product vision and
> features done and to be done. This chapter is still in progress.
> 
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
>  tests/image-fuzzer/docs/image-fuzzer.txt | 176 +++++++++++++++++++++++++++++++
>  1 file changed, 176 insertions(+)
>  create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt
> 
> diff --git a/tests/image-fuzzer/docs/image-fuzzer.txt b/tests/image-fuzzer/docs/image-fuzzer.txt
> new file mode 100644
> index 0000000..a9ed4b6
> --- /dev/null
> +++ b/tests/image-fuzzer/docs/image-fuzzer.txt
> @@ -0,0 +1,176 @@
> +Image fuzzer
> +============

You're no worse than the bulk of the files in this directory for
omitting copyright and license information, but it would be nice to buck
the trend and provide it explicitly.

> +
> +Description
> +-----------
> +
> +The goal of the image fuzzer is to catch crashes of qemu-io/qemu-img providing
> +to them randomly corrupted images.

s/providing to/by providing/

> +Test images are generated from scratch and have valid inner structure with some
> +elements, e.g. L1/L2 tables, having random invalid values.
> +
> +
> +Test runner
> +-----------
> +
> +The test runner generates test images, executes tests utilizing generated
> +images, indicates their results and collect all test related artifacts (logs,

s/collect/collects/

> +core dumps, test images).
> +The test means one start of a system under test (SUT), e.g. qemu-io, with
> +specified arguments and one test image.
> +By default, the test runner generates new tests and executes them till

s/till/until/

> +keyboard interruption. But if a test seed is specified via '-s' runner

s/via/via the/

> +parameter, then only one test with this seed will be executed, after its finish
> +the runner will exit.
> +
> +The runner uses an external image fuzzer to generate test images. An image
> +generator should be specified as a mandatory parameter of the test runner.
> +Details about interactions between the runner and fuzzers see "Module
> +interfaces".
> +
> +The runner activates generation of core dumps during test executions, but it
> +assumes that core dumps will be generated in the current working directory.
> +For comprehensive test results, please, set up your test environment
> +properly.
> +
> +Path to a SUT can be specified via environment variables (for now only
> +qemu-img) or via '--binary' parameter of the test runner. For details about
> +environment variables see qemu-iotests/check.

If both are specified, which wins?  (Command line should always trump
environment)

> +
> +
> +Qcow2 image generator
> +---------------------
> +
> +The 'qcow2' generator is a Python package providing 'create_image' method as
> +a single public API. See details in 'Test runner/image fuzzer' in 'Module
> +interfaces'.
> +
> +Qcow2 contains two submodules: fuzz.py and layout.py.
> +
> +'fuzz.py' contains all fuzzing functions one per image field. It's assumed that

s/functions/functions,/

> +after code analysis every field will have own constraints for its value. For
> +now only universal potentially dangerous values are used, e.g. type limits for
> +integers or unsafe symbols as '%s' for strings. For bitmasks random amount of
> +bits are set to ones. All fuzzed values are checked on non-equality to the
> +current valid value of the field. In case of equality the value will be
> +regenerated.
> +
> +'layout.py' creates a random valid image, fuzzes a random subset of the image
> +fields by 'fuzz.py' module and writes a fuzzed image to the file specified.
> +
> +For now only header fields and header extensions are generated, the remaining
> +file is filled with zeros.
> +
> +
> +Module interfaces
> +-----------------
> +
> +* Test runner/image fuzzer
> +
> +The runner calls an image generator specifying path to a test image file.

s/path/the path/

> +An image generator is expected to provide 'create_image(test_img_path)' method

s/provide/provide a/

> +that creates a test image and writes it to the specified file. The file should
> +be created if it doesn't exist or overwritten otherwise.
> +Random seed is set by the runner at every test execution for the regression
> +purpose, so an image generator is not recommended to modify it internally.
> +
> +* Test runner/SUT
> +
> +A full test command is composed from the SUT, its arguments specified
> +via '-c' runner parameter and the name of generated image. To indicate where
> +a test image name should be placed TEST_IMG placeholder can be used.
> +For example, for the next test command
> +
> + qemu-img convert -O vmdk fuzzed_image test.vmdk
> +
> +a call of the image fuzzer will be following:
> +
> + ./runner.py -b qemu-img -c 'convert -O vmdk TEST_IMG test.vmdk' work_dir qcow2
> +
> +
> +Overall fuzzer requirements
> +===========================
> +
> +Input data:
> +----------
> +
> + - image template (generator)
> + - work directory
> + - test run duration (optional)
> + - action vector (optional)
> + - seed (optional)
> + - SUT and its arguments (optional)
> +
> +
> +Fuzzer requirements:
> +-------------------
> +
> +1.  Should be able to inject random data
> +2.  Should be able to select a random value from the manually pregenerated
> +    vector (boundary values, e.g. max/min cluster size)
> +3.  Image template should describe a general structure invariant for all
> +    test images (image format description)
> +4.  Image template should be autonomous and other fuzzer parts should not
> +    relate on it

s/relate/rely/

> +5.  Image template should contain reference rules (not only block+size
> +    description)
> +6.  Should generate the test image with the correct structure based on an image
> +    template
> +7.  Should accept a seed as an argument (for regression purpose)
> +8.  Should generate a seed if it is not specified as an input parameter.
> +9.  The same seed should generate the same image, if no or the same action
> +    vector is specified

s/no or the same action vector/the same action vector (including the
case of no action)/

> +10. Should accept a vector of actions as an argument (for test reproducing and
> +    for test case specification, e.g. group of tests for header structure,
> +    group of test for snapshots, etc)
> +11. Action vector should be randomly generated from the pool of available
> +    actions, if it is not specified as an input parameter
> +12. Pool of actions should be defined automatically based on an image template
> +13. Should accept a SUT and its call parameters as an argument or select them
> +    randomly otherwise. As far as it's expected to be rarely changed, the list
> +    of all possible test commands can be available in the test runner
> +    internally.
> +14. Should accept a test run duration as an argument. Tests should be executed
> +    during a minimum period from a test run duration and time while fuzzer can
> +    generate different test images
> +15. Should support an external cancellation of a test run
> +16. Seed and action vector should be logged (for regression purpose)
> +17. All files related to test result should be collected: a test image,
> +    SUT logs, fuzzer logs and crash dumps
> +18. Should be compatible with python version 2.4-2.7
> +19. Usage of external libraries should be limited as much as possible.
> +
> +
> +Image formats:
> +-------------
> +
> +Main target image format is qcow2, but support of image templates should
> +provide an ability to add any other image format.
> +
> +
> +Effectiveness:
> +-------------
> +
> +Fuzzer can be controlled via template, seed and action vector;
> +it makes the fuzzer itself invariant to an image format and test logic.
> +It should be able to perform rather complex and precise tests, that can be
> +specified via action vector. Otherwise, knowledge about an image structure
> +allows the fuzzer to generate the pool of all available areas can be fuzzed
> +and randomly select some of them and so compose its own action vector.
> +Also complexity of a template defines complexity of the fuzzer, so its
> +functionality can be varied from simple model-independent fuzzing to smart
> +model-based one.
> +
> +
> +Glossary:
> +--------
> +
> +Action vector is a sequence of structure elements retrieved from an image
> +format, each of them will be fuzzed for the test image. It's a subset of
> +elements of the action pool. Example: header, refcount table, etc.
> +Action pool is all available elements of an image structure that generated
> +automatically from an image template.
> +Image template is a formal description of an image structure and relations
> +between image blocks.
> +Test image is an output image of the fuzzer defined by the current seed and
> +action vector.
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]