Re: [Qemu-devel] [PATCH 1/5] docs: Specification for the image fuzzer

[Eric Blake <eblake@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 9165 bytes --]

On 06/30/2014 05:48 AM, Maria Kustova wrote:
> 'Overall fuzzer requirements' chapter contains the current product vision and
> features done and to be done. This chapter is still in progress.
> 
> Signed-off-by: Maria Kustova <maria.k@catit.be>
> ---
>  tests/image-fuzzer/docs/image-fuzzer.txt | 176 +++++++++++++++++++++++++++++++
>  1 file changed, 176 insertions(+)
>  create mode 100644 tests/image-fuzzer/docs/image-fuzzer.txt
> 
> diff --git a/tests/image-fuzzer/docs/image-fuzzer.txt b/tests/image-fuzzer/docs/image-fuzzer.txt
> new file mode 100644
> index 0000000..a9ed4b6
> --- /dev/null
> +++ b/tests/image-fuzzer/docs/image-fuzzer.txt
> @@ -0,0 +1,176 @@
> +Image fuzzer
> +============

You're no worse than the bulk of the files in this directory for
omitting copyright and license information, but it would be nice to buck
the trend and provide it explicitly.

> +
> +Description
> +-----------
> +
> +The goal of the image fuzzer is to catch crashes of qemu-io/qemu-img providing
> +to them randomly corrupted images.

s/providing to/by providing/

> +Test images are generated from scratch and have valid inner structure with some
> +elements, e.g. L1/L2 tables, having random invalid values.
> +
> +
> +Test runner
> +-----------
> +
> +The test runner generates test images, executes tests utilizing generated
> +images, indicates their results and collect all test related artifacts (logs,

s/collect/collects/

> +core dumps, test images).
> +The test means one start of a system under test (SUT), e.g. qemu-io, with
> +specified arguments and one test image.
> +By default, the test runner generates new tests and executes them till

s/till/until/

> +keyboard interruption. But if a test seed is specified via '-s' runner

s/via/via the/

> +parameter, then only one test with this seed will be executed, after its finish
> +the runner will exit.
> +
> +The runner uses an external image fuzzer to generate test images. An image
> +generator should be specified as a mandatory parameter of the test runner.
> +Details about interactions between the runner and fuzzers see "Module
> +interfaces".
> +
> +The runner activates generation of core dumps during test executions, but it
> +assumes that core dumps will be generated in the current working directory.
> +For comprehensive test results, please, set up your test environment
> +properly.
> +
> +Path to a SUT can be specified via environment variables (for now only
> +qemu-img) or via '--binary' parameter of the test runner. For details about
> +environment variables see qemu-iotests/check.

If both are specified, which wins?  (Command line should always trump
environment)

> +
> +
> +Qcow2 image generator
> +---------------------
> +
> +The 'qcow2' generator is a Python package providing 'create_image' method as
> +a single public API. See details in 'Test runner/image fuzzer' in 'Module
> +interfaces'.
> +
> +Qcow2 contains two submodules: fuzz.py and layout.py.
> +
> +'fuzz.py' contains all fuzzing functions one per image field. It's assumed that

s/functions/functions,/

> +after code analysis every field will have own constraints for its value. For
> +now only universal potentially dangerous values are used, e.g. type limits for
> +integers or unsafe symbols as '%s' for strings. For bitmasks random amount of
> +bits are set to ones. All fuzzed values are checked on non-equality to the
> +current valid value of the field. In case of equality the value will be
> +regenerated.
> +
> +'layout.py' creates a random valid image, fuzzes a random subset of the image
> +fields by 'fuzz.py' module and writes a fuzzed image to the file specified.
> +
> +For now only header fields and header extensions are generated, the remaining
> +file is filled with zeros.
> +
> +
> +Module interfaces
> +-----------------
> +
> +* Test runner/image fuzzer
> +
> +The runner calls an image generator specifying path to a test image file.

s/path/the path/

> +An image generator is expected to provide 'create_image(test_img_path)' method

s/provide/provide a/

> +that creates a test image and writes it to the specified file. The file should
> +be created if it doesn't exist or overwritten otherwise.
> +Random seed is set by the runner at every test execution for the regression
> +purpose, so an image generator is not recommended to modify it internally.
> +
> +* Test runner/SUT
> +
> +A full test command is composed from the SUT, its arguments specified
> +via '-c' runner parameter and the name of generated image. To indicate where
> +a test image name should be placed TEST_IMG placeholder can be used.
> +For example, for the next test command
> +
> + qemu-img convert -O vmdk fuzzed_image test.vmdk
> +
> +a call of the image fuzzer will be following:
> +
> + ./runner.py -b qemu-img -c 'convert -O vmdk TEST_IMG test.vmdk' work_dir qcow2
> +
> +
> +Overall fuzzer requirements
> +===========================
> +
> +Input data:
> +----------
> +
> + - image template (generator)
> + - work directory
> + - test run duration (optional)
> + - action vector (optional)
> + - seed (optional)
> + - SUT and its arguments (optional)
> +
> +
> +Fuzzer requirements:
> +-------------------
> +
> +1.  Should be able to inject random data
> +2.  Should be able to select a random value from the manually pregenerated
> +    vector (boundary values, e.g. max/min cluster size)
> +3.  Image template should describe a general structure invariant for all
> +    test images (image format description)
> +4.  Image template should be autonomous and other fuzzer parts should not
> +    relate on it

s/relate/rely/

> +5.  Image template should contain reference rules (not only block+size
> +    description)
> +6.  Should generate the test image with the correct structure based on an image
> +    template
> +7.  Should accept a seed as an argument (for regression purpose)
> +8.  Should generate a seed if it is not specified as an input parameter.
> +9.  The same seed should generate the same image, if no or the same action
> +    vector is specified

s/no or the same action vector/the same action vector (including the
case of no action)/

> +10. Should accept a vector of actions as an argument (for test reproducing and
> +    for test case specification, e.g. group of tests for header structure,
> +    group of test for snapshots, etc)
> +11. Action vector should be randomly generated from the pool of available
> +    actions, if it is not specified as an input parameter
> +12. Pool of actions should be defined automatically based on an image template
> +13. Should accept a SUT and its call parameters as an argument or select them
> +    randomly otherwise. As far as it's expected to be rarely changed, the list
> +    of all possible test commands can be available in the test runner
> +    internally.
> +14. Should accept a test run duration as an argument. Tests should be executed
> +    during a minimum period from a test run duration and time while fuzzer can
> +    generate different test images
> +15. Should support an external cancellation of a test run
> +16. Seed and action vector should be logged (for regression purpose)
> +17. All files related to test result should be collected: a test image,
> +    SUT logs, fuzzer logs and crash dumps
> +18. Should be compatible with python version 2.4-2.7
> +19. Usage of external libraries should be limited as much as possible.
> +
> +
> +Image formats:
> +-------------
> +
> +Main target image format is qcow2, but support of image templates should
> +provide an ability to add any other image format.
> +
> +
> +Effectiveness:
> +-------------
> +
> +Fuzzer can be controlled via template, seed and action vector;
> +it makes the fuzzer itself invariant to an image format and test logic.
> +It should be able to perform rather complex and precise tests, that can be
> +specified via action vector. Otherwise, knowledge about an image structure
> +allows the fuzzer to generate the pool of all available areas can be fuzzed
> +and randomly select some of them and so compose its own action vector.
> +Also complexity of a template defines complexity of the fuzzer, so its
> +functionality can be varied from simple model-independent fuzzing to smart
> +model-based one.
> +
> +
> +Glossary:
> +--------
> +
> +Action vector is a sequence of structure elements retrieved from an image
> +format, each of them will be fuzzed for the test image. It's a subset of
> +elements of the action pool. Example: header, refcount table, etc.
> +Action pool is all available elements of an image structure that generated
> +automatically from an image template.
> +Image template is a formal description of an image structure and relations
> +between image blocks.
> +Test image is an output image of the fuzzer defined by the current seed and
> +action vector.
> 

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]