From: Jeff Cody <jcody@redhat.com>
To: Stefan Hajnoczi <stefanha@redhat.com>
Cc: qemu-devel@nongnu.org, Kevin Wolf <kwolf@redhat.com>,
Ishani Chugh <chugh.ishani@research.iiit.ac.in>,
qemu-block@nongnu.org
Subject: Re: [Qemu-devel] [Qemu-block] [PATCH] qemu-iotests: add a "how to" to ./README
Date: Wed, 26 Jul 2017 14:42:17 -0400 [thread overview]
Message-ID: <20170726184217.GE6240@localhost.localdomain> (raw)
In-Reply-To: <20170721093416.17816-1-stefanha@redhat.com>
On Fri, Jul 21, 2017 at 10:34:16AM +0100, Stefan Hajnoczi wrote:
> There is not much getting started documentation for qemu-iotests. This
> patch explains how to create a new test and covers the overall testing
> approach.
>
> Cc: Ishani Chugh <chugh.ishani@research.iiit.ac.in>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> tests/qemu-iotests/README | 83 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 83 insertions(+)
>
> diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README
> index 6079b40..8259b9f 100644
> --- a/tests/qemu-iotests/README
> +++ b/tests/qemu-iotests/README
> @@ -14,8 +14,91 @@ Just run ./check to run all tests for the raw image format, or ./check
> -qcow2 to test the qcow2 image format. The output of ./check -h explains
> additional options to test further image formats or I/O methods.
>
> +* Testing approach
> +
> +Each test is an executable file (usually a bash script) that is run by the
> +./check test harness. Standard out and standard error are captured to an
> +output file. If the output file differs from the "golden master" output file
> +for the test then it fails.
> +
> +Tests are simply a sequence of commands that produce output and the test itself
> +does not judge whether it passed or failed. If you find yourself writing
> +checks to determine success or failure then you should rethink the test and
> +rely on output diffing instead.
> +
> +** Filtering volatile output
> +
> +When output contains absolute file paths, timestamps, process IDs, hostnames,
> +or other volatile strings, the diff against golden master output will fail.
> +Such output must be filtered to replace volatile strings with fixed
> +placeholders.
> +
> +For example, the path to the temporary working directory changes between test
> +runs so it must be filtered:
> +
> + sed -e "s#$TEST_DIR/#TEST_DIR/#g"
> +
> +Commonly needed filters are available in ./common.filter.
> +
> +** Python tests
> +
> +Most tests are implemented in bash but it is difficult to interact with the QMP
> +monitor. A Python module called 'iotests' is available for tests that require
> +JSON and interacting with QEMU.
> +
There are some that prefer the bash tests still, and we do have a 'standard'
way to do so... so perhaps add as well:
** Bash tests
If you wish to create a test in Bash that interacts with the QMP monitor,
'common.qemu' provides functions for interacting with multiple QEMU
processes.
> +* How to create a test
> +
> +1. Choose an unused test number
> +
> +Tests are identified by a unique number. Look for the highest test case number
> +by looking at the test files. Then search the qemu-devel@nongnu.org mailing
> +list to check if anyone has already sent patches using the next available
> +number. You may need to increment the number a few times to reach an unused
> +number.
> +
> +2. Create the test file
> +
> +Copy an existing test (one that most closely resembles what you wish to test)
> +to the new test number:
> +
> + cp 001 <test-number>
> +
> +3. Assign groups to the test
> +
> +Add your test to the ./group file. This file is the index of tests and assigns
> +them to functional groups like "rw" for read-write tests. Most tests belong to
> +the "rw" and "auto" groups. "auto" means the test runs when ./check is invoked
> +without a -g argument.
> +
> +Consider adding your test to the "quick" group if it executes quickly (<1s).
> +This group is run by "make check-block" and is often included as part of build
> +tests in continuous integration systems.
> +
> +4. Write the test
> +
> +Edit the test script. Look at existing tests for examples.
> +
> +5. Generate the golden master file
> +
> +Run your test with "./check <test-number>". You may need to pass additional
> +options to use an image format or protocol.
> +
> +The test will fail because there is no golden master yet. Inspect the output
> +that your test generated with "cat <test-number>.out.bad".
> +
> +Verify that the output is as expected and contains no volatile strings like
> +timestamps. You may need to add filters to your test to remove volatile
> +strings.
> +
> +Once you are happy with the test output it can be used as the golden master
> +with "mv <test-number>.out.bad <test-number>.out". Rerun the test to verify
> +that it passes.
> +
> +Congratulations, you've created a new test!
> +
> * Feedback and patches
>
> Please send improvements to the test suite, general feedback or just
> reports of failing tests cases to qemu-devel@nongnu.org with a CC:
> to qemu-block@nongnu.org.
> +
> --
> 2.9.4
>
>
prev parent reply other threads:[~2017-07-26 18:42 UTC|newest]
Thread overview: 19+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-07-21 9:34 [Qemu-devel] [PATCH] qemu-iotests: add a "how to" to ./README Stefan Hajnoczi
2017-07-21 12:16 ` Eric Blake
2017-07-21 15:51 ` Stefan Hajnoczi
2017-07-23 14:18 ` Philippe Mathieu-Daudé
2017-07-24 10:06 ` Stefan Hajnoczi
2017-07-24 21:50 ` [Qemu-devel] [Qemu-block] " John Snow
2017-07-22 9:03 ` Manos Pitsidianakis
2017-07-24 9:11 ` [Qemu-devel] " Kevin Wolf
2017-07-25 15:22 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-07-24 10:20 ` [Qemu-devel] " Peter Maydell
2017-07-25 15:20 ` [Qemu-devel] [Qemu-block] " Stefan Hajnoczi
2017-07-25 15:34 ` Peter Maydell
2017-07-26 11:33 ` Stefan Hajnoczi
2017-07-26 11:50 ` Peter Maydell
2017-07-24 14:28 ` [Qemu-devel] " Eric Blake
2017-07-24 14:33 ` Kevin Wolf
2017-07-24 14:34 ` Peter Maydell
2017-07-24 14:41 ` Eric Blake
2017-07-26 18:42 ` Jeff Cody [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170726184217.GE6240@localhost.localdomain \
--to=jcody@redhat.com \
--cc=chugh.ishani@research.iiit.ac.in \
--cc=kwolf@redhat.com \
--cc=qemu-block@nongnu.org \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.