From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53513) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1da1td-0002rL-Lk for qemu-devel@nongnu.org; Tue, 25 Jul 2017 11:37:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1da1tZ-00014J-Nf for qemu-devel@nongnu.org; Tue, 25 Jul 2017 11:37:17 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50274) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1da1tZ-0000zY-Ew for qemu-devel@nongnu.org; Tue, 25 Jul 2017 11:37:13 -0400 From: Stefan Hajnoczi Date: Tue, 25 Jul 2017 16:36:45 +0100 Message-Id: <20170725153645.26412-1-stefanha@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: [Qemu-devel] [PATCH v2] qemu-iotests: add a "how to" to ./README List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: Kevin Wolf , John Snow , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Peter Maydell , qemu-block@nongnu.org, eblake@redhat.com, Stefan Hajnoczi , Ishani Chugh There is not much getting started documentation for qemu-iotests. This patch explains how to create a new test and covers the overall testing approach. Cc: Ishani Chugh Reviewed-by: Eric Blake Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Stefan Hajnoczi --- v2: * Added missing "touch .out" step [Kevin] * Added reference to SubmitAPatch wiki page [Eric & Philippe] --- tests/qemu-iotests/README | 94 +++++++++++++++++++++++++++++++++++++++++= ++++++ 1 file changed, 94 insertions(+) diff --git a/tests/qemu-iotests/README b/tests/qemu-iotests/README index 6079b401ae..245f509b14 100644 --- a/tests/qemu-iotests/README +++ b/tests/qemu-iotests/README @@ -14,8 +14,102 @@ Just run ./check to run all tests for the raw image f= ormat, or ./check -qcow2 to test the qcow2 image format. The output of ./check -h explain= s additional options to test further image formats or I/O methods. =20 +* Testing approach + +Each test is an executable file (usually a bash script) that is run by t= he +./check test harness. Standard out and standard error are captured to a= n +output file. If the output file differs from the "golden master" output= file +for the test then it fails. + +Tests are simply a sequence of commands that produce output and the test= itself +does not judge whether it passed or failed. If you find yourself writin= g +checks to determine success or failure then you should rethink the test = and +rely on output diffing instead. + +** Filtering volatile output + +When output contains absolute file paths, timestamps, process IDs, hostn= ames, +or other volatile strings, the diff against golden master output will fa= il. +Such output must be filtered to replace volatile strings with fixed +placeholders. + +For example, the path to the temporary working directory changes between= test +runs so it must be filtered: + + sed -e "s#$TEST_DIR/#TEST_DIR/#g" + +Commonly needed filters are available in ./common.filter. + +** Python tests + +Most tests are implemented in bash but it is difficult to interact with = the QMP +monitor. A Python module called 'iotests' is available for tests that r= equire +JSON and interacting with QEMU. + +* How to create a test + +1. Choose an unused test number + +Tests are identified by a unique number. Look for the highest test case= number +by looking at the test files. Then search the qemu-devel@nongnu.org mai= ling +list to check if anyone has already sent patches using the next availabl= e +number. You may need to increment the number a few times to reach an un= used +number. + +2. Create the test file + +Copy an existing test (one that most closely resembles what you wish to = test) +to the new test number: + + cp 001 + +3. Assign groups to the test + +Add your test to the ./group file. This file is the index of tests and = assigns +them to functional groups like "rw" for read-write tests. Most tests be= long to +the "rw" and "auto" groups. "auto" means the test runs when ./check is = invoked +without a -g argument. + +Consider adding your test to the "quick" group if it executes quickly (<= 1s). +This group is run by "make check-block" and is often included as part of= build +tests in continuous integration systems. + +4. Write the test + +Edit the test script. Look at existing tests for examples. + +5. Generate the golden master file + +Create an empty golden master file: + + touch .out + +Then run your test: + + ./check + +You may need additional command-line options to use an image format or +protocol like -qcow2. + +The test will fail because there is no golden master yet. Inspect the o= utput +that your test generated with "cat .out.bad". + +Verify that the output is as expected and contains no volatile strings l= ike +timestamps. You may need to add filters to your test to remove volatile +strings. + +Once you are happy with the test output it can be used as the golden mas= ter +with "mv .out.bad .out". Rerun the test to ve= rify +that it passes. + +Congratulations, you've created a new test! + +To contribute your test to qemu.git please follow the guidelines here: +http://wiki.qemu.org/Contribute/SubmitAPatch + * Feedback and patches =20 Please send improvements to the test suite, general feedback or just reports of failing tests cases to qemu-devel@nongnu.org with a CC: to qemu-block@nongnu.org. + --=20 2.13.3