[Qemu-devel] [PATCH] qemu-iotests: Make debugging python tests easier

[Qemu-devel] [PATCH] qemu-iotests: Make debugging python tests easier

[Fam Zheng]

Adding "-d" option. The output goes to "tee" so it appears in your
console. Also, raise the verbosity of unnitest runner.

When testing a topic branch, it's possible that a bug introduced by a
code change makes the python test case hang, with debug output, it is
much easier to locate the problem.

This can also be helpful if you want to watch the progress of a python
test, it offers you a way to sense the speed of each test case method
you're writing.

Signed-off-by: Fam Zheng <famz@redhat.com>
---
 tests/qemu-iotests/check      | 12 +++++++++---
 tests/qemu-iotests/common     |  6 ++++++
 tests/qemu-iotests/iotests.py | 11 +++++++++--
 3 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index baeae80..1fa6319 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -296,9 +296,15 @@ do
             run_command="./$seq"
         fi
         export OUTPUT_DIR=$PWD
-        (cd "$source_iotests";
-        MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(($RANDOM % 255 + 1))} \
-                $run_command >$tmp.out 2>&1)
+        if $debug; then
+            (cd "$source_iotests";
+            MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(($RANDOM % 255 + 1))} \
+                    $run_command -d 2>&1 | tee $tmp.out)
+        else
+            (cd "$source_iotests";
+            MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(($RANDOM % 255 + 1))} \
+                    $run_command >$tmp.out 2>&1)
+        fi
         sts=$?
         $timestamp && _timestamp
         stop=`_wallclock`
diff --git a/tests/qemu-iotests/common b/tests/qemu-iotests/common
index 1e556bb..1030aaf 100644
--- a/tests/qemu-iotests/common
+++ b/tests/qemu-iotests/common
@@ -32,6 +32,7 @@ check=${check-true}
 
 diff="diff -u"
 verbose=false
+debug=false
 group=false
 xgroup=false
 imgopts=false
@@ -132,6 +133,7 @@ s/ .*//p
 
 common options
     -v                  verbose
+    -d                  debug
 
 check options
     -raw                test raw (default)
@@ -322,6 +324,10 @@ testlist options
             verbose=true
             xpand=false
             ;;
+        -d)
+            debug=true
+            xpand=false
+            ;;
         -x)        # -x group ... exclude from group file
             xgroup=true
             xpand=false
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index e93e623..cc80d5e 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -338,6 +338,8 @@ def notrun(reason):
 def main(supported_fmts=[], supported_oses=['linux']):
     '''Run tests'''
 
+    debug = '-d' in sys.argv
+    verbosity = 1
     if supported_fmts and (imgfmt not in supported_fmts):
         notrun('not suitable for this image format: %s' % imgfmt)
 
@@ -347,10 +349,15 @@ def main(supported_fmts=[], supported_oses=['linux']):
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
     import StringIO
-    output = StringIO.StringIO()
+    if debug:
+        output = sys.stdout
+        verbosity = 2
+        sys.argv.remove('-d')
+    else:
+        output = StringIO.StringIO()
 
     class MyTestRunner(unittest.TextTestRunner):
-        def __init__(self, stream=output, descriptions=True, verbosity=1):
+        def __init__(self, stream=output, descriptions=True, verbosity=verbosity):
             unittest.TextTestRunner.__init__(self, stream, descriptions, verbosity)
 
     # unittest.main() will use sys.exit() so expect a SystemExit exception
-- 
2.4.0

Re: [Qemu-devel] [PATCH] qemu-iotests: Make debugging python tests easier

[John Snow]

On 05/15/2015 01:26 AM, Fam Zheng wrote:
> Adding "-d" option. The output goes to "tee" so it appears in your
> console. Also, raise the verbosity of unnitest runner.
> 
> When testing a topic branch, it's possible that a bug introduced by a
> code change makes the python test case hang, with debug output, it is
> much easier to locate the problem.
> 
> This can also be helpful if you want to watch the progress of a python
> test, it offers you a way to sense the speed of each test case method
> you're writing.
> 

Awesome idea!

Unfortunately, it explodes when I run ./check -v -d -qcow2 124:

+Traceback (most recent call last):
+  File "124", line 363, in <module>
+    iotests.main(supported_fmts=['qcow2'])
+  File "/home/bos/jhuston/src/qemu/tests/qemu-iotests/iotests.py", line
367, in main
+    sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1
tests', output.getvalue()))
+AttributeError: 'file' object has no attribute 'getvalue'
Failures: 124
Failed 1 of 1 tests

--js

Re: [Qemu-devel] [PATCH] qemu-iotests: Make debugging python tests easier

[Fam Zheng]

On Fri, 05/15 13:48, John Snow wrote:
> 
> 
> On 05/15/2015 01:26 AM, Fam Zheng wrote:
> > Adding "-d" option. The output goes to "tee" so it appears in your
> > console. Also, raise the verbosity of unnitest runner.
> > 
> > When testing a topic branch, it's possible that a bug introduced by a
> > code change makes the python test case hang, with debug output, it is
> > much easier to locate the problem.
> > 
> > This can also be helpful if you want to watch the progress of a python
> > test, it offers you a way to sense the speed of each test case method
> > you're writing.
> > 
> 
> Awesome idea!
> 
> Unfortunately, it explodes when I run ./check -v -d -qcow2 124:
> 
> +Traceback (most recent call last):
> +  File "124", line 363, in <module>
> +    iotests.main(supported_fmts=['qcow2'])
> +  File "/home/bos/jhuston/src/qemu/tests/qemu-iotests/iotests.py", line
> 367, in main
> +    sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1
> tests', output.getvalue()))
> +AttributeError: 'file' object has no attribute 'getvalue'
> Failures: 124
> Failed 1 of 1 tests
> 
> --js

I guess we can skip this line if debug is True. (However, I should have noted
in the commit message, by specifying "-d", the purpose of the execution is
solely *debugging the test itself*, and because I didn't find a easy way to get
*both* the verbose debug output and an output that would match $seq.out, this
patch always makes a "-d" execution fail.)

Will send another version with an additional "if debug".

Thanks,

Fam

Re: [Qemu-devel] [PATCH] qemu-iotests: Make debugging python tests easier

[John Snow]

On 05/17/2015 09:32 PM, Fam Zheng wrote:
> On Fri, 05/15 13:48, John Snow wrote:
>>
>>
>> On 05/15/2015 01:26 AM, Fam Zheng wrote:
>>> Adding "-d" option. The output goes to "tee" so it appears in your
>>> console. Also, raise the verbosity of unnitest runner.
>>>
>>> When testing a topic branch, it's possible that a bug introduced by a
>>> code change makes the python test case hang, with debug output, it is
>>> much easier to locate the problem.
>>>
>>> This can also be helpful if you want to watch the progress of a python
>>> test, it offers you a way to sense the speed of each test case method
>>> you're writing.
>>>
>>
>> Awesome idea!
>>
>> Unfortunately, it explodes when I run ./check -v -d -qcow2 124:
>>
>> +Traceback (most recent call last):
>> +  File "124", line 363, in <module>
>> +    iotests.main(supported_fmts=['qcow2'])
>> +  File "/home/bos/jhuston/src/qemu/tests/qemu-iotests/iotests.py", line
>> 367, in main
>> +    sys.stderr.write(re.sub(r'Ran (\d+) tests? in [\d.]+s', r'Ran \1
>> tests', output.getvalue()))
>> +AttributeError: 'file' object has no attribute 'getvalue'
>> Failures: 124
>> Failed 1 of 1 tests
>>
>> --js
> 
> I guess we can skip this line if debug is True. (However, I should have noted
> in the commit message, by specifying "-d", the purpose of the execution is
> solely *debugging the test itself*, and because I didn't find a easy way to get
> *both* the verbose debug output and an output that would match $seq.out, this
> patch always makes a "-d" execution fail.)
> 
> Will send another version with an additional "if debug".
> 
> Thanks,
> 
> Fam
> 

Ah, I misunderstood: I thought you wanted to just increase the
verbosity, and the failing test warnings gave me a bit of pause.

Best not to have things that cause a python stack trace laying around on
purpose.

--js