KVM autotest tip of the week - Running qemu-kvm unittests

KVM autotest tip of the week - Running qemu-kvm unittests

[Lucas Meneghel Rodrigues]

Hi guys, due to some personal issues this week this tip came later than
I wanted, but nevertheless, here it is:

http://autotest.kernel.org/wiki/KVMAutotest/RunQemuUnittests

Ever wanted to make kvm autotest to execute the qemu-kvm unittest suite
automatically on your local qemu-kvm branch? So this is for you!

Cheers,

Lucas

Re: KVM autotest tip of the week - Running qemu-kvm unittests

[Dor Laor]

On 07/15/2011 04:12 PM, Lucas Meneghel Rodrigues wrote:
> Hi guys, due to some personal issues this week this tip came later than
> I wanted, but nevertheless, here it is:
>
> http://autotest.kernel.org/wiki/KVMAutotest/RunQemuUnittests
>
> Ever wanted to make kvm autotest to execute the qemu-kvm unittest suite
> automatically on your local qemu-kvm branch? So this is for you!
>
> Cheers,
>
> Lucas
>

Nice write up!

Why not put 'preserve_srcdir = True' by default?
In general, you can simplify the documentation by using the most 
frequent use case at the beginning. For example, you can just point to 
another link to get you all the yum dependencies instead of adding it 
here in this page where they do not belong.

The get_started.py script is cool :)

Note that in your output screendump is not functioning.

The setup was really neat and the tests managed to execute easily. I did 
got some failures and exception below:

"16:13:23 ERROR| Test failed: TestFail: Unit tests failed: apic access 
emulator idt_test msr xsave rmap_chain svm svm-disabled
16:13:23 DEBUG| Terminating screendump thread
16:13:23 DEBUG| Param 'convert_ppm_files_to_png' specified, converting 
PPM files to PNG format
16:13:23 DEBUG| Param 'keep_ppm_files' not specified, removing all PPM 
files from debug dir
16:13:23 ERROR| Exception escaping from test:
Traceback (most recent call last):
   File "/home/dor/src/autotest/client/common_lib/test.py", line 418, in 
_exec
     _call_test_function(self.execute, *p_args, **p_dargs)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 623, in 
_call_test_function
     return func(*args, **dargs)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 291, in 
execute
     postprocess_profiled_run, args, dargs)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 211, in 
_call_run_once
     self.run_once(*args, **dargs)
   File "/home/dor/src/autotest/client/tests/kvm/kvm.py", line 88, in 
run_once
     run_func(self, params, env)
   File "/home/dor/src/autotest/client/tests/kvm/tests/unittest.py", 
line 121, in run_unittest
     raise error.TestFail("Unit tests failed: %s" % " ".join(tests_failed))
TestFail: Unit tests failed: apic access emulator idt_test msr xsave 
rmap_chain svm svm-disabled
16:13:23 DEBUG| Logging subprocess finished
16:13:23 DEBUG| Logging subprocess finished
16:13:23 DEBUG| Running 'gzip -9 
"/home/dor/src/autotest/client/results/default/kvm.unittest/sysinfo/dmesg"'
16:13:23 ERROR| /var/log/messages collection failed with [Errno 13] 
Permission denied: '/var/log/messages'
16:13:23 DEBUG| Running 'grep MemTotal /proc/meminfo'
16:13:23 DEBUG| Running 'rpm -qa'
16:13:27 ERROR| child process failed
16:13:27 DEBUG| Traceback (most recent call last):
   File "/home/dor/src/autotest/client/bin/parallel.py", line 18, in 
fork_start
     l()
   File "/home/dor/src/autotest/client/bin/job.py", line 520, in <lambda>
     l = lambda : test.runtest(self, url, tag, args, dargs)
   File "/home/dor/src/autotest/client/bin/test.py", line 115, in runtest
     job.sysinfo.log_after_each_iteration)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 710, in 
runtest
     mytest._exec(args, dargs)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 418, in 
_exec
     _call_test_function(self.execute, *p_args, **p_dargs)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 623, in 
_call_test_function
     return func(*args, **dargs)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 291, in 
execute
     postprocess_profiled_run, args, dargs)
   File "/home/dor/src/autotest/client/common_lib/test.py", line 211, in 
_call_run_once
     self.run_once(*args, **dargs)
   File "/home/dor/src/autotest/client/tests/kvm/kvm.py", line 88, in 
run_once
     run_func(self, params, env)
   File "/home/dor/src/autotest/client/tests/kvm/tests/unittest.py", 
line 121, in run_unittest
     raise error.TestFail("Unit tests failed: %s" % " ".join(tests_failed))
TestFail: Unit tests failed: apic access emulator idt_test msr xsave 
rmap_chain svm svm-disabled

16:13:27 INFO | 		FAIL	kvm.unittest	kvm.unittest	timestamp=1310908407 
localtime=Jul 17 16:13:27	Unit tests failed: apic access emulator 
idt_test msr xsave rmap_chain svm svm-disabled
16:13:27 INFO | 	END FAIL	kvm.unittest	kvm.unittest 
timestamp=1310908407	localtime=Jul 17 16:13:27	

"
Still, it's cool and simple to use. I'm was and still am an enthusiastic 
autotest fan!
Dor

Re: KVM autotest tip of the week - Running qemu-kvm unittests

[Lucas Meneghel Rodrigues]

On Sun, Jul 17, 2011 at 10:47 AM, Dor Laor <dlaor@redhat.com> wrote:
> On 07/15/2011 04:12 PM, Lucas Meneghel Rodrigues wrote:
>> Hi guys, due to some personal issues this week this tip came later than
>> I wanted, but nevertheless, here it is:
>>
>> http://autotest.kernel.org/wiki/KVMAutotest/RunQemuUnittests
>>
>> Ever wanted to make kvm autotest to execute the qemu-kvm unittest suite
>> automatically on your local qemu-kvm branch? So this is for you!
>>
>> Cheers,
>>
>> Lucas
>>
>
> Nice write up!
>
> Why not put 'preserve_srcdir = True' by default?

Yeah, I will make some tests and leave it as true, but configurable on
our config files.

> In general, you can simplify the documentation by using the most
> frequent use case at the beginning. For example, you can just point to
> another link to get you all the yum dependencies instead of adding it
> here in this page where they do not belong.
>
> The get_started.py script is cool :)
>
> Note that in your output screendump is not functioning.

Yes, during the unittests it's normal that some screendump requests
will fail. I'll update the config file to disable the screendump
thread during unittests run.

> The setup was really neat and the tests managed to execute easily. I did
> got some failures and exception below:
>
> "16:13:23 ERROR| Test failed: TestFail: Unit tests failed: apic access
> emulator idt_test msr xsave rmap_chain svm svm-disabled
> 16:13:23 DEBUG| Terminating screendump thread
> 16:13:23 DEBUG| Param 'convert_ppm_files_to_png' specified, converting
> PPM files to PNG format
> 16:13:23 DEBUG| Param 'keep_ppm_files' not specified, removing all PPM
> files from debug dir
> 16:13:23 ERROR| Exception escaping from test:
> Traceback (most recent call last):
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 418, in
> _exec
>     _call_test_function(self.execute, *p_args, **p_dargs)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 623, in
> _call_test_function
>     return func(*args, **dargs)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 291, in
> execute
>     postprocess_profiled_run, args, dargs)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 211, in
> _call_run_once
>     self.run_once(*args, **dargs)
>   File "/home/dor/src/autotest/client/tests/kvm/kvm.py", line 88, in
> run_once
>     run_func(self, params, env)
>   File "/home/dor/src/autotest/client/tests/kvm/tests/unittest.py",
> line 121, in run_unittest
>     raise error.TestFail("Unit tests failed: %s" % " ".join(tests_failed))
> TestFail: Unit tests failed: apic access emulator idt_test msr xsave
> rmap_chain svm svm-disabled
> 16:13:23 DEBUG| Logging subprocess finished
> 16:13:23 DEBUG| Logging subprocess finished
> 16:13:23 DEBUG| Running 'gzip -9
> "/home/dor/src/autotest/client/results/default/kvm.unittest/sysinfo/dmesg"'
> 16:13:23 ERROR| /var/log/messages collection failed with [Errno 13]
> Permission denied: '/var/log/messages'
> 16:13:23 DEBUG| Running 'grep MemTotal /proc/meminfo'
> 16:13:23 DEBUG| Running 'rpm -qa'
> 16:13:27 ERROR| child process failed
> 16:13:27 DEBUG| Traceback (most recent call last):
>   File "/home/dor/src/autotest/client/bin/parallel.py", line 18, in
> fork_start
>     l()
>   File "/home/dor/src/autotest/client/bin/job.py", line 520, in <lambda>
>     l = lambda : test.runtest(self, url, tag, args, dargs)
>   File "/home/dor/src/autotest/client/bin/test.py", line 115, in runtest
>     job.sysinfo.log_after_each_iteration)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 710, in
> runtest
>     mytest._exec(args, dargs)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 418, in
> _exec
>     _call_test_function(self.execute, *p_args, **p_dargs)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 623, in
> _call_test_function
>     return func(*args, **dargs)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 291, in
> execute
>     postprocess_profiled_run, args, dargs)
>   File "/home/dor/src/autotest/client/common_lib/test.py", line 211, in
> _call_run_once
>     self.run_once(*args, **dargs)
>   File "/home/dor/src/autotest/client/tests/kvm/kvm.py", line 88, in
> run_once
>     run_func(self, params, env)
>   File "/home/dor/src/autotest/client/tests/kvm/tests/unittest.py",
> line 121, in run_unittest
>     raise error.TestFail("Unit tests failed: %s" % " ".join(tests_failed))
> TestFail: Unit tests failed: apic access emulator idt_test msr xsave
> rmap_chain svm svm-disabled

Yes, this is normal autotest behavior, although it does look like a
bug because of the traceback. By default, when a test fails, autotest
will print the traceback to make it easy for test developers to figure
out things in case of a test bug. I just ommited the traceback on the
docs because it's too lengthy. I might look at supressing the
traceback on console execution or something.

Also, it's important to note that the result itself of the unittests
depends largely on kernel interfaces, and right now not even Avi's
kernel + qemu-kvm.git master HEAD give 100% PASS, so what you saw are
legitimate failures, but some of them might not apply to the kernel
you are running.

> 16:13:27 INFO |                 FAIL    kvm.unittest    kvm.unittest    timestamp=1310908407
> localtime=Jul 17 16:13:27       Unit tests failed: apic access emulator
> idt_test msr xsave rmap_chain svm svm-disabled
> 16:13:27 INFO |         END FAIL        kvm.unittest    kvm.unittest
> timestamp=1310908407    localtime=Jul 17 16:13:27
>
> "
> Still, it's cool and simple to use. I'm was and still am an enthusiastic
> autotest fan!

Thanks Dor, your feedback is much appreciated! I encourage the other
team members to try and provide us more feedback.

Lucas

Re: KVM autotest tip of the week - Running qemu-kvm unittests

[Lucas Meneghel Rodrigues]

On Sun, Jul 17, 2011 at 10:47 AM, Dor Laor <dlaor@redhat.com> wrote:
> Why not put 'preserve_srcdir = True' by default?

Ok, implemented on http://autotest.kernel.org/changeset/5500.

> In general, you can simplify the documentation by using the most
> frequent use case at the beginning. For example, you can just point to
> another link to get you all the yum dependencies instead of adding it
> here in this page where they do not belong.

Suggestions implemented, refactored the documentation in a way that
commonly used doc snippets can be reused among documents. Updated all
the docs based on this. see
http://autotest.kernel.org/wiki/KVMAutotest/RunQemuUnittests.

> Note that in your output screendump is not functioning.

Implemented a fix on http://autotest.kernel.org/changeset/5498.

> The setup was really neat and the tests managed to execute easily. I did
> got some failures and exception below:

.... snip

Sent an RFC to the list to the list proposing a way to reduce printing
tracebacks of a same exception twice and logging the traceback as a
DEBUG message, only for test writers to debug their tests.

http://patchwork.test.kernel.org/patch/3558/

Thanks for the feedback!

-- 
Lucas