Re: [Qemu-devel] [Autotest] [Autotest PATCH 2/2] virt run: add three logical case filters

[Lucas Meneghel Rodrigues <lmr@redhat.com>]

On 01/04/2013 11:13 AM, Andreas Färber wrote:
> Am 31.12.2012 20:55, schrieb Lucas Meneghel Rodrigues:
>> Hmmm, about this one, I'm worried about making things more complex...
>>
>> The way I see the problem at hand, I'd say if people want to customize
>> things, they'd be better of creating their own, specialized config
>> files rather than adding command line flags to manipulate the test
>> sets.
>>
>> So my initial stand on this particular patch is NACK, but you might
>> convince me otherwise :)
>
> Lucas, following your nice presentation at KVM Forum I have set up
> virt-test for local testing of my core CPU refactorings.
> When run using
> ./run -t kvm --qemu-bin=/.../x86_64-softmmu/qemu-system-x86_64
> it runs only 13 migration tests:
>
> (1/13) migrate.default.tcp: PASS (41.04 s)
> (2/13) migrate.default.unix: PASS (21.88 s)
> (3/13) migrate.default.exec: PASS (20.84 s)
> (4/13) migrate.default.fd: PASS (23.04 s)
> (5/13) migrate.default.mig_cancel: PASS (18.66 s)
> (6/13) migrate.with_set_speed.tcp: PASS (19.22 s)
> (7/13) migrate.with_set_speed.unix: PASS (19.07 s)
> (8/13) migrate.with_set_speed.exec: PASS (19.03 s)
> (9/13) migrate.with_set_speed.fd: PASS (18.93 s)
> (10/13) migrate.with_reboot.tcp: PASS (40.97 s)
> (11/13) migrate.with_reboot.unix: PASS (42.90 s)
> (12/13) migrate.with_reboot.exec: PASS (47.04 s)
> (13/13) migrate.with_reboot.fd: PASS (47.95 s)
>
> Whereas --list-tests shows 274 tests.
>
> So while I am impartial to this specific patch, some easy way to run a
> comprehensive test coverage without having to manually name each test
> using --tests= would be very handy! Something like --all-tests maybe?

Hmm, could be. Due to the long time it'd be needed to run --all-tests, I 
thought it wouldn't be necessary.

> If there is such a thing already, it is not obvious to the novice user
> and --help output may need to be extended.

One thing that does help here is to know that you can specify only the 
first component of the test name displayed on --help, and it'll run all 
derivated tests. For example, if you want to run all virtio_console 
tests, you can just specify:

./run -t kvm --qemu-bin=/path/to/qemu --tests virtio_console

SETUP: PASS (1.60 s)
DATA DIR: /home/area/virt_test/
DEBUG LOG: 
/home/lmr/Code/virt-test.git/logs/run-2013-01-04-12.20.20/debug.log
TESTS: 118
(1/118) 
virtio_console.spread_linear.specifiable.virtserialport.with_vm.open:

I need to think of a good way of exposing this information on help output...