Re: [Qemu-devel] [PATCH v2 02/20] Acceptance tests: show avocado test execution by default

[Cleber Rosa <crosa@redhat.com>]

On 2/6/19 9:36 AM, Cornelia Huck wrote:
> On Fri,  1 Feb 2019 19:55:52 -0500
> Cleber Rosa <crosa@redhat.com> wrote:
> 
>> The current version of the "check-acceptance" target will only show
>> one line for execution of all tests.  That's probably OK if the tests
>> to be run are quick enough and they're always the same.
>>
>> But, there's already one test alone that takes on average ~5 seconds
>> to run, we intend to adapt the list of tests to match the user's build
>> environment (among other choices).
> 
> Btw: What are our expectations regarding execution time for tests?
> Especially if we continue adding tests, and architecture-specific tests
> are bound to be slower if run on a foreign architecture via tcg.
> 
> Would a make check-acceptance-quick command make sense? ("I only want
> to verify quickly that I didn't break too much, so run the quicker
> tests only, probably only for my host architecture")
> 

Yes, it definitely makes sense.  Now, let me know if the following also
makes sense to you:

1) Because these tests focus on functional testing, the default
target/shortcut ("make check-acceptance") should run the complete set of
test cases (including the slow ones).

2) Requirements vary greatly from user to user, to while adding a
"check-acceptance-quick" is fine, you just mentioned one extra test
execution variation ("for my host architecture").  For those, the idea
is that:

 a) "make check-acceptance[-quick]" will adapt to the build environment
(if you only built s390x targets, that's all it's going to use)

 b) "avocado" command line interface *should* be easy enough to fulfill
other requirements, and not necessarily require a "make" target.  For
instance, if you're only interested in your host arch and one specific
machine type, a command line such as the following should do the trick:

  $ make check-venv
  $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t
machine:WHAT_I_CARE_ABOUT tests/acceptance/

How does that sound?

Regards,
- Cleber.

>>
>> Because of that, let's present the default Avocado UI by default.
>> Users can always choose a different output by setting the AVOCADO_SHOW
>> variable.
>>
>> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
>> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  .travis.yml            | 2 +-
>>  tests/Makefile.include | 2 +-
>>  2 files changed, 2 insertions(+), 2 deletions(-)