Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Cleber Rosa <crosa@redhat.com>]

----- Original Message -----
> From: "Eduardo Habkost" <ehabkost@redhat.com>
> To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Aleksandar Markovic"
> <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <wainersm@redhat.com>
> Sent: Wednesday, May 22, 2019 5:12:30 PM
> Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> 
> On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > Hi,
> > 
> > It was a rainy week-end here, so I invested it to automatize some
> > of my MIPS tests.
> > 
> > The BootLinuxSshTest is not Global warming friendly, it is not
> > meant to run on a CI system but rather on a workstation previous
> > to post a pull request.
> > It can surely be improved, but it is a good starting point.
> 
> Until we actually have a mechanism to exclude the test case on
> travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> please don't merge patch 4/4 yet or it will break travis-ci.
> 
> Cleber, Wainer, is it already possible to make "avocado run" skip
> tests tagged with "slow"?
> 

The mechanism exists, but we haven't tagged any test so far as slow.

Should we define/document a criteria for a test to be slow?  Given
that this is highly subjective, we have to think of:

 * Will we consider the average or maximum run time (the timeout
   definition)?
 
 * For a single test, what is "slow"? Some rough numbers from Travis
   CI[1] to help us with guidelines:
   - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
   - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
   - linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:  PASS (18.14 s)
   - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)

 * Do we want to set a maximum job timeout?  This way we can skip
   tests after a given amount of time has passed.  Currently we interrupt
   the test running when the job timeout is reached, but it's possible
   to add a option so that no new tests will be started, but currently
   running ones will be waited on.

Regards,
- Cleber. 

[1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518

> --
> Eduardo
>