Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Aleksandar Markovic <aleksandar.m.mail@gmail.com>]

On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
>
>
>
> ----- Original Message -----
> > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
"Aleksandar Markovic"
> > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <
amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <
wainersm@redhat.com>
> > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> >
> > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > >
> > > It was a rainy week-end here, so I invested it to automatize some
> > > of my MIPS tests.
> > >
> > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > meant to run on a CI system but rather on a workstation previous
> > > to post a pull request.
> > > It can surely be improved, but it is a good starting point.
> >
> > Until we actually have a mechanism to exclude the test case on
> > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > please don't merge patch 4/4 yet or it will break travis-ci.
> >
> > Cleber, Wainer, is it already possible to make "avocado run" skip
> > tests tagged with "slow"?
> >
>
> The mechanism exists, but we haven't tagged any test so far as slow.
>

Cleber,

For the test from patch 4/4, there is no dilemma - it should be in the
“slow” group, as Philippe envisioned and said, so that it is not humpered
with stricter requirements for “fast” (default) group. Could you explain us
how to do it, so that we can hopefully finally proceed?

Gratefully,
Aleksandar

> Should we define/document a criteria for a test to be slow?  Given
> that this is highly subjective, we have to think of:
>
>  * Will we consider the average or maximum run time (the timeout
>    definition)?
>
>  * For a single test, what is "slow"? Some rough numbers from Travis
>    CI[1] to help us with guidelines:
>    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
>    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
>    -
linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
PASS (18.14 s)
>    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
>
>  * Do we want to set a maximum job timeout?  This way we can skip
>    tests after a given amount of time has passed.  Currently we interrupt
>    the test running when the job timeout is reached, but it's possible
>    to add a option so that no new tests will be started, but currently
>    running ones will be waited on.
>
> Regards,
> - Cleber.
>
> [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
>
> > --
> > Eduardo
> >