Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests

[Cleber Rosa <crosa@redhat.com>]

----- Original Message -----
> From: "Aleksandar Markovic" <aleksandar.m.mail@gmail.com>
> To: "Cleber Rosa" <crosa@redhat.com>
> Cc: "Wainer dos Santos Moschetta" <wainersm@redhat.com>, "Aleksandar Markovic" <amarkovic@wavecomp.com>,
> qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>, "Eduardo Habkost" <ehabkost@redhat.com>,
> "Aurelien Jarno" <aurelien@aurel32.net>, "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> Sent: Wednesday, May 22, 2019 6:43:54 PM
> Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> 
> On May 22, 2019 11:46 PM, "Cleber Rosa" <crosa@redhat.com> wrote:
> >
> >
> >
> > ----- Original Message -----
> > > From: "Eduardo Habkost" <ehabkost@redhat.com>
> > > To: "Philippe Mathieu-Daudé" <f4bug@amsat.org>
> > > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arikalo@wavecomp.com>,
> "Aleksandar Markovic"
> > > <aleksandar.m.mail@gmail.com>, "Aleksandar Markovic" <
> amarkovic@wavecomp.com>, "Cleber Rosa" <crosa@redhat.com>,
> > > "Aurelien Jarno" <aurelien@aurel32.net>, "Wainer dos Santos Moschetta" <
> wainersm@redhat.com>
> > > Sent: Wednesday, May 22, 2019 5:12:30 PM
> > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests
> > >
> > > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote:
> > > > Hi,
> > > >
> > > > It was a rainy week-end here, so I invested it to automatize some
> > > > of my MIPS tests.
> > > >
> > > > The BootLinuxSshTest is not Global warming friendly, it is not
> > > > meant to run on a CI system but rather on a workstation previous
> > > > to post a pull request.
> > > > It can surely be improved, but it is a good starting point.
> > >
> > > Until we actually have a mechanism to exclude the test case on
> > > travis-ci, I will remove patch 4/4 from the queue.  Aleksandar,
> > > please don't merge patch 4/4 yet or it will break travis-ci.
> > >
> > > Cleber, Wainer, is it already possible to make "avocado run" skip
> > > tests tagged with "slow"?
> > >
> >
> > The mechanism exists, but we haven't tagged any test so far as slow.
> >
> 
> Cleber,
> 
> For the test from patch 4/4, there is no dilemma - it should be in the
> “slow” group, as Philippe envisioned and said, so that it is not humpered
> with stricter requirements for “fast” (default) group. Could you explain us
> how to do it, so that we can hopefully finally proceed?
> 

Hi Aleksandar,

The point is that there's no "group" definition at this point.  This is the
core of the discussion.

I think we're close to converging to something simple and effective.  Please
let us know what you think of the proposals given.

Thanks!
- Cleber.

> Gratefully,
> Aleksandar
> 
> > Should we define/document a criteria for a test to be slow?  Given
> > that this is highly subjective, we have to think of:
> >
> >  * Will we consider the average or maximum run time (the timeout
> >    definition)?
> >
> >  * For a single test, what is "slow"? Some rough numbers from Travis
> >    CI[1] to help us with guidelines:
> >    - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc:  PASS (6.04 s)
> >    - boot_linux_console.py:BootLinuxConsole.test_arm_virt:  PASS (2.91 s)
> >    -
> linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16:
> PASS (18.14 s)
> >    - boot_linux.py:BootLinuxAarch64.test_virt:  PASS (396.88 s)
> >
> >  * Do we want to set a maximum job timeout?  This way we can skip
> >    tests after a given amount of time has passed.  Currently we interrupt
> >    the test running when the job timeout is reached, but it's possible
> >    to add a option so that no new tests will be started, but currently
> >    running ones will be waited on.
> >
> > Regards,
> > - Cleber.
> >
> > [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518
> >
> > > --
> > > Eduardo
> > >
>