Re: [PATCH 8/8] Tests: add custom test jobs

[Cleber Rosa <crosa@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5363 bytes --]

On Fri, Apr 16, 2021 at 07:23:14AM +0200, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 4/15/21 11:51 PM, Cleber Rosa wrote:
> > Different users (or even companies) have different interests, and
> > may want to run a reduced set of tests during development, or a
> > larger set of tests during QE.
> > 
> > To cover these use cases, some example (but functional) jobs are
> > introduced here:
> > 
> > 1) acceptance-all-targets.py: runs all arch agnostic tests on all
> >    built targets, unless there are conditions that make them not work
> >    out of the box ATM, then run all tests that are specific to
> >    predefined targets.
> > 
> > 2) acceptance-kvm-only.py: runs only tests that require KVM and are
> >    specific to the host architecture.
> > 
> > 3) qtest-unit.py: runs a combination of qtest and unit tests (in
> >    parallel).
> > 
> > 4) qtest-unit-acceptance.py: runs a combineation of qtest, unit tests
> 
> Typo "combination".
> 
> >    and acceptance tests (all of them in parallel)
> > 
> > To run the first two manually, follow the example bellow:
> > 
> >  $ cd build
> >  $ make check-venv
> >  $ ./tests/venv/bin/python3 tests/jobs/acceptance-all-targets.py
> >  $ ./tests/venv/bin/python3 tests/jobs/acceptance-kvm-only.py
> > 
> > The third and fouth example depends on information coming from Meson,
> > so the easiest way to run it is:
> > 
> >  $ cd build
> >  $ make check-qtest-unit
> >  $ make check-qtest-unit-acceptance
> > 
> > These are based on Avocado's Job API, a way to customize an Avocado
> > job execution beyond the possibilities of command line arguments.
> > For more Job API resources, please refer to:
> > 
> > a) Job API Examples:
> >  - https://github.com/avocado-framework/avocado/tree/master/examples/jobs
> > 
> > b) Documentation about configurable features at the Job Level:
> >  - https://avocado-framework.readthedocs.io/en/87.0/config/index.html
> > 
> > c) Documentation about the TestSuite class
> >  - https://avocado-framework.readthedocs.io/en/87.0/api/core/avocado.core.html#avocado.core.suite.TestSuite
> > 
> > d) Documentation about the Job class
> >  - https://avocado-framework.readthedocs.io/en/87.0/api/core/avocado.core.html#avocado.core.job.Job
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >  configure                            |  2 +-
> >  tests/Makefile.include               |  8 ++++
> >  tests/jobs/acceptance-all-targets.py | 67 ++++++++++++++++++++++++++++
> >  tests/jobs/acceptance-kvm-only.py    | 35 +++++++++++++++
> >  tests/jobs/qtest-unit-acceptance.py  | 31 +++++++++++++
> >  tests/jobs/qtest-unit.py             | 24 ++++++++++
> >  tests/jobs/utils.py                  | 22 +++++++++
> >  7 files changed, 188 insertions(+), 1 deletion(-)
> >  create mode 100644 tests/jobs/acceptance-all-targets.py
> >  create mode 100644 tests/jobs/acceptance-kvm-only.py
> >  create mode 100644 tests/jobs/qtest-unit-acceptance.py
> >  create mode 100644 tests/jobs/qtest-unit.py
> >  create mode 100644 tests/jobs/utils.py
> 
> > +if __name__ == '__main__':
> > +    sys.exit(main())
> > diff --git a/tests/jobs/acceptance-kvm-only.py b/tests/jobs/acceptance-kvm-only.py
> > new file mode 100644
> > index 0000000000..acdcbbe087
> > --- /dev/null
> > +++ b/tests/jobs/acceptance-kvm-only.py
> > @@ -0,0 +1,35 @@
> > +#!/usr/bin/env python3
> > +
> > +import os
> > +import sys
> > +
> > +# This comes from tests/acceptance/avocado_qemu/__init__.py and should
> > +# not be duplicated here.  The solution is to have the "avocado_qemu"
> > +# code and "python/qemu" available during build
> > +BUILD_DIR = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
> > +if os.path.islink(os.path.dirname(os.path.dirname(__file__))):
> > +    # The link to the acceptance tests dir in the source code directory
> > +    lnk = os.path.dirname(os.path.dirname(__file__))
> > +    #: The QEMU root source directory
> > +    SOURCE_DIR = os.path.dirname(os.path.dirname(os.readlink(lnk)))
> > +else:
> > +    SOURCE_DIR = BUILD_DIR
> > +sys.path.append(os.path.join(SOURCE_DIR, 'python'))
> > +
> > +from avocado.core.job import Job
> > +
> > +from qemu.accel import kvm_available
> > +
> > +
> > +def main():
> > +    if not kvm_available():
> > +        sys.exit(0)
> > +
> > +    config = {'run.references': ['tests/acceptance/'],
> > +              'filter.by_tags.tags': ['accel:kvm,arch:%s' % os.uname()[4]]}
> 
> If we want forks to use their own set of tags, it would be better to
> provide an uniform way, not adding new test entry point for each set
> of fork tags. Could we consume a YAML config file instead? And provide
> templates so forks could adapt to their needs?
>
> Thanks,
> 
> Phil.
> 

Yes, it should be possible indeed.  BTW, starting this kind of
discussion is one of the main goals of this series.

With regards to your suggestion, I believe there's an audience and
value to this KVM only job.  But, your idea of a YAML config file is
also very much valid.

Maybe even a job that retrieves the list of tags from a CI variable?
That could allow people to run jobs for different subset of tests at
"push" time, without the need to add committed changes to (YAML)
config files.

Cheers,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]