Re: Avocado notes from KVM forum 2019

[Cleber Rosa <crosa@redhat.com>]

On Mon, Nov 25, 2019 at 01:35:13PM +0100, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> Here are my notes from talking about Avocado with various people during the
> KVM forum in Lyon last month.
> 
> All comments are QEMU oriented.
> 
> 
> 1) Working offline
> 
> Various people complained Avocado requires online access, and they would
> like to use it offline.
>

Just to be extra clear, Avocado itself doesn't require online access, but:

 * "make check-venv" will download packages from PyPI if the same
   Avocado version is not previously installed in the system

 * "make check-acceptance" depends on "make check-venv"

 * many tests depend on downlodable assets

>   Maintainer workflow example is:
> 
>   - run avocado
>   - hack QEMU, build
>   - git pull
>   - build
>   - hack QEMU
>   (go offline)
>   - hack QEMU
>   - build
>   - run avocado <- FAILS
> 
> Failure is because mainstream added new tests, which got pulled in, and the
> user only notice when running avocado again, but offline.
> Test example is boot_linux_console.py, which added various tests from other
> subsystems, so the maintainer has to disable the new tests manually to be
> able to run his previous tests.
> 
> Expected solution: skip tests when artifact is not available, eventually
> when the --offline option is used
>

Understood and agreed.  I've already adopted this approach in the
"boot_linux.py" series I'm working (about to send v8).  If the
download/preparation of images fail, we cancel the tests.  I'll look
into making that the default across all tests or in the base
avocado_qemu.Test class.

We could also have a "make check-acceptance-prepare" kind of target,
that won't run the tests, but will download all needed assets.

> 
> 2) Add artifacts manually to the cache
> 
> Not all artifacts can be easily downloadable, some are public but require
> the user to accept an End User License Agreement.
> Users would like to share their tests with the documentation about where/how
> to download the requisite files (accepting the EULA) to run the tests.
> 
>

OK, RFE taken.

> 2b) Add reference to artifact to the cache
> 
> Group of users might share group of files (like NFS storage) and would like
> to use directly their remote read-only files, instead of copying it to their
> home directory.
> 
>

The underlying "asset fetcher" utility code supports multiple locations
and multiple cache dirs (and one could be a read-only NFS-like storage):

  https://avocado-framework.readthedocs.io/en/73.0/api/utils/avocado.utils.html#avocado.utils.asset.Asset

But we need to make that easily accessible/configurable to users
of the fetch_asset() Test method.  RFE taken.

> 3) Provide qemu/avocado-qemu Python packages
> 
> The mainstream project uses Avocado to test QEMU. Others projects use QEMU
> to test their code, and would like to automatize that using Avocado. They
> usually not rebuild QEMU but use a stable binary from distributions. The
> python classes are not available, so they have to clone QEMU to use Avocado
> (I guess they only need 5 python files).
> When running on Continuous Integration, this is overkill, because when you
> clone QEMU you also clone various other submodules.
>

Agreed, and I already have a prototype.  I'll send the RFC/patches to
the list ASAP.

> 
> 4) Warn the user when Avocado is too old for the tests
> 
> Some users tried Avocado following the examples on the mailing list and the
> one in some commit's descriptions where we simply show "avocado run ...".
> They installed the distribution Avocado package and tried and it fails for
> few of them with no obvious reason (the .log file is hard to read when you
> are not custom to). IIUC their distribution provides a older Avocado (69?)
> while we use recent features (72).
> 
> We never noticed it because we use 'make check-venv' and do not test the
> distrib Avocado. While we can not test all distribs, we could add a version
> test if the Avocado version is too old, display a friendly message to the
> console (not the logfile).
>

OK, agreed.  RFE taken.

> 
> That it for my notes.
> 
> Eduardo/Wainer, are there other topics I forgot?
> 
> 
> Regards,
> 
> Phil.
> 

Thanks *so much* for this feedback!  I'll provide individual feedback as
each of those items progresses.

- Cleber.