Re: [RFC PATCH 0/8] Convert avocado tests to normal Python unittests

[Thomas Huth <thuth@redhat.com>]

On 16/07/2024 18.45, John Snow wrote:
> On Thu, Jul 11, 2024, 7:55 AM Thomas Huth <thuth@redhat.com 
> <mailto:thuth@redhat.com>> wrote:
...
>     - I haven't looked into logging yet ... this still needs some work
>        so that you could e.g. inspect the console output of the guests
>        somewhere

FWIW: This is now done in the next version of the patch series:

  https://lore.kernel.org/qemu-devel/20240716112614.1755692-10-thuth@redhat.com/

> This has spilled the most developer blood of any other problem with the 
> Python-based tests. Be very careful here.

Apart from 1:1 copying the functions from one __init__.py file to the other, 
and from setting up the logger so that it writes its output to a file, I 
didn't have to change anything. It currently simply seems to work.

> I still have a prototype for replacing QMPMachine with an asyncio variant 
> that should have more robust logging features, but I put it on the back-burner.
> 
> Avocado tests are the primary user of the QMP Machine interface I hate the 
> very most, a multi-threaded buffer-reader that works only by the grace of 
> god. If you do go down this path, I may want to take the opportunity to 
> abolish that interface once and for all.
>
 > I think simplifying the console buffering will help ease debuggability.

Feel free to do improvements on top! I think it should be easier now when 
there are no more complicated mixtures with the avocado test runner.

>     What's your thoughts? Is it worth to continue with this approach?
>     Or shall I rather forget about it and wait for the Avocado version
>     update?
> 
> 
> I'm personally ambivalent on avocado; I use it for the python self-tests as 
> dogfooding but I can likely switch back over to plain pytest if that's the 
> direction we head. I don't think I use any crazy features except some 
> asyncio helpers i advocated for. I'm not sure what pytest's asyncio support 
> looks like, but I have to imagine as the premier testing framework that it 
> has *something* for me to use.

There's no more pytest harness in the next iteration of the patch series, 
just the need for pycotap for TAP output. Console logging is completely 
independent of the test runner, I'll simply do normal logging to files there.

> My only ask is that we keep the tests running in the custom venv environment 
> we set up at build time. We have some funky post-hoc initialization of 
> avocado that allows us to use internet packages post-config for testing 
> purposes. If we move to pytest, it's possible we can eliminate that 
> funkiness, which would be a win.

I still need a way for making sure that pycotap is installed, though, so the 
venv is still there.

> I'm also not so sure about recreating all of the framework that pulls vm 
> images on demand, that sounds like it'd be a lot of work, but maybe I'm 
> wrong about that.

It likely does not make sense to rewrite the tests that use these cloud-init 
images (i.e. the ones that depend on the LinuxTest class). But we could 
likely simply continue to use avocado.utils for these, without using the 
avocado test runner.

> Tacit ACK from me on this project in general, provided we are still using 
> the configure venv.

  Thanks,
   Thomas