Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests

["Daniel P. Berrangé" <berrange@redhat.com>]

On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote:
> On 16/07/2024 19.03, Thomas Huth wrote:
> > On 16/07/2024 18.51, Daniel P. Berrangé wrote:
> > > On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> > ...
> > > > So instead of trying to update the python-based test suite in QEMU
> > > > to a newer version of Avocado, we should maybe try to better integrate
> > > > it with the meson test runner instead. Indeed most tests work quite
> > > > nicely without the Avocado framework already, as you can see with
> > > > this patch series - it does not convert all tests, just a subset so
> > > > far, but this already proves that many tests only need small modifi-
> > > > cations to work without Avocado.
> > ...
> > > > Now if you want to try out these patches: Apply the patches, then
> > > > recompile and then run:
> > > > 
> > > >   make check-functional
> > > > 
> > > > You can also run single targets e.g. with:
> > > > 
> > > >   make check-functional-ppc
> > > > 
> > > > You can also run the tests without any test runner now by
> > > > setting the PYTHONPATH environment variable to the "python" folder
> > > > of your source tree, and by specifying the build directory via
> > > > QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> > > > QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
> > > > 
> > > >   export PYTHONPATH=$HOME/qemu/python
> > > >   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
> > > >   export PYTHONPATH=$HOME/qemu/build
> > > >   ~/qemu/tests/functional/test_virtio_version.py
> > > 
> > > For the whole series as is
> > > 
> > >   Tested-by: Daniel P. Berrangé <berrange@redhat.com>
> > > 
> > > as it does what you claim it does here when I tried it.
> > 
> > Thanks!
> > 
> > > > The logs of the tests can be found in the build directory under
> > > > tests/functional/<testname> - console log and general logs will
> > > > be put in separate files there.
> > > 
> > > As an example, one dir name appears to be:
> > > 
> > >    __main__.MemAddrCheck.test_phybits_ok_pentium_pae
> > > 
> > > I'd rather prefer it if the dir name matched the test script
> > > file name - in this case test_mem_addr_space.py, as I don't
> > > want to have to lookup which class names were defined inside
> > > each test script. We could drop the "test_" prefix from the
> > > method name too
> > > 
> > > IOW, could we make this dir name be:
> > > 
> > >    test_mem_addr_space.phybits_ok_pentium_pae
> > 
> > I can try to change that, indeed ... but the boilerplate code will
> > increase a little bit, I guess, since I cannot simply rely on the
> > unittest.id() function in that case anymore...
> 
> After looking at this for a while, I think it's maybe best to ditch the idea
> of making the .py files directly runnable and run the tests via a simple
> pycotap runner instead. Then you get proper module names:

I'd really not want to loose that. To me, eliminating the test harness
entirely when debugging is the single biggest improvement of this new
approach, especially when I want to 'strace' the test without
extraneous processes.

> $ pyvenv/bin/python3 -m pycotap test_virtio_version
> TAP version 13
> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
> 1..

With the following change, you get the same output with direct
execution, by making argv look the same as you'd get when
running your pycotap example.

diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
index cc49fd4c94..3a3e65252d 100644
--- a/tests/functional/qemu_test/__init__.py
+++ b/tests/functional/qemu_test/__init__.py
@@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
     def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        unittest.main(testRunner = tr)
+        import sys
+        import os.path
+        path = os.path.basename(sys.argv[0])[:-3]
+        unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
 
 
 class QemuSystemTest(QemuBaseTest):

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|