Re: [PATCH v2 00/23] Convert avocado tests to normal Python unittests

[Thomas Huth <thuth@redhat.com>]

On 25/07/2024 01.35, Richard Henderson wrote:
> On 7/25/24 03:52, Thomas Huth wrote:
>> The Avocado v88 that we use in QEMU is already on a life support
>> system: It is not supported by upstream anymore, and with the latest
>> versions of Python, it won't work anymore since it depends on the
>> "imp" module that has been removed in Python 3.12.
>>
>> There have been several attempts to update the test suite in QEMU
>> to a newer version of Avocado, but so far no attempt has successfully
>> been merged yet.
>>
>> Additionally, the whole "make check" test suite in QEMU is using the
>> meson test runner nowadays, so running the python-based tests via the
>> Avocodo test runner looks and feels quite like an oddball, requiring
>> the users to deal with the knowledge of multiple test runners in
>> parallel (e.g. the timeout settings work completely differently).
>>
>> So instead of trying to update the python-based test suite in QEMU
>> to a newer version of Avocado, we should try to better integrate
>> it with the meson test runner instead. Indeed most tests work quite
>> nicely without the Avocado framework already, as you can see with
>> this patch series - it does not convert all tests, just a subset so
>> far, but this already proves that many tests only need small modifi-
>> cations to work without Avocado.
>>
>> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
>> classes (e.g. based on cloud-init images or using SSH) really depend
>> on the Avocado framework, so we'd need a solution for those if we
>> want to continue using them. One solution might be to simply use the
>> required functions from avocado.utils for these tests, and still run
>> them via the meson test runner instead, but that needs some further
>> investigation that will be done later.
>>
>>
>> Now if you want to try out these patches: Apply the patches, then
>> recompile and then run:
>>
>>   make check-functional
>>
>> You can also run single targets e.g. with:
>>
>>   make check-functional-ppc
>>
>> You can also run the tests without any test runner now by
>> setting the PYTHONPATH environment variable to the "python" folder
>> of your source tree, and by specifying the build directory via
>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>
>>   export PYTHONPATH=$HOME/qemu/python
>>   export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>   export QEMU_BUILD_ROOT=$HOME/qemu/build
>>   ~/qemu/tests/functional/test_virtio_version.py
>>
>> The logs of the tests can be found in the build directory under
>> tests/functional/<arch>/<testname> - console log and general logs will
>> be put in separate files there.
>>
>> Still to be done: Update the documentation for this new test framework.
> 
> I'll say again that the download *must* be handled separately from the test 
> with timeout. This is an absolute show-stopper.
> 
> I've tried this twice now, from a decently fast connection in central 
> Brisbane, and have had multiple downloads be canceled by the timeout.  Since 
> the download isn't clever enough to pick up where it left off, it will never 
> succeed.

  Hi Richard,

just for my understanding, did you try to run the tests in parallel (i.e. 
something like "make -j$(nproc)")? ... I think in that case you can easily 
clog your internet connection even on modern systems if a lot of tests are 
trying to download the assets in parallel.

For me, it works fine if I use normal serial testing with "-j" (btw. Avocado 
v88 is doing serial testing, too, so you won't lose much time during the 
first run here). But if downloading fails for you without "-j", too, I 
agree, we need to tackle that problem first, e.g. by implementing what 
Daniel suggested. That will take a little bit longer, of course, so I hope 
you meanwhile found a work-around for the problem with the missing "imp" 
package on your system?

  Thomas