Re: [PATCH 1/5] tests/functional: skip test if QEMU_TEST_QEMU_BINARY is not set

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Feb 06, 2025 at 10:01:45AM +0100, Thomas Huth wrote:
> On 05/02/2025 16.59, Daniel P. Berrangé wrote:
> > If QEMU_TEST_QEMU_BINARY is not set we currently assert in the setUp
> > function, resulting in a big traceback:
> > 
> >      TAP version 13
> >      Traceback (most recent call last):
> >        File "/var/home/berrange/src/virt/qemu/tests/functional/qemu_test/testcase.py", line 280, in setUp
> >          super().setUp('qemu-system-')
> >          ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
> >        File "/var/home/berrange/src/virt/qemu/tests/functional/qemu_test/testcase.py", line 196, in setUp
> >          self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
> >          ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >      AssertionError: unexpectedly None : QEMU_TEST_QEMU_BINARY must be set
> > 
> >      not ok 1 test_ppc_405.Ppc405Machine.test_ppc_ref405ep
> >      1..1
> > 
> > For every other test pre-requisite that's missing we will mark the test
> > as skipped. This does the same for missing QEMU_TEST_QEMU_BINARY, such
> > that we get
> > 
> >      TAP version 13
> >      ok 1 test_x86_64_kvm_xen.KVMXenGuest.test_kvm_xen_guest # SKIP QEMU_TEST_QEMU_BINARY env variable is not set
> >      ok 2 test_x86_64_kvm_xen.KVMXenGuest.test_kvm_xen_guest_noapic_nomsi # SKIP QEMU_TEST_QEMU_BINARY env variable is not set
> >      ok 3 test_x86_64_kvm_xen.KVMXenGuest.test_kvm_xen_guest_nomsi # SKIP QEMU_TEST_QEMU_BINARY env variable is not set
> >      ok 4 test_x86_64_kvm_xen.KVMXenGuest.test_kvm_xen_guest_novector # SKIP QEMU_TEST_QEMU_BINARY env variable is not set
> >      ok 5 test_x86_64_kvm_xen.KVMXenGuest.test_kvm_xen_guest_novector_noapic # SKIP QEMU_TEST_QEMU_BINARY env variable is not set
> >      ok 6 test_x86_64_kvm_xen.KVMXenGuest.test_kvm_xen_guest_novector_nomsi # SKIP QEMU_TEST_QEMU_BINARY env variable is not set
> >      ok 7 test_x86_64_kvm_xen.KVMXenGuest.test_kvm_xen_guest_vapic # SKIP QEMU_TEST_QEMU_BINARY env variable is not set
> >      1..7
> 
> Not sure whether this is the right approach, since a missing
> QEMU_TEST_QEMU_BINARY is a real error, and if we just skip, then the problem
> might go unnoticed if the user does not look closely.
> 
> But to ease the situation: We could maybe add some auto-detection logic that
> tries to guess the right qemu-system-$TARGET by looking at the file name of
> the test and/or the test function name? We already encode the target
> architecture in most of these... WDYT?

When the user provides a binary, a single binary applies to all tests,
so tests that don't match the binary get skipped. I think it would be
a bit wierd to auto-select a different binary per test.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|