how do the iotests pick a machine model to run on ?

how do the iotests pick a machine model to run on ?

[Peter Maydell]

If you build QEMU with support for the sh4 target only
(configure --target-list=sh4-softmmu) then 'make check' fails
in the iotests, because some iotests, including for instance 040,
try to create a machine with a virtio-scsi device, but they don't
do anything to ensure that the machine they create actually
has a PCI bus to plug the virtio-scsi device into, and so the test
fails:

+Traceback (most recent call last):
+  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/tests/qemu-iotests/040",
line 94, in setUp
+    self.vm.launch()
+  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/python/qemu/machine/machine.py",
line 455, in launch
+    raise VMLaunchFailure(
+qemu.machine.machine.VMLaunchFailure: ConnectError: Failed to
establish session: EOFError
+       Exit code: 1
+       Command:
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/sh4/qemu-system-sh4
-display none -vga none -chardev socket,id=mon,fd=5 -mon
chardev=mon,mode=control -chardev socket,id=qtest,fd=3 -qtest
chardev:qtest -accel qtest -nodefaults -display none -accel qtest
-drive if=none,id=drive0,file=/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/sh4/scratch/qcow2-file-040/test.img,format=qcow2,cache=writeback,aio=threads,node-name=top,backing.node-name=mid,backing.backing.node-name=base
-device virtio-scsi -device scsi-hd,id=scsi0,drive=drive0
+       Output: [I 0.000000] OPENED
+qemu-system-sh4: -device virtio-scsi: No 'PCI' bus found for device
'virtio-scsi-pci'

(It happens that the default machine type for 'sh4'
does not have a PCI bus.)

How are the iotests supposed to select a machine model to run
on, and how are they intended to mark themselves as requiring
particular facilities, like PCI? Presumably some of the tests
are missing the appropriate annotations to ensure they're
skipped when the facilities required aren't present.

thanks
-- PMM

Re: how do the iotests pick a machine model to run on ?

[Kevin Wolf]

Am 19.01.2024 um 13:55 hat Peter Maydell geschrieben:
> If you build QEMU with support for the sh4 target only
> (configure --target-list=sh4-softmmu) then 'make check' fails
> in the iotests, because some iotests, including for instance 040,
> try to create a machine with a virtio-scsi device, but they don't
> do anything to ensure that the machine they create actually
> has a PCI bus to plug the virtio-scsi device into, and so the test
> fails:
> 
> +Traceback (most recent call last):
> +  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/tests/qemu-iotests/040",
> line 94, in setUp
> +    self.vm.launch()
> +  File "/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/python/qemu/machine/machine.py",
> line 455, in launch
> +    raise VMLaunchFailure(
> +qemu.machine.machine.VMLaunchFailure: ConnectError: Failed to
> establish session: EOFError
> +       Exit code: 1
> +       Command:
> /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/sh4/qemu-system-sh4
> -display none -vga none -chardev socket,id=mon,fd=5 -mon
> chardev=mon,mode=control -chardev socket,id=qtest,fd=3 -qtest
> chardev:qtest -accel qtest -nodefaults -display none -accel qtest
> -drive if=none,id=drive0,file=/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/sh4/scratch/qcow2-file-040/test.img,format=qcow2,cache=writeback,aio=threads,node-name=top,backing.node-name=mid,backing.backing.node-name=base
> -device virtio-scsi -device scsi-hd,id=scsi0,drive=drive0
> +       Output: [I 0.000000] OPENED
> +qemu-system-sh4: -device virtio-scsi: No 'PCI' bus found for device
> 'virtio-scsi-pci'
> 
> (It happens that the default machine type for 'sh4'
> does not have a PCI bus.)
> 
> How are the iotests supposed to select a machine model to run
> on, and how are they intended to mark themselves as requiring
> particular facilities, like PCI? Presumably some of the tests
> are missing the appropriate annotations to ensure they're
> skipped when the facilities required aren't present.

Probably next to nobody builds only sh4 and then runs 'make check'...

I think the following patch should fix this test case (but as nobody
ever tested it, I fully expect other cases to fail, too). The fix could
of course be a lot more fine grained and skip only some individual tests
instead of all of them. I doubt we care, though, as long as it passes.

iotests use machine_map in tests/qemu-iotests/testenv.py to decide which
machine to use for which target (unless the individual test case
overrides that). If the target isn't the list, it gets the default
machine type.

Kevin

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index 5c18e413ec..19f0bb1010 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -952,5 +952,6 @@ class TestCommitWithOverriddenBacking(iotests.QMPTestCase):
         self.vm.event_wait('BLOCK_JOB_COMPLETED')
 
 if __name__ == '__main__':
+    iotests.verify_virtio_scsi_pci_or_ccw()
     iotests.main(supported_fmts=['qcow2', 'qed'],
                  supported_protocols=['file'])

Re: how do the iotests pick a machine model to run on ?

[Peter Maydell]

On Fri, 19 Jan 2024 at 14:07, Kevin Wolf <kwolf@redhat.com> wrote:
>
> Am 19.01.2024 um 13:55 hat Peter Maydell geschrieben:
> > How are the iotests supposed to select a machine model to run
> > on, and how are they intended to mark themselves as requiring
> > particular facilities, like PCI? Presumably some of the tests
> > are missing the appropriate annotations to ensure they're
> > skipped when the facilities required aren't present.
>
> Probably next to nobody builds only sh4 and then runs 'make check'...
>
> I think the following patch should fix this test case (but as nobody
> ever tested it, I fully expect other cases to fail, too). The fix could
> of course be a lot more fine grained and skip only some individual tests
> instead of all of them. I doubt we care, though, as long as it passes.

> iotests use machine_map in tests/qemu-iotests/testenv.py to decide which
> machine to use for which target (unless the individual test case
> overrides that). If the target isn't the list, it gets the default
> machine type.

Thanks. Unfortunately your suggested fix doesn't work.

Looking at the definition of  verify_virtio_scsi_pci_or_ccw():

def verify_virtio_scsi_pci_or_ccw() -> None:
    out = qemu_pipe('-M', 'none', '-device', 'help')
    if 'virtio-scsi-pci' not in out and 'virtio-scsi-ccw' not in out:
        notrun('Missing virtio-scsi-pci or virtio-scsi-ccw in QEMU binary')

this doesn't look to me like it's testing the right thing.
You want to know "will one of these devices work on the
machine type I am going to run", not "is this built in to
the binary at all". The -device help output for qemu-system-sh4
does include virtio-scsi-pci because there is a different
non-default machine type that has PCI; but the machine the
040 test actually runs doesn't have PCI.

(Also, we should probably put an entry for sh4 in machine_map,
because the default board type (shix) is about to be deprecated,
and the r2d board type is thus a better choice.)

-- PMM

Re: how do the iotests pick a machine model to run on ?

[Peter Maydell]

On Fri, 19 Jan 2024 at 15:26, Peter Maydell <peter.maydell@linaro.org> wrote:
> (Also, we should probably put an entry for sh4 in machine_map,
> because the default board type (shix) is about to be deprecated,
> and the r2d board type is thus a better choice.)

The good news is if we add r2d to the machine_map, then
only 3 iotests fail:

 191 -- not sure exactly what's going on. QEMU complains
        "machine type does not support if=ide,bus=0,unit=1".
        Side note: the test harness seems to throw away the
        stderr from QEMU with this error message, leaving the
        test failure log rather uninformative. I had to
        run everything under strace to get hold of it.
 203 -- this wants a machine type that can be migrated;
        sh4 CPUs don't support migration, so the test
        fails because the 'migrate' command returns the error
        {"error": {"class": "GenericError", "desc": "State blocked by
non-migratable device 'cpu'"}}
 267 -- similarly, wants a machine that supports snapshots,
        so fails when the loadvm/savevm get the error
        Error: State blocked by non-migratable device 'cpu'

How should a test indicate "I need a machine type that
supports migration" ?

thanks
-- PMM

Re: how do the iotests pick a machine model to run on ?

[Thomas Huth]

On 19/01/2024 17.18, Peter Maydell wrote:
> On Fri, 19 Jan 2024 at 15:26, Peter Maydell <peter.maydell@linaro.org> wrote:
>> (Also, we should probably put an entry for sh4 in machine_map,
>> because the default board type (shix) is about to be deprecated,
>> and the r2d board type is thus a better choice.)
> 
> The good news is if we add r2d to the machine_map, then
> only 3 iotests fail:
> 
>   191 -- not sure exactly what's going on. QEMU complains
>          "machine type does not support if=ide,bus=0,unit=1".
>          Side note: the test harness seems to throw away the
>          stderr from QEMU with this error message, leaving the
>          test failure log rather uninformative. I had to
>          run everything under strace to get hold of it.
>   203 -- this wants a machine type that can be migrated;
>          sh4 CPUs don't support migration, so the test
>          fails because the 'migrate' command returns the error
>          {"error": {"class": "GenericError", "desc": "State blocked by
> non-migratable device 'cpu'"}}
>   267 -- similarly, wants a machine that supports snapshots,
>          so fails when the loadvm/savevm get the error
>          Error: State blocked by non-migratable device 'cpu'
> 
> How should a test indicate "I need a machine type that
> supports migration" ?

We could maybe add a flag to the machine_map to indicate whether the machine 
is capable of migration or not. In the latter case, we could skip all tests 
that are in the "migration" group ?

  Thomas