Re: [PATCH] tests: improve performance of device-introspect-test

[Thomas Huth <thuth@redhat.com>]

On 10/07/2020 22.03, Markus Armbruster wrote:
[...]
>   With -m slow, we test 2 * #machines * #devices introspections,
>   i.e. from 132 (tricore) to over 10k (ppc 13046, ppc64 23426, arm
>   82708, aarch64 89760).  Median is ~1600, sum is ~260k.
> 
>   Except we actually test just 89k now, because the test *fails* for arm
>   and aarch64 after some 500 introspections: introspecting device
>   msf2-soc with machine ast2600-evb makes QEMU terminate unsuccessfully
>   with "Unsupported NIC model: ftgmac100".  Cause: m2sxxx_soc_initfn()
>   calls qemu_check_nic_model().  Goes back to commit 05b7374a58 "msf2:
>   Add EMAC block to SmartFusion2 SoC", merged some ten weeks ago.  This
>   is exactly the kind of mistake the test is designed to catch.  How
>   come it wasn't?  Possibly due to unlucky combination with the slowdown
>   discussed in the next item (but that was less than four weeks ago).

Well, the explanation is simpler: Nobody ran the
device-introspection-test with the arm target in slow mode! Hardly
anybody runs the tests with SPEED=slow, and in the CI, we currently only
run the test in slow mode for the i386-softmmu, ppc64-softmmu and
mips64-softmmu targets. Simply because testing with arm was toooo slow
when I wrote that CI script, I didn't want to wait endlessly here.

But now with the speedup patch from Daniel, and maybe with some smarter
checks in the YML file (I now know that there are things like "only:
changes:" keywords so we could e.g. only run that test if something in
hw/arm/* has changed), I think it should be feasible to get this
included in the CI, too.

 Thomas