dbus-display-test is flakey

dbus-display-test is flakey

[Cole Robinson]

Hi Marc-André,

dbus-display-test seems flakey. I'm occasionally seeing:

▶ 692/746
ERROR:../tests/qtest/dbus-display-test.c:68:test_dbus_display_vm:
assertion failed
(qemu_dbus_display1_vm_get_name(QEMU_DBUS_DISPLAY1_VM(vm)) ==
"dbus-test"): (NULL == "dbus-test") ERROR

Examples:

fedora rawhide x86_64:
https://kojipkgs.fedoraproject.org//work/tasks/4945/87834945/build.log
fedora rawhide aarch64:
https://kojipkgs.fedoraproject.org//work/tasks/4946/87834946/build.log
fedora 35 x86_64:
https://download.copr.fedorainfracloud.org/results/@virtmaint-sig/virt-preview/fedora-35-x86_64/04491978-qemu/builder-live.log.gz

This is qemu v7.0.0 with some unrelated patches on top. /usr/bin/make -O
-j5 V=1 VERBOSE=1 check

Side question: I know I can patch meson.build to skip the test, or
similar patch changes, but is there a non-patch way to skip specific tests?

Thanks,
Cole

Re: dbus-display-test is flakey

[Marc-André Lureau]

Hi Cole,


On Sun, Jun 5, 2022 at 6:46 PM Cole Robinson <crobinso@redhat.com> wrote:
>
> Hi Marc-André,
>
> dbus-display-test seems flakey. I'm occasionally seeing:
>
> ▶ 692/746
> ERROR:../tests/qtest/dbus-display-test.c:68:test_dbus_display_vm:
> assertion failed
> (qemu_dbus_display1_vm_get_name(QEMU_DBUS_DISPLAY1_VM(vm)) ==
> "dbus-test"): (NULL == "dbus-test") ERROR
>

Quite an interesting error. The generated dbus proxy code is all
gobject async init, but the proxy is created synchronously with
qemu_dbus_display1_vm_proxy_new_sync(), and all cached properties must
have been set at construction time.

And nobody else reported this issue so far, but I also noticed that it
does not seem to be covered by the CI at this point (only centos has
--enabled-modules, but is missing the glib 2.64 requirements)

> Examples:
>
> fedora rawhide x86_64:
> https://kojipkgs.fedoraproject.org//work/tasks/4945/87834945/build.log
> fedora rawhide aarch64:
> https://kojipkgs.fedoraproject.org//work/tasks/4946/87834946/build.log
> fedora 35 x86_64:
> https://download.copr.fedorainfracloud.org/results/@virtmaint-sig/virt-preview/fedora-35-x86_64/04491978-qemu/builder-live.log.gz
>

Are you able to reproduce outside koji? I have been running the test
in a loop for a long while, not seeing any error.

> This is qemu v7.0.0 with some unrelated patches on top. /usr/bin/make -O
> -j5 V=1 VERBOSE=1 check
>
> Side question: I know I can patch meson.build to skip the test, or
> similar patch changes, but is there a non-patch way to skip specific tests?

"meson test" currently supports specifying the tests or subproject to
run, or using --suite / --no-suite SUITE.

(mtest2make.py creates convenience check-SUITE rules)

We could probably add a suite "dbus", and add the test(s) there, so
you could run "meson test --no-suite dbus".

Re: dbus-display-test is flakey

[Cole Robinson]

On 6/7/22 4:57 PM, Marc-André Lureau wrote:
> Hi Cole,
> 
> 
> On Sun, Jun 5, 2022 at 6:46 PM Cole Robinson <crobinso@redhat.com> wrote:
>>
>> Hi Marc-André,
>>
>> dbus-display-test seems flakey. I'm occasionally seeing:
>>
>> ▶ 692/746
>> ERROR:../tests/qtest/dbus-display-test.c:68:test_dbus_display_vm:
>> assertion failed
>> (qemu_dbus_display1_vm_get_name(QEMU_DBUS_DISPLAY1_VM(vm)) ==
>> "dbus-test"): (NULL == "dbus-test") ERROR
>>
> 
> Quite an interesting error. The generated dbus proxy code is all
> gobject async init, but the proxy is created synchronously with
> qemu_dbus_display1_vm_proxy_new_sync(), and all cached properties must
> have been set at construction time.
> 
> And nobody else reported this issue so far, but I also noticed that it
> does not seem to be covered by the CI at this point (only centos has
> --enabled-modules, but is missing the glib 2.64 requirements)
> 

Interesting. redhat distro family and debian distro family are both
using --enable-modules for their packages, for a while now. Maybe qemu
CI should be using --enable-modules more. I have a patch on the list
that fixes a qemu-iotests failure that's indirectly caused --enable-modules

>> Examples:
>>
>> fedora rawhide x86_64:
>> https://kojipkgs.fedoraproject.org//work/tasks/4945/87834945/build.log
>> fedora rawhide aarch64:
>> https://kojipkgs.fedoraproject.org//work/tasks/4946/87834946/build.log
>> fedora 35 x86_64:
>> https://download.copr.fedorainfracloud.org/results/@virtmaint-sig/virt-preview/fedora-35-x86_64/04491978-qemu/builder-live.log.gz
>>
> 
> Are you able to reproduce outside koji? I have been running the test
> in a loop for a long while, not seeing any error.
> 

I tried a couple local qemu.git x86_64 builds but couldn't reproduce. If
environment is a factor my guess is its race/load dependent, maybe
exacerbated by 'mock' used used by both koji and copr.

>> This is qemu v7.0.0 with some unrelated patches on top. /usr/bin/make -O
>> -j5 V=1 VERBOSE=1 check
>>
>> Side question: I know I can patch meson.build to skip the test, or
>> similar patch changes, but is there a non-patch way to skip specific tests?
> 
> "meson test" currently supports specifying the tests or subproject to
> run, or using --suite / --no-suite SUITE.
> 
> (mtest2make.py creates convenience check-SUITE rules)
> 
> We could probably add a suite "dbus", and add the test(s) there, so
> you could run "meson test --no-suite dbus".
> 

Ah that's good to know. Looks like we can use `MTESTFLAGS='--no-suite
block' make check` to skip iotests which is flaky for other reasons
reported elsewhere.

Adding a suite just for this dbus case doesn't matter to me now. But IMO
it would be useful if there was a way to skip an individual test without
patching code. QEMU_TEST_SKIP="dbus-display-test.c:bar.c" or something.
Just a thought; maybe there's no simple way to wire that in.

Thanks,
Cole