Re: [PATCH v8 00/13] vhost-user: support any POSIX system (tested on macOS, FreeBSD, OpenBSD)

["Michael S. Tsirkin" <mst@redhat.com>]

On Fri, Jul 05, 2024 at 10:39:33AM +0200, Stefano Garzarella wrote:
> On Wed, Jul 03, 2024 at 06:49:30PM GMT, Michael S. Tsirkin wrote:
> > On Tue, Jun 18, 2024 at 12:00:30PM +0200, Stefano Garzarella wrote:
> > > As discussed with Michael and Markus [1], this version also includes the patch
> > > on which v7 depended to simplify the merge in Michael's tree.
> > > 
> > > The series is all reviewed, so if there are no new changes required, I would
> > > ask to merge it.
> > 
> > 
> > I dropped patches 9 and 10 for now since otherwise make vm-build-freebsd
> > fails.
> > 
> > Pls figure it out and resend just 9 and 10.
> 
> I replicated locally, but I can't understand why it only happens in certain
> architectures, in my case on loongarch64, ppc64, and riscv32:
> 
> 326/846 qemu:qtest+qtest-loongarch64 / qtest-loongarch64/qos-test
> ERROR          116.10s   killed by signal 6 SIGABRT
> 337/846 qemu:qtest+qtest-ppc64 / qtest-ppc64/qos-test
> ERROR          115.10s   killed by signal 6 SIGABRT
> 339/846 qemu:qtest+qtest-riscv32 / qtest-riscv32/qos-test
> ERROR          107.65s   killed by signal 6 SIGABRT
> 
> I focused on ppc64 running `gmake --output-sync -j6 check-qtest-ppc64` in
> the FreeBSD VM and it fails every time. In particular, the test that fails
> is the `vhost-user/reconnect` test, in fact disabling it this way, the
> qos-test tests always pass:
> 
> diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
> index 0fa8951c9f..c3d686f0ee 100644
> --- a/tests/qtest/vhost-user-test.c
> +++ b/tests/qtest/vhost-user-test.c
> @@ -1118,9 +1119,11 @@ static void register_vhost_user_test(void)
>                   "virtio-net",
>                   test_migrate, &opts);
> 
> +#if 0
>      opts.before = vhost_user_test_setup_reconnect;
>      qos_add_test("vhost-user/reconnect", "virtio-net",
>                   test_reconnect, &opts);
> +#endif
> 
>      opts.before = vhost_user_test_setup_connect_fail;
>      qos_add_test("vhost-user/connect-fail", "virtio-net",
> 
> Analyzing the test, what happens is that after the disconnection, the test
> doesn't receive VHOST_USER_SET_MEM_TABLE message, so the second
> `wait_for_fds()` fails after the 5 sec timeout (increasing it doesn't help),
> not having received the fds.
> 
> diff --git a/tests/qtest/vhost-user-test.c b/tests/qtest/vhost-user-test.c
> index 0fa8951c9f..c3d686f0ee 100644
> --- a/tests/qtest/vhost-user-test.c
> +++ b/tests/qtest/vhost-user-test.c
> @@ -976,6 +976,7 @@ static void test_reconnect(void *obj, void *arg,
> QGuestAllocator *alloc)
>      g_source_set_callback(src, reconnect_cb, s, NULL);
>      g_source_attach(src, s->context);
>      g_source_unref(src);
> +    // THIS one is failing
>      g_assert(wait_for_fds(s));
>      wait_for_rings_started(s, 2);
>  }
> 
> This is the test log (note: IIUC QEMU failures happen after the test exits
> on the assertion, so so it could mean that the chardev reconnected
> correctly):
> 
>     ▶ 28/30 /ppc64/pseries/spapr-pci-host-bridge/pci-bus-spapr/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/reconnect - ERROR:../src/tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/ppc64/pseries/spapr-pci-host-bridge/pci-bus-spapr/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/reconnect/subprocess [54991]) failed unexpectedly FAIL
>     ▶ 28/30                                                                   ERROR
>     [28-30/30] 🌒 qemu:qtest+qtest-ppc64 / qtest-ppc64/qmp-cmd-test                       [28-30/30] 🌓 qemu:qtest+qtest-ppc64 / qtest-ppc64/migration-test                     28/30 qemu:qtest+qtest-ppc64 / qtest-ppc64/qos-test                       ERROR           21.53s   killed by signal 6 SIGABRT
>     >>> PYTHON=/usr/home/qemu/qemu-test.OD8v2L/build/pyvenv/bin/python3.9 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 G_TEST_DBUS_DAEMON=/usr/home/qemu/qemu-test.OD8v2L/src/tests/dbus-vmstate-daemon.sh QTEST_QEMU_BINARY=./qemu-system-ppc64 MALLOC_PERTURB_=141 QTEST_QEMU_IMG=./qemu-img QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /usr/home/qemu/qemu-test.OD8v2L/build/tests/qtest/qos-test --tap -k
>     ―――――――――――――――――――――――――――――――――――――――― ✀  ――――――――――――――――――――――――――――――――――――――――
>     stderr:
>     Vhost user backend fails to broadcast fake RARP
>     qemu-system-ppc64: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-Z5VMQ2/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-Z5VMQ2/reconnect.sock,server=on
>     **
>     ERROR:../src/tests/qtest/vhost-user-test.c:255:wait_for_fds: assertion failed: (s->fds_num)
>     qemu-system-ppc64: Failed to set msg fds.
>     qemu-system-ppc64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>     qemu-system-ppc64: Failed to set msg fds.
>     qemu-system-ppc64: vhost_set_vring_endian failed: Invalid argument (22)
>     qemu-system-ppc64: Failed to set msg fds.
>     qemu-system-ppc64: vhost VQ 1 ring restore failed: -22: Invalid argument
> (22)
>     qemu-system-ppc64: Failed to set msg fds.
>     qemu-system-ppc64: vhost_set_vring_endian failed: Invalid argument (22)
>     qemu-system-ppc64: Failed to write msg. Wrote -1 instead of 12.
>     qemu-system-ppc64: vhost_backend_init failed: Protocol error
>     qemu-system-ppc64: failed to init vhost_net for queue 0
>     **
>     ERROR:../src/tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/ppc64/pseries/spapr-pci-host-bridge/pci-bus-spapr/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/reconnect/subprocess [54991]) failed unexpectedly
>     (test program exited with status code -6)
> 
> 
> I would think of some endianness problem, but it's strange that it only
> happens in the reconnect test.

loongarch64 is LE and I think so is riscv in practice.

> Next week I'll try to figure out why this is
> systematic only on some architectures, does anyone have any ideas?
> 
> Thanks,
> Stefano