Re: [PULL v2 31/82] vhost: Change the sequence of device start

["Michael S. Tsirkin" <mst@redhat.com>]

On Sat, Nov 05, 2022 at 05:35:57PM +0100, Bernhard Beschow wrote:
> 
> 
> On Wed, Nov 2, 2022 at 5:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> 
>     From: Yajun Wu <yajunw@nvidia.com>
> 
>     This patch is part of adding vhost-user vhost_dev_start support. The
>     motivation is to improve backend configuration speed and reduce live
>     migration VM downtime.
> 
>     Moving the device start routines after finishing all the necessary device
>     and VQ configuration, further aligning to the virtio specification for
>     "device initialization sequence".
> 
>     Following patch will add vhost-user vhost_dev_start support.
> 
>     Signed-off-by: Yajun Wu <yajunw@nvidia.com>
>     Acked-by: Parav Pandit <parav@nvidia.com>
> 
>     Message-Id: <20221017064452.1226514-2-yajunw@nvidia.com>
>     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>     ---
>      hw/block/vhost-user-blk.c | 18 +++++++++++-------
>      hw/net/vhost_net.c        | 12 ++++++------
>      2 files changed, 17 insertions(+), 13 deletions(-)
> 
> 
> A git bisect tells me that this is the first bad commit for failing qos-tests
> which only fail when parallel jobs are enabled, e.g. `make check-qtest -j8`:
> 
> Summary of Failures:
> 
>  76/541 qemu:qtest+qtest-aarch64 / qtest-aarch64/qos-test                      
>   ERROR          18.68s   killed by signal 6 SIGABRT
>  77/541 qemu:qtest+qtest-arm / qtest-arm/qos-test                              
>   ERROR          17.60s   killed by signal 6 SIGABRT
>  93/541 qemu:qtest+qtest-i386 / qtest-i386/qos-test                            
>   ERROR          18.98s   killed by signal 6 SIGABRT
> 108/541 qemu:qtest+qtest-ppc64 / qtest-ppc64/qos-test                          
>   ERROR          16.40s   killed by signal 6 SIGABRT
> 112/541 qemu:qtest+qtest-i386 / qtest-i386/bios-tables-test                    
>   ERROR          145.94s   killed by signal 6 SIGABRT
> 130/541 qemu:qtest+qtest-x86_64 / qtest-x86_64/qos-test                        
>   ERROR          17.32s   killed by signal 6 SIGABRT
> 243/541 qemu:qtest+qtest-x86_64 / qtest-x86_64/bios-tables-test                
>   ERROR          127.70s   killed by signal 6 SIGABRT
> 
> Ok:                 500
> Expected Fail:      0  
> Fail:               7  
> Unexpected Pass:    0  
> Skipped:            34  
> Timeout:            0  
> 
> Can anyone else reproduce this?

Could you pls try latest for_upstream in my tree?
That should have this fixed.

Thanks!


> Here is a log of one failed case:
> 
> # child process (/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/
> virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> [384604]) killed by signal 6 (Aborted), core dumped
> # child process (/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/
> virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> [384604]) stdout: ""
> # child process (/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/
> virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> [384604]) stderr: "qemu-system-aarch64: -chardev socket,id=
> chr-flags-mismatch,path=/tmp/vhost-test-BBEAV1/flags-mismatch.sock,server=on:
> info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-BBEAV1/
> flags-mismatch.sock,server=on\nqemu-system-aarch64: Failed to write msg. Wrote
> -1 instead of 52.\nqemu-system-aarch64: vhost_set_mem_table failed: Invalid
> argument (22)\nqemu-system-aarch64: Failed to set msg fds.\
> nqemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument
> (22)\nBroken pipe\n../src/tests/qtest/libqtest.c:188: kill_qemu() detected QEMU
> death from signal 11 (Segmentation fault) (core dumped)\n"
> Bail out! ERROR:../src/tests/qtest/qos-test.c:191:subprocess_run_one_test:
> child process (/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/
> virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
> [384604]) failed unexpectedly
> ----------------------------------- stderr -----------------------------------
> qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-aarch64: Failed to set msg fds.
> qemu-system-aarch64: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-aarch64: -chardev socket,id=chr-reconnect,path=/tmp/
> vhost-test-FTJ0U1/reconnect.sock,server=on: info: QEMU waiting for connection
> on: disconnected:unix:/tmp/vhost-test-FTJ0U1/reconnect.sock,server=on
> qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-aarch64: Failed to set msg fds.
> qemu-system-aarch64: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-aarch64: -chardev socket,id=chr-connect-fail,path=/tmp/
> vhost-test-4W4DV1/connect-fail.sock,server=on: info: QEMU waiting for
> connection on: disconnected:unix:/tmp/vhost-test-4W4DV1/
> connect-fail.sock,server=on
> qemu-system-aarch64: -netdev vhost-user,id=hs0,chardev=
> chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of
> 12. Original request 1.
> qemu-system-aarch64: -netdev vhost-user,id=hs0,chardev=
> chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
> qemu-system-aarch64: -netdev vhost-user,id=hs0,chardev=
> chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
> qemu-system-aarch64: -netdev vhost-user,id=hs0,chardev=
> chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on:
> disconnected:unix:/tmp/vhost-test-4W4DV1/connect-fail.sock,server=on
> qemu-system-aarch64: Failed to set msg fds.
> qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-aarch64: Failed to set msg fds.
> qemu-system-aarch64: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-aarch64: -chardev socket,id=chr-flags-mismatch,path=/tmp/
> vhost-test-BBEAV1/flags-mismatch.sock,server=on: info: QEMU waiting for
> connection on: disconnected:unix:/tmp/vhost-test-BBEAV1/
> flags-mismatch.sock,server=on
> qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 52.
> qemu-system-aarch64: vhost_set_mem_table failed: Invalid argument (22)
> qemu-system-aarch64: Failed to set msg fds.
> qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> Broken pipe
> ../src/tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal
> 11 (Segmentation fault) (core dumped)
> **
> ERROR:../src/tests/qtest/qos-test.c:191:subprocess_run_one_test: child process
> (/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/virtio-net-pci/
> virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess [384604])
> failed unexpectedly
> 
> (test program exited with status code -6)
> ==============================================================================
> 
> =================================== 77/541 ===================================
> test:         qemu:qtest+qtest-arm / qtest-arm/qos-test
> start time:   16:04:10
> duration:     17.60s
> result:       killed by signal 6 SIGABRT
> command:      MALLOC_PERTURB_=98 QTEST_QEMU_IMG=./qemu-img
> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
> QTEST_QEMU_BINARY=./qemu-system-arm G_TEST_DBUS_DAEMON=../src/tests/
> dbus-vmstate-daemon.sh ./tests/qtest/qos-test --tap -k
> 
> Best regards,
> Bernhard