migration-test random intermittent failure, openbsd VM

migration-test random intermittent failure, openbsd VM

[Peter Maydell]

Random intermittent in migration-test when running the tests
in the openbsd VM (i.e. what you get from 'make -C build vm-build-openbsd')
Any ideas?

106/847 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test
           ERROR          157.51s   killed by signal 6 SIGABRT
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
qemu-system-x86_64: multifd_send_sync_main: channel 12 has already quit
qemu-system-x86_64: TLS handshake failed: The TLS connection was
non-properly terminated.
qemu-system-x86_64: TLS handshake failed: Error in the push function.
qemu-system-x86_64: Failed to connect to '127.0.0.1:23083': Address
already in use
qemu-system-x86_64: TLS handshake failed: Error in the push function.
qemu-system-x86_64: TLS handshake failed: Error in the push function.
qemu-system-x86_64: TLS handshake failed: Error in the push function.
qemu-system-x86_64: TLS handshake failed: Error in the push function.
**
ERROR:../src/tests/qtest/migration-helpers.c:180:check_migration_status:
assertion failed (current_status != "failed"): ("failed" != "failed")
qemu-system-x86_64: TLS handshake failed: Error in the push function.

(test program exited with status code -6)


thanks
-- PMM

Re: migration-test random intermittent failure, openbsd VM

[Fabiano Rosas]

Peter Maydell <peter.maydell@linaro.org> writes:

+cc Daniel.

> Random intermittent in migration-test when running the tests
> in the openbsd VM (i.e. what you get from 'make -C build vm-build-openbsd')
> Any ideas?

Where's your HEAD at?

>
> 106/847 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test
>            ERROR          157.51s   killed by signal 6 SIGABRT
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> qemu-system-x86_64: multifd_send_sync_main: channel 12 has already quit
> qemu-system-x86_64: TLS handshake failed: The TLS connection was
> non-properly terminated.
> qemu-system-x86_64: TLS handshake failed: Error in the push function.
> qemu-system-x86_64: Failed to connect to '127.0.0.1:23083': Address
> already in use

Looks like a legitimate port clash here when doing the TLS
connection. It seems the test is not prepared to deal with it.

> qemu-system-x86_64: TLS handshake failed: Error in the push function.
> qemu-system-x86_64: TLS handshake failed: Error in the push function.
> qemu-system-x86_64: TLS handshake failed: Error in the push function.
> qemu-system-x86_64: TLS handshake failed: Error in the push function.
> **
> ERROR:../src/tests/qtest/migration-helpers.c:180:check_migration_status:
> assertion failed (current_status != "failed"): ("failed" != "failed")
> qemu-system-x86_64: TLS handshake failed: Error in the push function.
>
> (test program exited with status code -6)
>
>
> thanks
> -- PMM

Re: migration-test random intermittent failure, openbsd VM

[Daniel P. Berrangé]

On Thu, Feb 08, 2024 at 03:04:23PM -0300, Fabiano Rosas wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> +cc Daniel.
> 
> > Random intermittent in migration-test when running the tests
> > in the openbsd VM (i.e. what you get from 'make -C build vm-build-openbsd')
> > Any ideas?
> 
> Where's your HEAD at?
> 
> >
> > 106/847 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test
> >            ERROR          157.51s   killed by signal 6 SIGABRT
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > qemu-system-x86_64: multifd_send_sync_main: channel 12 has already quit
> > qemu-system-x86_64: TLS handshake failed: The TLS connection was
> > non-properly terminated.
> > qemu-system-x86_64: TLS handshake failed: Error in the push function.
> > qemu-system-x86_64: Failed to connect to '127.0.0.1:23083': Address
> > already in use
> 
> Looks like a legitimate port clash here when doing the TLS
> connection. It seems the test is not prepared to deal with it.

The migration-test.c code always says to listen on ':0' so the
kernel dynamically allocates an address, we when query that
port and connect to it.  So we should never fail on listen/bind
server side.

The error message though is about the client side as it says
"failed to connect".

I've no idea how on earth you trigger "Address already in use" as
a client !

Every outbound connection does require a port, but QEMU always
leave the port on 0, so the kenrel dynamically allocates an
outbound port. Somehow this is failing, but I can't see how
it is QEMU's fault ?!?!?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: migration-test random intermittent failure, openbsd VM

[Peter Maydell]

On Thu, 8 Feb 2024 at 18:04, Fabiano Rosas <farosas@suse.de> wrote:
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> +cc Daniel.
>
> > Random intermittent in migration-test when running the tests
> > in the openbsd VM (i.e. what you get from 'make -C build vm-build-openbsd')
> > Any ideas?
>
> Where's your HEAD at?

03e4bc0bc02 in this case.

thanks
-- PMM

Re: migration-test random intermittent failure, openbsd VM

[Fabiano Rosas]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 8 Feb 2024 at 18:04, Fabiano Rosas <farosas@suse.de> wrote:
>>
>> Peter Maydell <peter.maydell@linaro.org> writes:
>>
>> +cc Daniel.
>>
>> > Random intermittent in migration-test when running the tests
>> > in the openbsd VM (i.e. what you get from 'make -C build vm-build-openbsd')
>> > Any ideas?
>>
>> Where's your HEAD at?
>
> 03e4bc0bc02 in this case.
>
> thanks
> -- PMM

I spent some time on this today, it reproduces inconsistently and I
couldn't find much.

The only thing I noticed so far is that when the tests fail, netstat
shows several ports at TIME_WAIT TCP state, including the one that just
got rejected. Maybe some issue with how we're closing these sockets
vs. the openbsd network stack implementation. But that's a long shot
without more investigation.