migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Peter Maydell]

Hi; I see this migration qtest failure on my x86 macos box:


▶  32/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
assertion fail
ed: (!g_str_equal(status, "failed")) ERROR
 32/591 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
           ERROR          152.27s   killed by signal 6 SIGABRT
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
query-migrate shows failed migration: Unable to write to socket: Broken pipe
**
ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
assertion failed: (!g_str_equal(status, "failed"))
Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
../../tests/qtest/libqtest.c, line 207.

(test program exited with status code -6)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

and similarly:

▶  34/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
assertion failed: (!g_str_equal(status, "failed")) ERROR
 34/591 qemu:qtest+qtest-i386 / qtest-i386/migration-test
           ERROR          169.44s   killed by signal 6 SIGABRT
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
query-migrate shows failed migration: Unable to write to socket: Broken pipe
**
ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
assertion failed: (!g_str_equal(status, "failed"))
Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
../../tests/qtest/libqtest.c, line 207.

(test program exited with status code -6)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

It seems to be fairly consistent. Any ideas what it might be?
Maybe the QEMU process has already exited before the test binary
gets round to querying the status ?

thanks
-- PMM

Re: migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> Hi; I see this migration qtest failure on my x86 macos box:
> 
> 
> ▶  32/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> assertion fail
> ed: (!g_str_equal(status, "failed")) ERROR
>  32/591 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
>            ERROR          152.27s   killed by signal 6 SIGABRT
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> query-migrate shows failed migration: Unable to write to socket: Broken pipe
> **
> ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> assertion failed: (!g_str_equal(status, "failed"))
> Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> ../../tests/qtest/libqtest.c, line 207.
> 
> (test program exited with status code -6)
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> 
> and similarly:
> 
> ▶  34/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> assertion failed: (!g_str_equal(status, "failed")) ERROR
>  34/591 qemu:qtest+qtest-i386 / qtest-i386/migration-test
>            ERROR          169.44s   killed by signal 6 SIGABRT
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> query-migrate shows failed migration: Unable to write to socket: Broken pipe
> **
> ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> assertion failed: (!g_str_equal(status, "failed"))
> Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> ../../tests/qtest/libqtest.c, line 207.
> 
> (test program exited with status code -6)
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> 
> It seems to be fairly consistent. Any ideas what it might be?
> Maybe the QEMU process has already exited before the test binary
> gets round to querying the status ?

Yes, it sounds like it, can you get a backtrace to figure out which test
it was in/where it was upto when it died?

Dave

> thanks
> -- PMM
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Peter Maydell]

On Thu, 15 Dec 2022 at 11:40, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
>
> * Peter Maydell (peter.maydell@linaro.org) wrote:
> > Hi; I see this migration qtest failure on my x86 macos box:
> >
> >
> > ▶  32/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > assertion fail
> > ed: (!g_str_equal(status, "failed")) ERROR
> >  32/591 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
> >            ERROR          152.27s   killed by signal 6 SIGABRT
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > query-migrate shows failed migration: Unable to write to socket: Broken pipe
> > **
> > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > assertion failed: (!g_str_equal(status, "failed"))
> > Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> > ../../tests/qtest/libqtest.c, line 207.
> >
> > (test program exited with status code -6)
> > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> >
> > and similarly:
> >
> > ▶  34/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > assertion failed: (!g_str_equal(status, "failed")) ERROR
> >  34/591 qemu:qtest+qtest-i386 / qtest-i386/migration-test
> >            ERROR          169.44s   killed by signal 6 SIGABRT
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > query-migrate shows failed migration: Unable to write to socket: Broken pipe
> > **
> > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > assertion failed: (!g_str_equal(status, "failed"))
> > Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> > ../../tests/qtest/libqtest.c, line 207.
> >
> > (test program exited with status code -6)
> > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> >
> > It seems to be fairly consistent. Any ideas what it might be?
> > Maybe the QEMU process has already exited before the test binary
> > gets round to querying the status ?
>
> Yes, it sounds like it, can you get a backtrace to figure out which test
> it was in/where it was upto when it died?

The logfile says it had just done
ok 23 /aarch64/migration/multifd/tcp/plain/none
so I think the one it was in the middle of when it failed was
/aarch64/migration/multifd/tcp/plain/cancel.
Similarly the log suggests the x86 failure was for
/i386/migration/miltifd/tcp/plain/cancel.

It doesn't seem to repro running manually, my guess is that
it happens because the machine is heavily loaded doing the
whole build-and-test cycle.

thanks
-- PMM

Re: migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On Thu, 15 Dec 2022 at 11:40, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >
> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> > > Hi; I see this migration qtest failure on my x86 macos box:
> > >
> > >
> > > ▶  32/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion fail
> > > ed: (!g_str_equal(status, "failed")) ERROR
> > >  32/591 qemu:qtest+qtest-aarch64 / qtest-aarch64/migration-test
> > >            ERROR          152.27s   killed by signal 6 SIGABRT
> > > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > > stderr:
> > > query-migrate shows failed migration: Unable to write to socket: Broken pipe
> > > **
> > > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion failed: (!g_str_equal(status, "failed"))
> > > Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> > > ../../tests/qtest/libqtest.c, line 207.
> > >
> > > (test program exited with status code -6)
> > > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > >
> > > and similarly:
> > >
> > > ▶  34/591 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion failed: (!g_str_equal(status, "failed")) ERROR
> > >  34/591 qemu:qtest+qtest-i386 / qtest-i386/migration-test
> > >            ERROR          169.44s   killed by signal 6 SIGABRT
> > > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > > stderr:
> > > query-migrate shows failed migration: Unable to write to socket: Broken pipe
> > > **
> > > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
> > > assertion failed: (!g_str_equal(status, "failed"))
> > > Assertion failed: (pid == s->qemu_pid), function qtest_wait_qemu, file
> > > ../../tests/qtest/libqtest.c, line 207.
> > >
> > > (test program exited with status code -6)
> > > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > >
> > > It seems to be fairly consistent. Any ideas what it might be?
> > > Maybe the QEMU process has already exited before the test binary
> > > gets round to querying the status ?
> >
> > Yes, it sounds like it, can you get a backtrace to figure out which test
> > it was in/where it was upto when it died?
> 
> The logfile says it had just done
> ok 23 /aarch64/migration/multifd/tcp/plain/none
> so I think the one it was in the middle of when it failed was
> /aarch64/migration/multifd/tcp/plain/cancel.
> Similarly the log suggests the x86 failure was for
> /i386/migration/miltifd/tcp/plain/cancel.
> 
> It doesn't seem to repro running manually, my guess is that
> it happens because the machine is heavily loaded doing the
> whole build-and-test cycle.

Yeh; I think we'll still need a backtrace or better qmp log though to
figure it out.
If I read that correctly, what that test does is:

   start 'from'
   start 'to'
   slow migrate from->to
   <cancel>
   start 'to2'
   migrate from->to2

I'd only expect the 'to' to quit by itself, and I don't think we should
be doing a 'failed' check on a destination (I don't think).
Even if the migration finished quickly 'from' shouldn't just quit - so
the QMP should still be working, so we shouldn't get the broken pipe on
the actual qmp pipe.
Assuming then it's a broken socket on the migration stream, that sounds
like a bug - a 'cancel' might cause the destination to get an error
but the source shouldn't - it should know it's cancelling.

Dave

> thanks
> -- PMM
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Peter Maydell]

On Thu, 15 Dec 2022 at 12:52, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
>
> * Peter Maydell (peter.maydell@linaro.org) wrote:
> > It doesn't seem to repro running manually, my guess is that
> > it happens because the machine is heavily loaded doing the
> > whole build-and-test cycle.
>
> Yeh; I think we'll still need a backtrace or better qmp log though to
> figure it out.

Unfortunately, often all you get is "what does 'make check' output".
Is there a way we can improve the test so it outputs something
more useful when it detects a failure ?

thanks
-- PMM

Re: migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Dr. David Alan Gilbert]

* Peter Maydell (peter.maydell@linaro.org) wrote:
> On Thu, 15 Dec 2022 at 12:52, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> >
> > * Peter Maydell (peter.maydell@linaro.org) wrote:
> > > It doesn't seem to repro running manually, my guess is that
> > > it happens because the machine is heavily loaded doing the
> > > whole build-and-test cycle.
> >
> > Yeh; I think we'll still need a backtrace or better qmp log though to
> > figure it out.
> 
> Unfortunately, often all you get is "what does 'make check' output".
> Is there a way we can improve the test so it outputs something
> more useful when it detects a failure ?

I can't think how to improve the test itself like that; could you bind
assertion failures to call/print a backtrace:

https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/backtrace.3.html

?

Dave

> thanks
> -- PMM
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: migration qtest failure: "query-migrate shows failed migration: Unable to write to socket: Broken pipe"

[Peter Maydell]

On Thu, 15 Dec 2022 at 11:17, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> Hi; I see this migration qtest failure on my x86 macos box:

Same assertion failure, this time on the OpenBSD VM, with
a different error string:


 34/616 qemu:qtest+qtest-i386 / qtest-i386/migration-test
           ERROR
      139.66s   killed by signal 6 SIGABRT
――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
qemu: thread naming not supported on this host
query-migrate shows failed migration: Failed to connect to
'127.0.0.1:4562': Address alread
y in use
**
ERROR:../src/tests/qtest/migration-helpers.c:151:migrate_query_not_failed:
assertion failed
: (!g_str_equal(status, "failed"))
assertion "pid == s->qemu_pid" failed: file
"../src/tests/qtest/libqtest.c", line 207, func
tion "qtest_wait_qemu"

(test program exited with status code -6)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――


thanks
-- PMM