Re: [Qemu-devel] test-filter-mirror hangs

["Dr. David Alan Gilbert" <dgilbert@redhat.com>]

* Jason Wang (jasowang@redhat.com) wrote:
> 
> On 2019/1/22 上午2:56, Peter Maydell wrote:
> > On Thu, 17 Jan 2019 at 09:46, Jason Wang <jasowang@redhat.com> wrote:
> > > 
> > > On 2019/1/15 上午12:33, Zhang Chen wrote:
> > > > 
> > > > On Sat, Jan 12, 2019 at 12:15 AM Dr. David Alan Gilbert
> > > > <dgilbert@redhat.com <mailto:dgilbert@redhat.com>> wrote:
> > > > 
> > > >      * Peter Maydell (peter.maydell@linaro.org
> > > >      <mailto:peter.maydell@linaro.org>) wrote:
> > > >      > Recently I've noticed that test-filter-mirror has been hanging
> > > >      > intermittently, typically when run on some other TCG architecture.
> > > >      > In the instance I've just looked at, this was with s390x guest on
> > > >      > x86-64 host, though I've also seen it on other host archs and
> > > >      > perhaps with other guests.
> > > > 
> > > >      Watch out to see if you really do see it for other guests;
> > > >      it carefully avoids using virtio-net to avoid vhost; but on s390x it
> > > >      uses virtio-net-ccw - could that hit the vhost it was trying to avoid?
> > > > 
> > > >      > Below is a backtrace, though it seems to be pretty unhelpful.
> > > >      > Anybody got any theories ? Does the mirror test rely on dirty
> > > >      > memory bitmaps like the migration test (which also hangs
> > > >      > occasionally with TCG due to some bug I'm sure we've investigated
> > > >      > in the past) ?
> > > > 
> > > >      I don't think it relies on the CPU at all.
> > > >   I have no idea about this currently, but Jason and I designed the
> > > > test case.
> > > > Add Jason: Have any comments about this ?
> > > 
> > > I can't reproduce this locally with s390x-softmmu. It looks to me the
> > > test should be independent to any kinds of emulation. It should pass
> > > when mainloop work.
> > I've just seen a hang with ppc64 guest on s390x host, so it is
> > indeed not specific to s390x guest (and so not specific to
> > virtio-net either, since the ppc64 guest setup uses e1000).
> > 
> > thanks
> > -- PMM
> 
> 
> Finally reproduced locally after hundreds (sometimes thousands) times of
> running.
> 
> Bisection points to OOB monitor[1].
> 
> It looks to me after OOB is used unconditionally we lose a barrier to make
> sure socket is connected before sending packets in test-filter-mirror.c. Is
> there any other similar and simple thing that we could do to kick the
> mainloop?

Do you mean the:

    /* send a qmp command to guarantee that 'connected' is setting to true. */
    qmp_discard_response(qts, "{ 'execute' : 'query-status'}");

why was that ever sufficient to know the socket was ready?

Dave

> Thanks
> 
> [1]
> 
> commit 8258292e18c39480b64eba9f3551ab772ce29b5d (HEAD, refs/bisect/bad)
> Author: Peter Xu <peterx@redhat.com>
> Date:   Tue Oct 9 14:27:15 2018 +0800
> 
>     monitor: Remove "x-oob", offer capability "oob" unconditionally
> 
>     Out-of-band command execution was introduced in commit cf869d53172.
>     Unfortunately, we ran into a regression, and had to turn it into an
>     experimental option for 2.12 (commit be933ffc23).
> 
> http://lists.gnu.org/archive/html/qemu-devel/2018-03/msg06231.html
> 
>     The regression has since been fixed (commit 951702f39c7 "monitor: bind
>     dispatch bh to iohandler context").  A thorough re-review of OOB
>     commands led to a few more issues, which have also been addressed.
> 
>     This patch partly reverts be933ffc23 (monitor: new parameter "x-oob"),
>     and makes QMP monitors again offer capability "oob" whenever they can
>     provide it, i.e. when the monitor's character device is capable of
>     running in an I/O thread.
> 
>     Some trivial touch-up in the test code is required to make sure qmp-test
>     won't break.
> 
>     Reviewed-by: Markus Armbruster <armbru@redhat.com>
>     Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
>     Signed-off-by: Peter Xu <peterx@redhat.com>
>     Message-Id: <20181009062718.1914-4-peterx@redhat.com>
>     [Conflict with "monitor: check if chardev can switch gcontext for OOB"
>     resolved, commit message updated]
>     Signed-off-by: Markus Armbruster <armbru@redhat.com>
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK