[Qemu-devel] Hang in aio/multi/mutex/mcs

[Qemu-devel] Hang in aio/multi/mutex/mcs

[Stefan Hajnoczi]

Hi Paolo,
Alex Benne reported that Travis CI is hanging in aio/multi/mutex/mcs:

https://travis-ci.org/qemu/qemu/builds/203811867

0x00007fe9d680052d in nanosleep () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
(gdb) info threads
  Id   Target Id         Frame
* 1    LWP 15116 "test-aio-multit" 0x00007fe9d680052d in nanosleep ()
from target:/lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt
#0  0x00007fe9d680052d in nanosleep () from
target:/lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fe9d74c6072 in g_usleep () from
target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2  0x000055b90b2e9dca in test_multi_fair_mutex (seconds=1, threads=5)
at /tmp/qemu-test/src/tests/test-aio-multithread.c:363
#3  0x00007fe9d74c365b in ?? () from
target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
#4  0x00007fe9d74c37d6 in ?? () from
target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
#5  0x00007fe9d74c37d6 in ?? () from
target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
#6  0x00007fe9d74c37d6 in ?? () from
target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
#7  0x00007fe9d74c3b2b in g_test_run_suite () from
target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
#8  0x000055b90b2e758c in main (argc=1, argv=0x7ffd65cb4928) at
/tmp/qemu-test/src/tests/test-aio-multithread.c:462

Stefan

Re: [Qemu-devel] Hang in aio/multi/mutex/mcs

[Alex Bennée]

Stefan Hajnoczi <stefanha@gmail.com> writes:

> Hi Paolo,
> Alex Benne reported that Travis CI is hanging in aio/multi/mutex/mcs:
>
> https://travis-ci.org/qemu/qemu/builds/203811867
>
> 0x00007fe9d680052d in nanosleep () from
> target:/lib/x86_64-linux-gnu/libpthread.so.0
> (gdb) info threads
>   Id   Target Id         Frame
> * 1    LWP 15116 "test-aio-multit" 0x00007fe9d680052d in nanosleep ()
> from target:/lib/x86_64-linux-gnu/libpthread.so.0
> (gdb) bt
> #0  0x00007fe9d680052d in nanosleep () from
> target:/lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007fe9d74c6072 in g_usleep () from
> target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
> #2  0x000055b90b2e9dca in test_multi_fair_mutex (seconds=1, threads=5)
> at /tmp/qemu-test/src/tests/test-aio-multithread.c:363
> #3  0x00007fe9d74c365b in ?? () from
> target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
> #4  0x00007fe9d74c37d6 in ?? () from
> target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
> #5  0x00007fe9d74c37d6 in ?? () from
> target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
> #6  0x00007fe9d74c37d6 in ?? () from
> target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
> #7  0x00007fe9d74c3b2b in g_test_run_suite () from
> target:/lib/x86_64-linux-gnu/libglib-2.0.so.0
> #8  0x000055b90b2e758c in main (argc=1, argv=0x7ffd65cb4928) at
> /tmp/qemu-test/src/tests/test-aio-multithread.c:462

To reproduce:

  make docker-test-quick@travis J=4 DEBUG=1
  /tmp/qemu-test/src/tests/docker/test-quick

This usually hangs for me. In another winder you can run:

  docker exec -it $CONTAINER_NAME bash

To get access to the filesystem and see the hung tasks.

If it doesn't trigger during this stage you may need to:

  cd /var/tmp/qemu-build
  ./tests/test-aio-multithread

Hope that helps the debugging ;-)

--
Alex Bennée

Re: [Qemu-devel] Hang in aio/multi/mutex/mcs

[Paolo Bonzini]

On 23/02/2017 20:48, Alex Bennée wrote:
> Hope that helps the debugging ;-)

Worst case we can just remove the test (it's only there for performance
comparison, not a bug in actual QEMU code), but this seems to help here:

diff --git a/tests/test-aio-multithread.c b/tests/test-aio-multithread.c
index f11e990..8b0b40e 100644
--- a/tests/test-aio-multithread.c
+++ b/tests/test-aio-multithread.c
@@ -309,7 +309,7 @@ static void mcs_mutex_lock(void)
 static void mcs_mutex_unlock(void)
 {
     int next;
-    if (nodes[id].next == -1) {
+    if (atomic_read(&nodes[id].next) == -1) {
         if (atomic_read(&mutex_head) == id &&
             atomic_cmpxchg(&mutex_head, id, -1) == id) {
             /* Last item in the list, exit.  */
@@ -323,7 +323,7 @@ static void mcs_mutex_unlock(void)
     }

     /* Wake up the next in line.  */
-    next = nodes[id].next;
+    next = atomic_read(&nodes[id].next);
     nodes[next].locked = 0;
     qemu_futex_wake(&nodes[next].locked, 1);
 }

Paolo

Re: [Qemu-devel] Hang in aio/multi/mutex/mcs

[Alex Bennée]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 23/02/2017 20:48, Alex Bennée wrote:
>> Hope that helps the debugging ;-)
>
> Worst case we can just remove the test (it's only there for performance
> comparison, not a bug in actual QEMU code), but this seems to help here:
>
> diff --git a/tests/test-aio-multithread.c b/tests/test-aio-multithread.c
> index f11e990..8b0b40e 100644
> --- a/tests/test-aio-multithread.c
> +++ b/tests/test-aio-multithread.c
> @@ -309,7 +309,7 @@ static void mcs_mutex_lock(void)
>  static void mcs_mutex_unlock(void)
>  {
>      int next;
> -    if (nodes[id].next == -1) {
> +    if (atomic_read(&nodes[id].next) == -1) {
>          if (atomic_read(&mutex_head) == id &&
>              atomic_cmpxchg(&mutex_head, id, -1) == id) {
>              /* Last item in the list, exit.  */
> @@ -323,7 +323,7 @@ static void mcs_mutex_unlock(void)
>      }
>
>      /* Wake up the next in line.  */
> -    next = nodes[id].next;
> +    next = atomic_read(&nodes[id].next);
>      nodes[next].locked = 0;
>      qemu_futex_wake(&nodes[next].locked, 1);
>  }

Well it certainly looks like it improves the results on Travis.

Tested-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

--
Alex Bennée